Librenix
Headlines | Linux | Apps | Coding | BSD | Admin | News
Information for Linux System Administration 

Scripting: A parallel Linux backup script

Up
vote
Down

This example bash shell script demonstrates a simple method of creating backups of multiple filesystems to multiple tape devices simultaneously. While the script presented writes to four tape drives in parallel, it can easily be modified to write to other device types and to create a different number of backup streams. The script is set up for the bash shell under Linux, but modifying it for another variety of Unix should simply be a matter of changing the locations of utility files such as tar, echo, cp, and sleep.

The script can be downloaded from http://librenix.com/scripts/par.tar.sh. Download the file now and load it into an editor as this article will refer to it frequently. Also, you may want to modify bits of it to match your filesystem names and your devices.

The first line of the script looks like this:
 #!/bin/bash
If the bash shell isn’t in the /bin directory on your system, you’ll need to modify this line. Enter the command which bash now to verify the location of bash. My Fedora Linux system and my Mac OS X system both have bash in /bin, but my FreeBSD system does not. If you have a non-Linux flavor of Unix, you’ll probably need to use the ‘which’ command to verify the locations of each command used in the script. The commands used are:
 bash
cd
sleep
echo
date
tar
wait
ls
wc
Note that ‘wait’ and ‘cd’ are usually implemented as internal shell commands and may not have external commands associated with them. If that is true for your system, leave ‘cd’ and ‘wait’ with no directory prefix just as they are in the original script.

Now, the first command in the script resets the current working directory to ‘/’:
 cd /
Since the script precedes each directory to be backed up with a ‘.’ to represent the current working directory, starting out at ‘/’ is necessary. The reason for this precaution is that some implementations of the tar command will only load files from a tar archive into the exact directory that was specified when the file was backed up. By prefixing the names with a ‘.’ we preserve the ability to recover the files into any subdirectory we want, without overwriting the original files.

Immediately after the ‘cd /’ command is where you would put any commands to shut down all services that must be quieted prior to a backup. The example script has a (commented out) command to initiate an Oracle database shutdown followed by a ‘sleep’ command to allow time for the shutdown to complete. The example database shutdown and the following delay probably don’t apply to your system. Obviously, you’ll have to add commands yourself to stop any applications that might interfere with the backup.

Next, we use the ‘date’ command to create two sets of four tiny files to stick at the start and end of each tape. Note that the presence of a ‘date.#’ file at the beginning of each tape lets you quickly find out when a tape was created and on which drive. The ‘zzzz.#’ files, appended to the end of each tape, only serve to let you easily verify that a backup completed without overrunning the end of the tape.

Next, we start the four actual ‘tar’ backup commands, each with sample directories named ‘./dir1’, ‘./dir2’, etc. Of course, you’ll need to modify the list of directories to match the actual directories you wish to back up. Note that you’ll probably want to balance the directory sizes so that all of the largest directores aren’t on the same tape. Also, note that each ‘tar’ command is run in the background and logs to a tar.#.log file in the /tmp directory. Obviously, you might want to put the logfiles somewhere else.

After each ‘tar’ command there is an entry like this: ‘TASK=$0’, or ‘TASK=$1’. These arbitrarily-named ‘TASK’ variables are used to store the process ID of each ‘tar’ command so that the script can wait for them with the four ‘wait’ commands that follow in the next block of code. There, we have the four ‘wait’ commands waiting on the $TASK0, etc, variables. (The addition of the ‘$’ to each TASK# shell variable is not a typo -- it’s necessary to read back the contents of the variable.)

Next, after the script has waited for the completion of each of the four ‘tar’ commands, it appends some information to a history file for later reference. It stores the date of the backup, the filesize of the logfile, and the number of files backed up on each tape to each of four history files. While the script will overwrite the logfiles (tar.#.log) each time it is run, it will append these three lines to each of the four history files (tar.#.history).

The final steps in the script are commented out. Those are the commands necessary to restart any applications that were brought down for the backup. Again, in the example we assume an Oracle database needs to be restarted. You’ll need to add the commands necessary to start any applications that were stopped at the beginning of the script.
mail this link | permapage | score:9129 | -Ray, April 10, 2005

Rsnapshot: Archive, Snapshot, Backup MySQL Databases on Debian

Up
vote
Down

This howto will show you how to install and set up Rsnapshot, enable archiving of snapshots and how to back up MySQL databases on Debian. Rsnapshot is a filesystem snapshot utility for making backups of local and remote systems. Using rsync and hard links, it is possible to keep multiple, full backups instantly available. The disk space required is just a little more than the space of one full backup, plus incrementals. read more...
permapage | score:8852 | -falko, January 14, 2013

BackupPC Installation And Configuration

Up
vote
Down

BackupPC is a high-performance, enterprise-grade system for backing up Linux and WinXX PCs and laptops to a server's disk. BackupPC is highly configurable and easy to install and maintain.
A clever pooling scheme minimizes disk storage and disk I/O. Identical files across multiple backups of the same or different PCs are stored only once resulting in substantial savings in disk storage and disk I/O.
read more...
permapage | score:8776 | -gg234, June 19, 2006

Tutorial: Create Encrypted FTP backups with duplicity, duply on Debian 6

Up
vote
Down

When you rent a dedicated server nowadays, almost all providers give you FTP backup space for your server on one of the provider's backup systems. This tutorial shows how you can use duplicity and duply to create encrypted backups on the provider's remote backup server over FTP. duply is a duplicity wrapper script that allows us to use duplicity without interaction (i.e., you do not have to type in any passwords). read more...
permapage | score:8642 | -falko, July 9, 2012

Tutorial: Backup LVM Partitions with Snapshots

Up
vote
Down

This tutorial shows how you can create backups of LVM partitions with an LVM feature called LVM snapshots. An LVM snapshot is an exact copy of an LVM partition that has all the data from the LVM volume from the time the snapshot was created. The big advantage of LVM snapshots is that you do not have to worry about open files and database connections, and you do not have to interrupt/halt services on the live partition because a snapshot is usually created in fractions of a second, so your users will not notice any disruption, and your snapshot holds consistent data. read more...
mail this link | permapage | score:8603 | -falko, April 18, 2007

Back up MySQL Databases with mylvmbackup on Debian 6

Up
vote
Down

mylvmbackup is a Perl script for quickly creating MySQL backups. It uses LVM's snapshot feature to do so. To perform a backup, mylvmbackup obtains a read lock on all tables and flushes all server caches to disk, creates a snapshot of the volume containing the MySQL data directory, and unlocks the tables again. This article shows how to use it on a Debian Squeeze server. read more...
permapage | score:8448 | -falko, May 11, 2012

Tutorial: Making Linux / UNIX snapshot backups

Up
vote
Down

I’d like to configure my Debian box to backup two remote servers using rsnapshot software. It should make incremental snapshots of local and remote filesystems for any number of machines on 2nd hard disk located at /disk1 ( /dev/sdb2). How do I make backups?

rsnapshot saves much more disk space than you might imagine. The amount of space required is roughly the size of one full backup, plus a copy of each additional file that is changed. rsnapshot makes extensive use of hard links, so if the file doesn’t change, the next snapshot is simply a hard link to the exact same file. read more...
mail this link | permapage | score:8413 | -nixcraft, February 12, 2008

Tutorial: Set up Postfix as a backup MX

Up
vote
Down

In this tutorial I will show how you can set up a Postfix mailserver as a backup mail exchanger for a domain so that it accepts mails for this domain in case the primary mail exchanger is down or unreachable, and passes the mails on to the primary MX once that one is up again. read more...
permapage | score:8326 | -falko, June 25, 2007

Backup MySQL with Zmanda Recovery Manager

Up
vote
Down

Zmanda Recovery Manager (ZRM) for MySQL simplifies life of a database administrator who needs an easy to use yet flexible and robust backup and recovery solution for MySQL server.
  • Schedule full and incremental logical or raw backups of your MySQL database
  • Centralized backup management with monitoring and email status notification
  • Verify your backup images
  • Compress and encrypt your backup images
  • Recover database easily to any point in time or to a particular database event
read more...
mail this link | permapage | score:8228 | -gg234, September 10, 2008

JBackpack: Personal backup program

Up
vote
Down

JBackpack is a personal backup program. It features incremental backups, network transparency and encryption. JBackpack uses rdiff-backup for all backup and restore functions, SSHFS and SMB to access remote file systems and EncFS to encrypt backup destination directories.JBackpack uses rdiff-backup for all backup and restore functions. The most interesting feature of rdiff-backup is incremental backups. read more...
permapage | score:8116 | -gg234, April 23, 2012

Linux automated remote backup: rsnapshot

Up
vote
Down

Use rsnapshot to keep multiple snapshots of your data without taking up all that space, and do it from a remote machine...
Rsnapshot is included in the native distributions of OpenDarwin, the three BSDs, GenToo, and Debian Linux. There is a version for SlackWare available here. If your platform is not in that list, you can download the latest source tarball here and build it yourself. You'll need to have both perl and rsync installed on your system for rsnapshot to work. It's also good idea to have OpenSSH, logger, GNU cp, and GNU du as well.
read more...
mail this link | permapage | score:8085 | -Ray, May 9, 2005

Bacula Network Backup Implementation Guide

Up
vote
Down

This is very detailed tutorial for implemeting bacula network backup in debian linux. This tutorial consists of four pages.

Bacula is a set of computer programs that permits you (or the system administrator) to manage backup, recovery, and verification of computer data across a network of computers of different kinds. Bacula can also run entirely upon a single computer, and can backup to various types of media, including tape and disk.

In technical terms, it is a network Client/Server based backup program. Bacula is relatively easy to use and efficient, while offering many advanced storage management features that make it easy to find and recover lost or damaged files. Due to its modular design, Bacula is scalable from small single computer systems to systems consisting of hundreds of computers located over a large network. read more...
mail this link | permapage | score:8072 | -gg234, June 7, 2006

mysqlhotcopy: Hot Backup of MySQL

Up
vote
Down

If you need to make live backups of your MySQL database, this is how you can do it.
If you want to make a hotcopy from your SQL database instead of a dump to a text file, you can use the mysqlhotcopy tool. This tool locks a table, copy it and than unlocks it again.

Use this command if you want to transfer the MySQL data to an other place on the same server...
read more...
permapage | score:8065 | -Ray, December 22, 2005

Amanda: Disk backup on Debian

Up
vote
Down

Amanda is an open source client/server solution to back up filesystems. Backups are triggered by the backup server, backup definitions are located on the servers but exclusion lists are located on the client. read more...
permapage | score:7964 | -falko, June 17, 2010

Tutorial: Postfix backup mail server anti-spam setup

Up
vote
Down

According to RFC2821 the lowest-numbered records are the most preferred MX for domain. So a target Postfix backup server is used to keep the messages in a queue waiting for the primary server to become available. This ensures that if my primary MX goes down you do not loss any emails. However, spammers are connects to backup MX to avoid anti spam filters that are running on the primary MX server. This also hides their real IP from my primary MX. This tutorial shows how to configure anti-spam and anti-virus for Red Hat / CentOS Linux ver5.3 based Postfix mx server. read more...
mail this link | permapage | score:7940 | -nixcraft, May 5, 2009

Linux Backup: Simple Backup Suite

Up
vote
Down

Simplify your backup procedures with the Simple Backup Suite.
The simple-backup-config tool allows users to choose either a simple backup scheme that performs incremental backups daily and full backups weekly, or to modify the default scheme with their own settings. Users list directories to backup in the Include tab, and select items to exclude in the Exclude tab. Exclusions can be based on directory (e.g., /var/cache), file type (sbackup excludes multimedia files by default), file size (10MB is the default upper limit), or regular expression.
read more...
mail this link | permapage | score:7907 | -Ray, November 29, 2005

zyxBackup: A Linux Backup Shell Script

Up
vote
Down

zyxBackup is a shell script, designed to perform different backups on a local computer. It makes a compressed archive of all backup selected in the configuration. It can also encrypt the archive, and upload it on a FTP server, or copy it via SCP.

Features (can be activated separately) :
  • Backup a selection of files and/or directories.
  • Dump MySQL databases.
  • Backup users' homes (full or only hidden files).
  • Backup the list of installed packages (Debian / Ubuntu).
  • Encrypt the final archive using GPG.
  • Upload final archive via FTP (using ftp-upload) and/or SCP.
  • Send a report by email.
read more...
mail this link | permapage | score:7863 | -tchetche, November 7, 2008 (Updated: November 15, 2008)

Tutorial: Red hat / CentOS Remote Snapshot Backup Server

Up
vote
Down

rsnapshot is easy, reliable and disaster recovery backup solution. It is a remote backup program that uses rsync to take backup snapshots of filesystems. It uses hard links to save space on disk.

You can easily build remote / local backup server to keep snapshots in in hourly, daily and monthly format. read more...
permapage | score:7846 | -nixcraft, July 10, 2008

Newbie Command Line: Linux Backup

Up
vote
Down

Backups aren't important -- until you need them.
Quick. When is the last time you backed up your system? Unless you are a system administrator by trade, I bet 95% of you haven't done so recently. If you suffer a hard drive crash, you stand to lose all your recent email, work, notes, photos, bookmarks, and whatever. One reason for that might be the paucity of a easy to use backup tools. Let's bury that reason once and for all.
read more...
permapage | score:7779 | -Ray, May 21, 2004

Tutorial: Easy Automated Linux Snapshot Backups

Up
vote
Down

Automate those backups!
This document describes a method for generating automatic rotating "snapshot"-style backups on a Unix-based system, with specific examples drawn from the author's GNU/Linux experience. Snapshot backups are a feature of some high-end industrial file servers; they create the illusion of multiple full (up to ownership/permission) backups per day without the space or processing overhead. All of the snapshots are read-only, and are accessible directly by users as special system directories. It is often possible to store several hours, days, and even weeks' worth of snapshots with slightly more than 2x storage. This method, while not as space-efficient as some of the proprietary technologies (which, using special copy-on-write filesystems, can operate on slightly more than 1x storage), makes use of only standard file utilities and the common rsync program, which is installed by default on most Linux distributions. Properly configured, the method can also protect against hard disk failure, root compromises, or even back up a network of heterogeneous desktops automatically.
read more...
mail this link | permapage | score:7754 | -Ray, September 9, 2002
More articles...
Abstract Art Prints on Canvas

Recent headlines

Linux mail command examples: send mails from command line

10 basic examples of Linux ps command

Set up Oneiric PVHVM at Xen 4.1.2 Ubuntu 11.10 Dom0

Tutorial: Debian 7 Samba Server with tdbsam

Ubuntu Edge: Is there life after an unsuccessful crowd-funding campaign?

LG 8-inch G Pad 8.3 Android tablet

How to install Orange in Ubuntu 13.04

Hand of Thief trojan and your favorite Linux distribution

Epoptes – Open source computer lab management and monitoring tool

Tutorial: Automatically add a disclaimer to emails with alterMIME (Postfix on Debian 6)

Tutorial: Ubuntu Jeos 12.04 LTS Spam Filter Gateway Server

Dual-boot Windows 7 and Ubuntu 12.04 on a PC with UEFI board, SSD and HDD

Pipelight – Using Silverlight in Linux browsers

Tutorial: Running CS-Cart on Nginx on Debian 7/Ubuntu 13.04

Plasma Media Center 1.1 and digiKam 3.3

Tahoe Least-Authority File System for secure, distributed data storage

Upgrade Fedora 18 to 19 With FedUp

Nuvola Player: Enjoy all your Cloud music services from one interface

Beginner Ubuntu Tips

Sagemath in the Cloud and Sagemath 5.11

Using Multiple PHP Versions (PHP-FPM FastCGI) With ISPConfig 3 (Ubuntu 12.04)

OpenShift Online: a non-developer guide

Install FB4Linux in Eclipse

Tutorial: Create an NFS-like Storage Server with GlusterFS on Ubuntu 12.10

Preview: Snapdragon SDK for Android

Pre-release Ubuntu 12.10 has partial support for manual LVM and disk encryption

Setup Nginx + php-FPM + apc + MariaDB on Debian: The perfect LEMP server

StartOS 6 GNOME 3 and KDE preview

Add an entry for a Linux distribution in Windows boot menu

Unix: Shell Script Wrapper Examples

Better Grails apps with CSS

Microsoft Surface RT is an Unmitigated Disaster

Tutorial: Fedora 19 Samba server with tdbsam

Triple-boot Windows 7, Ubuntu 12.10 and Fedora 18 on one HDD

Migrate Mailboxes between IMAP Servers with imapcopy

Using OpenVZ on Debian 7 (AMD64)

Tutorial: Webcam streaming your desktop plus audio with ffmpeg, crtmpserver, Flowplayer

Linpus Lite 1.9 review

Linux Iptables Examples

Build a Firebird 2.5.1 and FreeBSD 9 database server

 

Firefox sidebar

Site map

Site info

News feed

Features

Login
(to post)

Search

 
Articles are owned by their authors.   © 2000-2012 Ray Yeargin