Scripting: A parallel Linux backup script
|This example bash shell script demonstrates a simple method of creating backups of multiple filesystems to multiple tape devices simultaneously. While the script presented writes to four tape drives in parallel, it can easily be modified to write to other device types and to create a different number of backup streams. The script is set up for the bash shell under Linux, but modifying it for another variety of Unix should simply be a matter of changing the locations of utility files such as tar, echo, cp, and sleep. |
The script can be downloaded from http://librenix.com/scripts/par.tar.sh. Download the file now and load it into an editor as this article will refer to it frequently. Also, you may want to modify bits of it to match your filesystem names and your devices.
The first line of the script looks like this:
#!/bin/bashIf the bash shell isn’t in the /bin directory on your system, you’ll need to modify this line. Enter the command which bash now to verify the location of bash. My Fedora Linux system and my Mac OS X system both have bash in /bin, but my FreeBSD system does not. If you have a non-Linux flavor of Unix, you’ll probably need to use the ‘which’ command to verify the locations of each command used in the script. The commands used are:
bashNote that ‘wait’ and ‘cd’ are usually implemented as internal shell commands and may not have external commands associated with them. If that is true for your system, leave ‘cd’ and ‘wait’ with no directory prefix just as they are in the original script.
Now, the first command in the script resets the current working directory to ‘/’:
cd /Since the script precedes each directory to be backed up with a ‘.’ to represent the current working directory, starting out at ‘/’ is necessary. The reason for this precaution is that some implementations of the tar command will only load files from a tar archive into the exact directory that was specified when the file was backed up. By prefixing the names with a ‘.’ we preserve the ability to recover the files into any subdirectory we want, without overwriting the original files.
Immediately after the ‘cd /’ command is where you would put any commands to shut down all services that must be quieted prior to a backup. The example script has a (commented out) command to initiate an Oracle database shutdown followed by a ‘sleep’ command to allow time for the shutdown to complete. The example database shutdown and the following delay probably don’t apply to your system. Obviously, you’ll have to add commands yourself to stop any applications that might interfere with the backup.
Next, we use the ‘date’ command to create two sets of four tiny files to stick at the start and end of each tape. Note that the presence of a ‘date.#’ file at the beginning of each tape lets you quickly find out when a tape was created and on which drive. The ‘zzzz.#’ files, appended to the end of each tape, only serve to let you easily verify that a backup completed without overrunning the end of the tape.
Next, we start the four actual ‘tar’ backup commands, each with sample directories named ‘./dir1’, ‘./dir2’, etc. Of course, you’ll need to modify the list of directories to match the actual directories you wish to back up. Note that you’ll probably want to balance the directory sizes so that all of the largest directores aren’t on the same tape. Also, note that each ‘tar’ command is run in the background and logs to a tar.#.log file in the /tmp directory. Obviously, you might want to put the logfiles somewhere else.
After each ‘tar’ command there is an entry like this: ‘TASK=$0’, or ‘TASK=$1’. These arbitrarily-named ‘TASK’ variables are used to store the process ID of each ‘tar’ command so that the script can wait for them with the four ‘wait’ commands that follow in the next block of code. There, we have the four ‘wait’ commands waiting on the $TASK0, etc, variables. (The addition of the ‘$’ to each TASK# shell variable is not a typo -- it’s necessary to read back the contents of the variable.)
Next, after the script has waited for the completion of each of the four ‘tar’ commands, it appends some information to a history file for later reference. It stores the date of the backup, the filesize of the logfile, and the number of files backed up on each tape to each of four history files. While the script will overwrite the logfiles (tar.#.log) each time it is run, it will append these three lines to each of the four history files (tar.#.history).
The final steps in the script are commented out. Those are the commands necessary to restart any applications that were brought down for the backup. Again, in the example we assume an Oracle database needs to be restarted. You’ll need to add the commands necessary to start any applications that were stopped at the beginning of the script.
|mail this link | permapage | score:9414 | -Ray, April 10, 2005|
Librenix T-Shirts and Coffee Mugs!
|For today's example of my (semi)elite C programming skilz, I submit for your inspection the Librenix T-Shirts! Yes, I created the images on these shirts and coffee mugs entirely with C code. While the code isn't up to the standards *cough* of my open source Space Tyrant project, at least the output is colorful and not entirely textual!|
click either image to see the T-Shirts, Coffee Mugs, etc.
(If you like the images but don't care for 'librenix' on your shirt, these same styles are available for all 50 US state names as well as with the signs of the zodiac here)
(and here are some modern prints)
|mail this link | permapage | score:9395 | -Ray, June 6, 2010 (Updated: May 13, 2014)|
Missing the point of the Mac Mini
|I've read several articles and numerous comments over the past week detailing just how overpriced Apple's new Mac Mini is. Reviewers seem to conclude that because they can assemble a PC of similar performance to the Mini for less money, that the new Mac simply costs too much.|
What they have not done, however, is duplicate the Mac Mini in any important way. The closest comparison I've seen pitted the Mini against a machine 2.5 times its size. At least that reviewer understood that size matters. I'm a fan of small systems. I own 2 Biostar iDEQ cubes, one Shuttle, and three Book PC's. The Book PC's are the oldest and most obsolete, of course, with the fastest one containing a Pentium III 667. I've gotten rid of several systems over the past two years that were faster than the Book PC's. Why keep the slower computers while getting rid of systems up to twice as fast, you might ask.
I've kept the Book PC's because they are so small that keeping them around isn't a burden. They take about as much space on a bookshelf as an unabridged dictionary. I have one, currently disconnected, functioning only as a monitor stand for the system I'm using right now. A book PC will fit in my briefcase; I've hauled them around with me as if they were laptops. With a dozen computers around the house, space is precious and small is beautiful.
The Book PC's are 3.2" x 10.5" x 11.9". That's 4.8 times the volume of a Mac Mini. The Mac is truly tiny. I've worked to build fast, small, quiet Linux systems for years now. The iDEQ 200V is the cheapest system I've made that is fast, quiet, and runs Linux without complaint. Without software and with only the on-board graphics chip, it cost about the same amount as the Mac Mini. At 12.5" x 7" x 8", however, it is much larger than the Mini and weighs several times as much.
I challenge the anti-Mini crowd to build a PC of any shape that displaces approximately the same volume as the Mini plus power supply. Then, compare prices again. The SFF computer fans are clearly going to notice this machine and are going to buy a few truckloads of them. In the small form factor (SFF) computer market, even ignoring the software, this machine is clearly a bargain.
SFF computer fans who are committed to Windows will still covet this system; a few of them might even make the switch to OS X just to get one. I even expect some SFF Linux geeks to buy them because they're tiny, cheap, and can run Linux. Conclusion: the anti-Mini reviewers and posters are not SFF people.
Next, the Mini is an affordable and typically stylish Mac. A smallish PC does not run OS X. The Mini comes with OS X and will make a great second (or third) computer for many Mac users. I use Linux as my primary desktop OS (SuSE 9.2 Professional for the last three weeks, Fedora Core 2 the previous year) and FreeBSD and Linux (Fedora, Slackware) for my servers. I'm hardly a Mac guy but, as a Unix geek, I'm perfectly fine with OS X. I used a Mac as my primary desktop for a couple of weeks after a recent move.
Many Mac users -- at least those who need a second system -- will find the price -- and the size -- of this system quite appealing. Clearly, the negative reviewers and posters are not OS X users.
Therefore, I've come to the conclusion that these anti-Mac Mini arguments are coming from people who appreciate neither of the core characteristics of the machine. They don't understand the appeal of the SFF systems market, nor are they OS X / Mac users.
Apple, on the other hand, appreciates both and they have produced an impressively priced small form factor OS X system.
I wish for Apple responsive suppliers with scalable production facilities. They will surely need them in order to satisfy the demand for the Mac Mini.
|mail this link | permapage | score:9339 | -Ray, January 21, 2005|
Install a Mail Server with Antivirus and Antispam in minutes
|This article illustrates a situation where you need to set up your own mail server (be it your home mail server, or a small office one). It actually shows that, if using an integrated service mail server, anyone can do the job, all in a matter of minutes.|
AXIGEN Mail Server, the solution chosen for this example, can send and receive e-mails securely via "mydomain.com" and is able to retrieve them in a WebMail interface - this means that it includes all mail services needed for a fully functional mail server (SMTP, IMAP, POP3, WebMail, WebAdmin).
To get an idea of the amount of time you can spare by installing such a solution, just think of all the different open source applications you would need to install instead (i.e. an MTA, Squirrelmail for Webmail, QmailAdmin for web configuration, Courier for IMAP and POP3 and many others.)
AXIGEN Mail Server can virtually integrate with any Antivirus/Antispam application and it comes with built-in connectors ClamAV Antivirus and SpamAssassin. The second part of this article shows you how to install these applications and configure these connectors for use with AXIGEN.
Thus, at the end of this process which can take up half an hour at most, you will not only have your mail server up and running, but also virus and spam protection for your incoming and outgoing mail traffic.
AXIGEN runs on several Linux distributions (Gentoo, Redhat/Fedora Core, Slackware, Debian, Ubuntu, Mandrake/Mandriva, SUSE), on BSD versions (FreeBSD, OpenBSD and NetBSD) and on Solaris but for the purpose of this article, let's suppose you are setting up your mail system on a Fedora Core 6 platform.In five easy steps, you will have your server installed, your primary domain running and access to the Web configuration interface (WebAdmin).
1. Download / unpack corresponding package
Download AXIGEN rpm package from the AXIGEN website (packages are available as 30 day evaluation versions). Save the corresponding package for Fedora Core 6 "axigen-2.0.4.i386.rpm.gcc4.tar.gz" on your local machine and unpack the file, by issuing in the same directory as the download file:
tar xzvf axigen-2.0.4.i386.rpm.gcc4.tar.gz
2. Install command
Then, in order to install the RPM package, issue (while logged in as root) the following command, from the same directory as the rpm file:
rpm -ivh axigen-2.0.4.gcc4-1.i386.rpmThis will create the entire directory structure needed for AXIGEN to run. After the installation, no daemons or related application will be started.
3. Configuration options
AXIGEN provides several configuration options (configuration file, Command Line Interface), but the most intuitive and comprehensive one is WebAdmin, the Web configuration interface.
The corresponding WebAdmin service is enabled by default, as well as the other default services: IMAP, Logging, POP3, Processing and SMTP.
4. Initial configuration
The first configuration steps take place using the configuration wizard. You will set the administrator's password, select which services are started and what interfaces will be used. In this stage of the setup you also create the primary domain that your server will use.
The wizard can be run by issuing the following command in the console right after the installation of the package has finished:
/opt/axigen/bin/axigen-cfg-wizardNOTE: You have to make sure you do not start the mail server before the initial configuration.
5. Start AXIGEN
You can then start AXIGEN, using its initscript, by issuing this command:
/etc/init.d/axigen startNow that your server is running, you can connect the antivirus and anti-spam applications. By default, AXIGEN comes with connectors for the ClamAV Antivirus and SpamAssasin Antispam application. The setup process below describes how to make these two applications work with AXIGEN. However, note that AXIGEN implements a proprietary filter scripting language that allows you to implement connectors for any third party Antivirus and Antispam applications.
Connecting to ClamAV
A. Install ClamAV (daemon), on the same machine on which AXIGEN Mail Server is installed. Follow these steps in order to configure ClamAv for use with AXIGEN and start the clamd daemon.
1. Install clamav-server, using yum (Yellow Dog Updater, Modified):
yum install clamav-server2. Copy the sample config file shipped with clamav-server:
cp /usr/share/doc/clamav-server-*/clamd.conf /etc/clamd.d/axigen.conf3. Edit: /etc/clamd.d/axigen.conf
# comment out the Example line 4. Create a link to the clamd binary:
# insert/modify the following lines:
ln -s /usr/sbin/clamd /usr/sbin/clamd.axigen5. Create the run directory, where the PID file and clamd socket will be stored, and change its permissions:
mkdir -p /var/run/clamd.axigen chown axigen:axigen /var/run/clamd.axigen6. Create and setup the initscript:
cp /usr/share/doc/clamav-server-*/clamd.init /etc/init.d/clamd.axigen 7. Edit: /etc/init.d/clamd.axigen and modify the following lines, as specified below:
chmod 755 /etc/init.d/clamd.axigen /sbin/chkconfig clamd.axigen on
# description: The clamd server running for axigen CLAMD_SERVICE=axigen8. Finally, start the clamd daemon:
B. Configure AXIGEN antivirus filter at server level using WebAdmin
In order to activate the ClamAV filter, go through the following steps:
In the "Server" context, click on the Add new filter button. This will open up and display the Active Filter list. It is empty right now, so we need to add the clamav filter to the list.
In the Priority field, enter a priority between 0 and 500 (a filter with priority 0 will be applied first and the one with 500, last).
Important - the domain-level filters have the priority limited to range 100-400 and the user-level filters are limited to the 200-300 range. A value of "10" should be fine, leaving you space to apply some other future filters before this one.
After setting the filter priority, select the socket value from in the Filter type dropdown list and the clamav value from the Filter Name list.
In the Apply on checklist, select the relay option, to apply the filter on outgoing mails. To make sure you scan both incoming and outgoing mails, you have to create the filter and select both values, local and relay.
In AXIGEN, it is possible to enable filters either at domain or user level, in the corresponding WebAdmin tabs. The filters activated at server level will be automatically applied for all domains and accounts. However, you have the possibility to add additional filters at domain or account level.
Connecting to SpamAssasin
The process for Connecting SpamAssassin is similar and even less time-consuming as no configurations are necessary after the product installation.
C. Install SpamAssassin using the yum application:
yum install spamassassinNo further configurations are necessary.
D. Configure SpamAssassin at server level, using Webadmin
The connector for SpamAssassin is a socket filter for AXIGEN, so the configuration procedure is the same as for ClamAV. The difference would be that for SpamAssassin, a TCP socket is more likely to be used.
Also, when activating the SpamAssassin filter, you need to keep in mind the following:
- Enter a different priority value for the SpamAssasin filter (if you have chosen 10 for ClamAV, choose a higher value for SpamAssassin in order to apply this filter after ClamAV in the filtering chain)
- Select the corresponding filter name, spamassassin in the Filter name list
Access AXIGEN WebMail
At this step of the way, your mail server is ready to go, and you can also you can access the AXIGEN WebMail to send and receive test messages. Then, use the full email address and password to log on to AXIGEN WebMail, at the default address: http://127.0.0.1:8000, or use the address you specified in the initial configuration phase when you ran the setup wizard.
Now you're really done: you can securely send and receive messages from your home domain and easily make any further configurations, to accommodate your specific network requirements. As you have seen, installing all mail services from one single executable and an intuitive Web configuration interface make things a lot easier and a lot less time-consuming.
Liviu Anghel, Chief Security Officer, Gecad Technologies
Ciprian Negrila, Technical Support Engineer, Gecad Technologies
|mail this link | permapage | score:9221 | -Kayla Vincent, February 6, 2007||