How to optimize our tape backups ? Post: 302292064

Sponsored Content

Operating Systems AIX How to optimize our tape backups ? Post 302292064 by bakunin on Thursday 26th of February 2009 08:17:12 PM

02-26-2009

Registered User

If you have a multiprocessor machine (which is most likely when you have a database server) you could do the following: pipe the output of tar to a gzip process, like the following command:

Code:

tar -cvf - <file-list> | gzip -9 > /some/file

The problem is the following: gzip is a single-threaded process (naturally). If you only issue the command as shown above you would use one single processor and the rest of the processing power would be idle.

Therefore you will have to determine the file sizes of the files to be backed up before and create as many instances of this process as you have processors, where you distribute the files as equally as possible and run these processes in parallel. This would utilize all the processors in your system and probably (since compressing is the most work-intensive task) drastically reduce the backup time. The following is a sketch of such a script:

Code:

no_of_procs=5
files[1]="file1 file2 file3"
files[2]="file4 file5 file6"
files[3]="file7 file8 file9"
files[4]="file10 file11 file12"
files[5]="file13 file14 file15"

(( i = 1 ))
while [ $i -le $no_of_procs ] ; do
     tar -cvf - ${files[$i]} | gzip -9 > backup.part${i} &
     (( i += 1 ))
done

If you have enough free file space write the backups to disk first (this is faster) and only write these files to tape. Your backup will in fact be done when the files are written to disk and it will not matter how long it takes to shuffle them off to tape.

I hope this helps.

bakunin

Last edited by bakunin; 02-26-2009 at 09:23 PM..

bakunin

View Public Profile for bakunin

Find all posts by bakunin

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Multiple backups on one tape

For those with backup tapes (and I just bought and installed a Seagate one for my FreeBSD box) I want to know how to get the most out of each tape by placing multiple backups on each tape (potentially 20GB). Please correct me if I'm wrong: First, retension the tape: # mt retension next, turn...

2. AIX

backups getting it so the tape doesn't rew.

Currently am running the backup command for AIX 5L and see that the tape is rewinding after the completion of the back. backup -0 -u -f /dev/rmt0 / >> $file 2>&1 What can I do to stop allow the backup to rew after the completion of this job? Any thoughts? Thanks again.

3. UNIX for Dummies Questions & Answers

verifying tape backups

Hello all, how would i go about verifying that a tape is backing up data correctly other than restoring the backup. for example, what command would i use to check the tape for errors? Any and all help is appreciated -Coffee

4. SCO

Tape Status shows 2 Hard errors and 5 Underruns on new tape

when I do a tape status /dev/rStp0 I get the following on a new tape and I have tried several: Status : ready beginning-of-tape soft errors : 0 hard errors: 2 underruns: 5 My BackupEdge has stopped backing up my system because it asks for a new volume yet my total system data is under 20...

5. Filesystems, Disks and Memory

Tape drives used for taking backups

Hi, I am a abit new in AIX system administration field. I want to gather knowledge about backup techniques. As per my knowledge we use Tape archives for taking backups. Can anyone pls explain me in detail abt tape archive? Whether these tape archives come along with the systems or we have to...

6. Solaris

Remote Backups to a Tape

My tape library is broken but backups still need to go on .I have 2 boxes running Solaris 10 Got SCSI tape drive attached to the DEV box ( my PROD Box has only fibre) I want to take the prod backup from the DEV box using ufsdump. ie /usr/sbin/ufsdump 0uvf DEV:/dev/rmt/1n...

7. AIX

Tape backups: do you always verify them after doing them ?

It may seam a bit odd that I ask this question. After you have done your backups to tapes, do you verify the content of the tapes ? - never - sometimes - always The reason I am asking is that here in the office, all the backup procedures include verifying the content of the tapes (no...

8. UNIX for Dummies Questions & Answers

Question about a tape write error doing backups

Hello all. UNIX dummy here :p Anyway I was trying to do a full backup of my work server SUN SPARC SERVER 1000 machine (yes we are actually using this dinosaur). I did the ufsdump comand and everything was fine until I got to the dumping of regular files. During the run I got the following...

9. SCO

SCO tape backups won't restore in Ubuntu Linux environment

Hello folks. I have the following problem: I'm trying to create a tape backup of a list of files on a 10 year old server, running SCO Openserver 5.0.5 (the tape drive is a Seagate STD224000N, connected as a SCSI drive). I then want to restore the contents of this tape onto a new server...

10. AIX

Best method to encrypt AIX LTO6 tape backups?

Hello, I need to be able to encrypt LTO tapes that our AIX writes to for backups. We have a tape library (IBM TS3100) that our AIX host uses to write to LTO6 tapes. We then take those tapes off-site and restore to another AIX system using a 3580-H6S LTO6 tape drive - this is a very simple...

LEARN ABOUT DEBIAN

barrybackup

BARRYBACKUP(1)						      General Commands Manual						    BARRYBACKUP(1)

NAME

       barrybackup - Barry Project's backup program for the BlackBerry handheld

SYNOPSIS

       barrybackup [-?][-d]

DESCRIPTION

       barrybackup is a GUI application for backing up and restoring Blackberry handheld databases.

       The  application  allows for filtering of databases for both backup and restore, so not all databases need to be backed up at once, nor all
       restored.

       Backups and configuration files are stored by default in the user's home directory, under ~/.barry/backup/PIN.	This  destination  can	be
       changed in the config dialogs, per device.

       The backup files are compressed tarballs containing specially named files for each record of the databases.

OPTIONS

       -d     --debug-output Enables low level protocol debug output written to stdout/stderr.

       --display=DISPLAY
	      Specify which X display to use.

       -?     --help Show summary of options.

       -h, --help
	      Show summary of options.

TAR FORMAT

       Backups are stored in tar format, compressed with gzip.	Backup files are named with the following pattern:

	    PIN-YYYYMMDD-HHMMSS[-tag_name].tar.gz

       The tag name is optional and is used to name a particular backup.

       Each record is appended to the tar file using the following pattern for the filename:

	    DBname/RecordID RecordTypeID

       That is, the database name is used as the directory name, and the filename contains the record ID and record type ID separated by a space.

       Database names can contain spaces.

       Record  IDs  are generally unique, but not all Blackberry devices mandate this, so it is possible, but rare, to have two records in the tar
       file with the same filename.  This is ok.  The only problem you'd see is if you expanded such a tar file  to  a	filesystem.   The  restore
       process just reads in the filename sequentially and writes them to the device, so duplicate record IDs are not a problem.

AUTHOR

       barrybackup is part of the Barry project.  This manual page was written by Chris Frey.

SEE ALSO

       http://www.netdirect.ca/software/packages/barry

								   July 28, 2009						    BARRYBACKUP(1)

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Multiple backups on one tape

Discussion started by: WIntellect

2. AIX

backups getting it so the tape doesn't rew.

Discussion started by: justinburbridge

3. UNIX for Dummies Questions & Answers

verifying tape backups

Discussion started by: coffeebrown

4. SCO

Tape Status shows 2 Hard errors and 5 Underruns on new tape

Discussion started by: psytropic