Sponsored Content
Top Forums Shell Programming and Scripting Rsync Scripts for Offsite backups Post 302234728 by mchasse on Wednesday 10th of September 2008 07:51:22 AM
Old 09-10-2008
Lightbulb Rsync Scripts for Offsite backups

I am currently bringing up an offsite location, right now I am in the process of copying some data offsite (about 400GB).
The problem I see is that running a single rsync for everything is not using the available bandwidth and testing shows that I double in speed for each instance of Rsync I am running upto about 3 concurrent.

The script I am using runs only a single instance, but I would like to be able to scan the directories and start 2-3 rsync at a time. Is there a way to do this?

Current script:
Code:
#!/bin/bash
    PIDFILE=/var/log/rsync/fortrust_sync.pid
    LOGFILE=/var/log/rsync/fortrust_sync.log

    [[ -r $PIDFILE ]] && ps -p $(< $PIDFILE) 2>/dev/null && {
      echo "program already running" >> $LOGFILE
      exit 1
    }
rm -f $PIDFILE
echo $$ > $PIDFILE

# Source Machine Name (This is local)
SOURCE="SOURCEMACHINE"

# Destination host machine name (This is a remote machine)
DEST="DESTINATIONMACHINE"


# User that rsync will connect as
# Are you sure that you want to run as root, though?
USER="User"

# Directory to copy from on the source machine.
BACKDIR="/storback/rman"

# Directory to copy to on the destination machine.
DESTDIR="/mnt/storage/storback/"

# excludes file - Contains wildcard patterns of files to exclude.
# i.e., *~, *.bak, etc.  One "pattern" per line.
# You must create this file.
#EXCLUDES=/rsync/excludes

# Options.
# -n Don't do any copying, but display what rsync *would* copy. For testing.
# -a Archive. Mainly propogate file permissions, ownership, timestamp, etc.
# -u Update. Don't copy file if file on destination is newer.
# -v Verbose -vv More verbose. -vvv Even more verbose.
# See man rsync for other options.

# For testing.  Only displays what rsync *would* do and does no actual copying.
# OPTS="-n -vv -u -a --exclude-from=$EXCLUDES --stats --progress"
# Does copy, but still gives a verbose display of what it is doing
OPTS="-v -u -a --progress --rsh=ssh --exclude-from=$EXCLUDES --stats"
# Copies and does no display at all.
#OPTS="--archive --update --rsh=ssh --exclude-from=$EXCLUDES --quiet"

# May be needed if run by cron
export PATH=$PATH:/bin:/usr/bin:/usr/local/bin

# Only run rsync if $DEST responds.
VAR=`ping -s 1 -c 1 $DEST > /dev/null; echo $?`
if [ $VAR -eq 0 ]; then
echo $DEST ALIVE
else
    echo "Cannot connect to $DEST."
fi

# Only run rsync if $SOURCE responds.
VAR=`ping -s 1 -c 1 $SOURCE > /dev/null; echo $?`
if [ $VAR -eq 0 ]; then
echo $SOURCE ALIVE
rsync $OPTS $BACKDIR $USER@$DEST:$DESTDIR
else
    echo "Cannot connect to $SOURCE."
fi
rm -f $PIDFILE


Last edited by Yogesh Sawant; 12-10-2010 at 05:24 AM.. Reason: added code tags
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

backups

When using hostdump.sh to backup a system I can do it fine. But how can I restore what I backuped up? :) Thx in advance (2 Replies)
Discussion started by: merlin
2 Replies

2. UNIX for Dummies Questions & Answers

offsite backup for solaris

any hot sites out there for backup of a sun unix server. can't verify my tape backups and want a "disaster recovery" option (1 Reply)
Discussion started by: jbksman
1 Replies

3. UNIX for Dummies Questions & Answers

Backups.

Hello everyone my ? is about backups. I'am running SCO OS 505 and currently backing up the hole HD. Well the back up is taking too long and this is becoming a problem for the users since we are a 24-7 bussines, I whant stop backing up every thing on the HD. What are the most important files and... (1 Reply)
Discussion started by: kikkin
1 Replies

4. Shell Programming and Scripting

how to write orginal scripts to backups

hi iam new of scripting. give me some informtion of how to write backups file thanks&regards Naveen.g (1 Reply)
Discussion started by: naveeng.81
1 Replies

5. AIX

Offsite Backup Solution

I'm looking for a program or some way to backup my server to another location onto another server. Does anyone have any ideas? What utility, server O/S(I would like to use Linux RH or CentOS)? Whats the best way to do this? I am fairly new with Unix. Thanks in advance (6 Replies)
Discussion started by: ITAdmin08
6 Replies

6. UNIX for Dummies Questions & Answers

Rsync Scripts

hi problem mutiple remote servers which need a nightly backup to our main server. Most servers are red hat but a few suse. Would rather so remote ssh sync then mounting them on servers but not set in stone. need to be ale to run a full monitor f each sync with a clear report being created. ... (0 Replies)
Discussion started by: treds
0 Replies

7. UNIX for Advanced & Expert Users

Rsync Backups on HostGator VPS

I'm wanting to do remote backups of the entire /home/* directory structure, which is where HG puts all www files and sets the user and group independently of any other user ( so /home/user1 will be using group user1, instead of a generic apache group) The problem I'm running into is that only... (0 Replies)
Discussion started by: kettlewell
0 Replies

8. Shell Programming and Scripting

Backups using rsync

Hello all, I'm using nas4free as a SAN and am having troubles getting a backup of it's data to work properly. I've posted in the nas4free forums, but haven't received much help. Here is the code I'm using: #!/bin/sh {... (1 Reply)
Discussion started by: dpatino
1 Replies

9. Shell Programming and Scripting

Rsync Error: rsync: link_stat failed: No such file or directory (2)

I wish to copy all the files & folder under /web/Transfer_Files/data/ on mymac1 (Linux) to remote server mybank.intra.com (Solaris 10) /tmp/ location I am using Ansible tool synchronize module which triggers the unix rsync command as below:rsync --delay-updates -F --compress --archive --rsh=ssh... (2 Replies)
Discussion started by: mohtashims
2 Replies
RDUP-BACKUPS(7) 						       rdup							   RDUP-BACKUPS(7)

NAME
rdup-backups - introduction into making backups with rdup INTRODUCTION
rdup is a simple program that prints out a list of files and directories that are changed changed on a filesystem. It is more sophisticated than for instance find, because rdup will find files that are removed or directories that are renamed. A long time ago rdup included a bunch of shell and Perl scripts that implemented a backup policy. These could be used in a pipeline to per- form a backup. Currently rdup consists out of three basic utilities: rdup With rdup you create the file list on which later programs in the pipeline can work. The default output format also includes the files' content. rdup can be seen as a tar replacement in this respect, but rdup also allows for all kinds of transformations of the content (encryption, compression, reversal), see the -P switch in rdup(1) for more information. rdup-tr With rdup-tr you can transform the files rdup delivers to you. You can create tar, cpio or pax files. You can encrypt pathnames. rdup-tr is filter that reads from standard input and writes to standard output. See rdup-tr(1) for more information. With rdup and rdup-tr you can create an encrypted archive which is put in a directory structure that is also encrypted. rdup-up With rdup-up you can update an existing directory structure with the updates as described by rdup. rdup-up reads rdup input and will create the files, symbolic links, hard links and directories (and sockets, pipes and devices) in the file system. See rdup-up(1) for more information. So the general backup pipeline for rdup will look something like this: create filelist | transform | update filesystem ( rdup | rdup-tr | rdup-up ) Note 1: The same sequence is used for restoring. In both cases you want to move files from location A to B. The only difference is that the transformation is reversed when you restore. Note 2: The use of rdup-tr is optional. BACKUPS AND RESTORES
For rdup there is no difference between backups and restores. If you think about this for a minute you understand why. Making a backup means copying a list of files somewhere else. Restoring files is copying a list of files back to the place they came from. Same difference. So rdup can be used for both, if you did any transformation with rdup during the backup you just need to reverse those operations during the restore. BACKUPS
It is always best to backup to another medium, be it a different local harddisk or a NFS/CIFS mounted filesystem. You can also use ssh to store file on a remote server, ala rsync (although not as network efficient). If you backup to a local disk you can just as well use rsync or plain old tar, but if you store your files at somebody else's disk you will need encryption. This is where you go beyond rsync and rdup comes in. Rsync cannot do per-file encryption, sure you can encrypt the network traffic with ssh, but at the remote side your files are kept in plain view. If you implement remote backups, the easy route is to allow root access on the backup medium. If the backup runs without root access the created files will not have their original ownership. For NFS this can be achieved by using no_root_squash, for ssh you could enable PermitRootLogin. Note that this may be a security risk. SNAPSHOT BACKUPS
We need a little help here in the form of the rdup-simple script. Keep in mind that the following scripts can also be run remotely with the help of ssh. The following script implements the algorithm of rdup-simple. #!/bin/bash # some tmp files are saved in ~/.rdup. This directory must exist DIR=/home # what to backup BACKUP=/vol/backup TODAY=$(date +%Y%m/%d) LIST=~/.rdup/list-$HOSTNAME STAMP=~/.rdup/timestamp-$HOSTNAME # for remote backup, this has to run on the remote host! BUGBUG RET=$? case $RET in 2|*) echo Error >&2 exit 1 ;; 1) # full dump, remove file-list and time-stamp file rm $LIST $STAMP ;; 0) # inc dump # do nothing here ;; esac # this is the place where you want to modify the command line # right now, nothing is translated we just use 'cat' rdup -N $STAMP -Pcat $LIST $DIR | rdup-up $BACKUP/$HOSTNAME/$TODAY # or do a remote backup #rdup -N $STAMP -Pcat $LIST $DIR | ssh root@remotehost # rdup-up $BACKUP/$HOSTNAME/$TODAY LOCAL BACKUPS
With rdup-simple you can easily create backups. Backing up my home directory to a backup directory: rdup-simple ~ /vol/backup/$HOSTNAME This will create a backup in /vol/backup/$HOSTNAME/200705/15. So each day will have its own directory. Multiple sources are allowed, so: rdup-simple ~ /etc/ /var/lib /vol/backup/$HOSTNAME Will backup your home directory, /etc and /var/lib to the backup location. Also if you need to compress your backup, simple add a '-z' switch: rdup-simple -z ~ /etc/ /var/lib /vol/backup/$HOSTNAME REMOTE BACKUPS
For a remote backup to work, both the sending machine and the receiving machine must have rdup installed. The currently implemented proto- col is ssh. Dumping my homedir to the remote server: rdup-simple ~ ssh://miekg@remote/vol/backup/$HOSTNAME The syntax is almost identical, only the destination starts with the magic string 'ssh://'. Compression and encryption are just as easily enabled as with a local backup, just add '-z' and/or a '-k keyfile' argument: rdup-simple -z -k 'secret-file' ~ ssh://miekg@remote/vol/backup/$HOSTNAME Remember though, that because of these advanced features (compression, encryption, etc, ...) the network transfer can never be as efficient as rsync. ALSO SEE
rdup(1), rdup-tr(1), rdup-up(1) and http://www.miek.nl/projects/rdup/ 1.1.x 15 Dec 2008 RDUP-BACKUPS(7)
All times are GMT -4. The time now is 06:28 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy