Rsync Scripts for Offsite backups

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Rsync Scripts for Offsite backups
# 1  
Old 09-10-2008
Lightbulb Rsync Scripts for Offsite backups

I am currently bringing up an offsite location, right now I am in the process of copying some data offsite (about 400GB).
The problem I see is that running a single rsync for everything is not using the available bandwidth and testing shows that I double in speed for each instance of Rsync I am running upto about 3 concurrent.

The script I am using runs only a single instance, but I would like to be able to scan the directories and start 2-3 rsync at a time. Is there a way to do this?

Current script:

    [[ -r $PIDFILE ]] && ps -p $(< $PIDFILE) 2>/dev/null && {
      echo "program already running" >> $LOGFILE
      exit 1
rm -f $PIDFILE
echo $$ > $PIDFILE

# Source Machine Name (This is local)

# Destination host machine name (This is a remote machine)

# User that rsync will connect as
# Are you sure that you want to run as root, though?

# Directory to copy from on the source machine.

# Directory to copy to on the destination machine.

# excludes file - Contains wildcard patterns of files to exclude.
# i.e., *~, *.bak, etc.  One "pattern" per line.
# You must create this file.

# Options.
# -n Don't do any copying, but display what rsync *would* copy. For testing.
# -a Archive. Mainly propogate file permissions, ownership, timestamp, etc.
# -u Update. Don't copy file if file on destination is newer.
# -v Verbose -vv More verbose. -vvv Even more verbose.
# See man rsync for other options.

# For testing.  Only displays what rsync *would* do and does no actual copying.
# OPTS="-n -vv -u -a --exclude-from=$EXCLUDES --stats --progress"
# Does copy, but still gives a verbose display of what it is doing
OPTS="-v -u -a --progress --rsh=ssh --exclude-from=$EXCLUDES --stats"
# Copies and does no display at all.
#OPTS="--archive --update --rsh=ssh --exclude-from=$EXCLUDES --quiet"

# May be needed if run by cron
export PATH=$PATH:/bin:/usr/bin:/usr/local/bin

# Only run rsync if $DEST responds.
VAR=`ping -s 1 -c 1 $DEST > /dev/null; echo $?`
if [ $VAR -eq 0 ]; then
    echo "Cannot connect to $DEST."

# Only run rsync if $SOURCE responds.
VAR=`ping -s 1 -c 1 $SOURCE > /dev/null; echo $?`
if [ $VAR -eq 0 ]; then
    echo "Cannot connect to $SOURCE."
rm -f $PIDFILE

Last edited by Yogesh Sawant; 12-10-2010 at 06:24 AM.. Reason: added code tags
# 2  
Old 09-10-2008
It sounds like rsync could use a bit of parallelization. The only thing I can imagine doing is getting the contents of BACKDIR, splitting it into equal parts, and running rsync on each part.
dirsize=`find $BACKDIR -type d -a -prune |wc -l`
let splits= "($dirsize+1) / $THREADS"
find  $BACKDIR-type d -a -prune  | split -l $splits 
#split outputs files like xxa xxb etc
#tell each rsync instance to use a different file

However, this doesn't work well if the files in the subdirectories of BACKDIR are not equally distributed. Another mechanism would be to do a find of all directories, and split that list in a way that rsync can use... but that's tricky because you might have subdirectories of one list also in another list.
# 3  
Old 09-11-2008
Well the only thing that might help is that the directories under BACKDIR are static, only the dmp's in them change. I was wondering if it might help to run rsync --list-only dump that to a static file, split it and run it in a for loop. But then I am pretty new to this.
# 4  
Old 09-14-2008
Haven't tried that, but that sounds like a great idea.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Rsync Error: rsync: link_stat failed: No such file or directory (2)

I wish to copy all the files & folder under /web/Transfer_Files/data/ on mymac1 (Linux) to remote server (Solaris 10) /tmp/ location I am using Ansible tool synchronize module which triggers the unix rsync command as below:rsync --delay-updates -F --compress --archive --rsh=ssh... (2 Replies)
Discussion started by: mohtashims
2 Replies

2. Shell Programming and Scripting

Backups using rsync

Hello all, I'm using nas4free as a SAN and am having troubles getting a backup of it's data to work properly. I've posted in the nas4free forums, but haven't received much help. Here is the code I'm using: #!/bin/sh {... (1 Reply)
Discussion started by: dpatino
1 Replies

3. UNIX for Advanced & Expert Users

Rsync Backups on HostGator VPS

I'm wanting to do remote backups of the entire /home/* directory structure, which is where HG puts all www files and sets the user and group independently of any other user ( so /home/user1 will be using group user1, instead of a generic apache group) The problem I'm running into is that only... (0 Replies)
Discussion started by: kettlewell
0 Replies

4. UNIX for Dummies Questions & Answers

Rsync Scripts

hi problem mutiple remote servers which need a nightly backup to our main server. Most servers are red hat but a few suse. Would rather so remote ssh sync then mounting them on servers but not set in stone. need to be ale to run a full monitor f each sync with a clear report being created. ... (0 Replies)
Discussion started by: treds
0 Replies

5. AIX

Offsite Backup Solution

I'm looking for a program or some way to backup my server to another location onto another server. Does anyone have any ideas? What utility, server O/S(I would like to use Linux RH or CentOS)? Whats the best way to do this? I am fairly new with Unix. Thanks in advance (6 Replies)
Discussion started by: ITAdmin08
6 Replies

6. Shell Programming and Scripting

how to write orginal scripts to backups

hi iam new of scripting. give me some informtion of how to write backups file thanks&regards Naveen.g (1 Reply)
Discussion started by: naveeng.81
1 Replies

7. UNIX for Dummies Questions & Answers


Hello everyone my ? is about backups. I'am running SCO OS 505 and currently backing up the hole HD. Well the back up is taking too long and this is becoming a problem for the users since we are a 24-7 bussines, I whant stop backing up every thing on the HD. What are the most important files and... (1 Reply)
Discussion started by: kikkin
1 Replies

8. UNIX for Dummies Questions & Answers

offsite backup for solaris

any hot sites out there for backup of a sun unix server. can't verify my tape backups and want a "disaster recovery" option (1 Reply)
Discussion started by: jbksman
1 Replies

9. UNIX for Dummies Questions & Answers


When using to backup a system I can do it fine. But how can I restore what I backuped up? :) Thx in advance (2 Replies)
Discussion started by: merlin
2 Replies
Login or Register to Ask a Question

Featured Tech Videos