Showing off my rsync-to-USB script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Showing off my rsync-to-USB script
# 1  
Old 10-19-2009
Bug Showing off my rsync-to-USB script

Thank you all for helping me figure out how to manage spaces in paths. The following script is the result. The script uses rsync to backup files to a USB device. Special thanks to ScrutinizerSmilie.

rsync2u is a free menu-driven script. Down load the script from http://sites.google.com/site/rsync2u/home

Last edited by wolfv; 03-12-2010 at 09:56 PM.. Reason: move rsync2u script to its own site
# 2  
Old 10-19-2009
Nice Smilie

You might want to add a check that the destination path is writable as well.

I'll try this on my Ubuntu server.

Cheers

---------- Post updated at 09:00 AM ---------- Previous update was at 08:55 AM ----------

A check of the exit values from rsync would be a good idea: -0Success 1Syntax or usage error 2Protocol incompatibility 3Errors selecting input/output files, dirs 4Requested action not supported: an attempt was made to manipulate 64-bit files on a platform that cannot support them; or an option was specified that is supported by the client and not by the server. 5Error starting client-server protocol 6Daemon unable to append to log-file 10Error in socket I/O 11Error in file I/O 12Error in rsync protocol data stream 13Errors with program diagnostics 14Error in IPC code 20Received SIGUSR1 or SIGINT 21Some error returned by waitpid() 22Error allocating core memory buffers 23Partial transfer due to error 24Partial transfer due to vanished source files 25The --max-delete limit stopped deletions 30Timeout in data send/receive 35Timeout waiting for daemon connection
# 3  
Old 10-19-2009
If you want to do rotating (incremental) backups, rsync can create hard links on the destination server. Hard links will conserve disk space on your USB key while doing rotating backups.

Check option --link-dest in the rsync manual.

I personally prefer the good old cp -l method to create hardlinks. May be I should try again to use rsync to manage the hardlinks but it's so easy with cp.

With that hard link trick I do rotating snapshots every day (31 days max) and one snapshot every month (max 12 month). Depending on how often your files change, you can save *a lot* of space on your destination.

Google "hardlink rsync" and you will find interesting scripts.
# 4  
Old 10-21-2009
Quote:
Originally Posted by steadyonabix
Nice Smilie

You might want to add a check that the destination path is writable as well.
steadyonabix,

Thanks for the suggestion. Here is the code change:
Code:
abort()
{
	echo "$1"
	echo "Backup aborted"
	exit
}

#check destination_path
if [ ! -d "$destination_path" ]
then
	abort "Destination path does not exist: $destination_path"
fi
if [ ! -w "$destination_path" ]
then
	abort "Destination path write permission is not granted: $destination_path"
fi



---------- Post updated at 02:27 AM ---------- Previous update was at 02:08 AM ----------

Quote:
Originally Posted by ripat
If you want to do rotating (incremental) backups, rsync can create hard links on the destination server. Hard links will conserve disk space on your USB key while doing rotating backups.
Hi ripat,

The script uses the --link-dest option. So by implication, hard links are used. However, the script does not automatically delete old backups like some other rotating-backup scripts. The script was designed to backup to USB memory, so there is not reason to automate. I use it on my home PC:
  1. insert the USB memory
  2. delete old backups manually (if more space is needed)
  3. run the script manually
  4. physically remove the USB memory
  5. store the USB memory in a safe place

Last edited by wolfv; 10-21-2009 at 08:01 PM..
# 5  
Old 10-21-2009
SmilieThis is useful as I do back up my development server to a usb stick but not incrementally so I will use this.Remember you are still outputing a completed message even if it fails if you don't check the return codes though, or does rsync put up a failure message before exiting?
# 6  
Old 10-21-2009
i think this is double entry
Code:
#set destination_path (no spaces allowed)
destination_path="/media/USBdrive/backups/"
destination_path="/media_test/USB drive/backups/"

# 7  
Old 10-21-2009
Well after a little tinkering to get it to work in my korn shell it seems to create backups just fine. The code for ksh with a minimum of tweaking was: -

Code:
#! /usr/bin/ksh
######################################### rsyc2usb MANUAL #########################################
#OVER VIEW
#The rsyc2usb script uses rsync to backup files.
#Files that have changed since the previous backup are copied to destination.
#Destination can be on a local HD or USB device.
#Works on FAT and ntfs file systems (most USB flash memories use FAT32).
#Tested on rsync version 3.0.5  protocol version 30; Ubuntu 9.10. 
 
#MAKE BACKUPS
#First customize the four rsync parameters in the section "SET YOUR VARIABLES HERE" (below).
#Backups should be more than one hour apart (because FAT/ntfs uses DST and UNIX does not).
#From Terminal: sudo path/rsyc2usb
#RESTORE
#Restoring files is as simple as a cp command.  Backup files are not compressed.
#Backup folder naming convention is "backup-yymmdd-hhmmss"
#e.g. backup-090511-223344 was made on 2009 May 11 at 22:33:44.
##################################### SET YOUR VARIABLES HERE #####################################
#rsync options from man page on rsync:
#    -r, --recursive
#    -l, --preserve links 
#    -t, --preserve times
#    -x, --one-file-system    don't cross filesystem boundaries
#    -i, --itemize-changes   output a change-summary for all updates
#    -n, --dry-run           perform a trial run with no changes made
#    -v, --verbose
#    --modify-window=3601    allow times to differ by 1 hour and 1 second 
#    (FAT uses DST and UNIX does not DST change and date comparisons)
#    (FAT uses 2-second resolution rsync FAQ)  
#    (FAT does not support options -pgoD, so don't use option -a)
#set options
options="-rltx --modify-window=3601"    #for real backups
##options="-rltxin"            #for testing
#set --exclude-from file (empty string "" means no exclude-from file)
exclude_from=""
#set sources, one line per source, each source in single quotes ('), end each line with space and escape ( \)
sources="\
'/home/brad/muse/' \
"
#set destination_path (no spaces allowed)
destination_path="/media/usb/"
################################### NO NEED TO EDIT CODE BELOW ####################################
#check destination_path
if [[ ! -d "$destination_path" || ! -w "$destination_path" ]]
then
    echo "Destination path does not exist or write permission is not granted:"
    echo "    $destination_path"
    echo "Backup aborted"
  echo
    exit
fi
#find the latest_backup
latest_backup=`eval ls -drX \'$destination_path\'backup-[0-9][0-9][0-1][0-9][0-3][0-9]-[0-2][0-9][0-6][0-9][0-6][0-9] 2>/dev/null| head -1`
#set link_dest
if [ -n "$latest_backup" ]
then
    #only files that have changed since the latest_backup will be copied into destination
    link_dest="--link-dest='$latest_backup'"
else
    #should we make a full backup?
    echo "latest_backup not found.  Make a full backup? (y)"
    read answer
    if [ "$answer" = "y" ]
    then
        link_dest=""
    else
        echo "Backup aborted"
        exit
    fi
fi
#set exclude_from
if [ "$exclude_from" != "" ]
then
    exclude_from="--exclude-from='$exclude_from'"
fi
#set destination (FAT does not allow colon in filename, so use dashes)
destination=\'`date "+${destination_path}backup-%y%m%d-%H%M%S"`\'
#print wait message
echo "  rsync parameters: -"
echo
echo "    options = $options"
echo "    exclude_from = $exclude_from"
echo "    link_dest = $link_dest"
echo "    sources = $sources"
echo "    destination = $destination"
echo `date "+rsync  started at %H:%M:%S. Please do not close this window until rsync is finished."`
echo
#run rsync
EC=$(eval rsync $options $exclude_from $link_dest $sources $destination)
if [[ $EC -eq 0 ]];then
 #print done message
 echo `date "+rsync finished at %H:%M:%S."`
 echo
else
 echo "Failed - rsync failed with exit code ($EC)"
 echo
fi
exit $EC
########################################### END OF FILE ###########################################

The output was a bit messy when it did not find a backup existing: -

Code:

ls: cannot access /media/usb/backup-[0-9][0-9][0-1][0-9][0-3][0-9]-[0-2][0-9][0-6][0-9][0-6][0-9]: No such file or directory
latest_backup not found. Make a full backup? (y)

So I changed the code to dump stderr to /dev/null: -

Code:

latest_backup not found. Make a full backup? (y)
y

Output now reads: -

Code:
latest_backup not found.  Make a full backup? (y)
y
  rsync parameters: -
    options = -rltx --modify-window=3601
    exclude_from = 
    link_dest = 
    sources = '/home/brad/muse/' 
    destination = '/media/usb/backup-091021-210906'
rsync started at 21:09:06. Please do not close this window until rsync is finished.
rsync finished at 21:09:06.
poweredge:/home/brad/bin>

Thanks

Suppose I had better learn how to do a restore now....... Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Rsync Error: rsync: link_stat failed: No such file or directory (2)

I wish to copy all the files & folder under /web/Transfer_Files/data/ on mymac1 (Linux) to remote server mybank.intra.com (Solaris 10) /tmp/ location I am using Ansible tool synchronize module which triggers the unix rsync command as below:rsync --delay-updates -F --compress --archive --rsh=ssh... (2 Replies)
Discussion started by: mohtashims
2 Replies

2. Shell Programming and Scripting

Script showing incorrect output

Hello scripting geeks, I am new to scripting and facing some issues in writing the logic of the script. Request your kind help here Actually when i run a command i get o/p as below o/p : 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 these are hex values i guess...now i want to... (15 Replies)
Discussion started by: kulvant29
15 Replies

3. Red Hat

USB HDD not showing valid partition on one Solaris machine

Hi All, We have got a USB HDD with important data in it. The data has been copied to the disk(formatted ext3) via a LINUX system. The data was supposed to be copied to a second LINUX machine. Surprisingly, it doesn't show up any valid partitions in the new box. dmesg: usb1-3:... (2 Replies)
Discussion started by: Bikash Mishra
2 Replies

4. Shell Programming and Scripting

user attributes not showing when I run the script

I am new to linux/unix scripting and working in one company on linux project. I got a script that when it executes should give us the users atributes showing who is retriving data? the script should show us who are the users reriving information. I ran that script as sudo ./test4 but finding the... (0 Replies)
Discussion started by: starter2011
0 Replies

5. Ubuntu

FAQ adding rules in udev for my usb 8G to autorun rsync

Hi, I would like to ask if someone knows and show/point me or simple help me how udev rules work in lucyd, upon my quest and search all the tuts i used is based on old udev since something i read the udev change in lucyd version.. I want to accomplised when i plug my usb 8Gb device and automount... (0 Replies)
Discussion started by: jao_madn
0 Replies

6. Shell Programming and Scripting

Script for showing only selected nodes

Dear all, I am bit confused lately, I have a xmlfile here: file: book.xml <?xml version="1.0" encoding="ISO-8859-1"?> <bookstore> <book category="COOKING"> <title lang="en">Everyday Italian</title> <author>Giada De Laurentiis</author> <year>2005</year> ... (2 Replies)
Discussion started by: penyu
2 Replies

7. Shell Programming and Scripting

bash script for showing last users

Hi! I'm new in scripting and I need some help with one simple script. I have to write a program that shows in a predetermined period (using "last" command), to draw up a list of users who have used the machine during this period. Each user to indicate how many sessions it has been during this... (9 Replies)
Discussion started by: vassu
9 Replies

8. Shell Programming and Scripting

Script is showing abnormal behavior...

Hi, facing unusual problem, below are 2 same scripts, one is working and other is not. please help --- THIS SCRIPT IS WORKING FINE!!!! #! /bin/sh phone=`grep "<phone>" data.xml | sed 's:<phone>::;s:</phone>::'` echo "Phone Number is:"$phone repnum=554156 cat data.xml | sed -e... (3 Replies)
Discussion started by: Prateek007
3 Replies

9. Filesystems, Disks and Memory

Cant' mount usb drive, /dev/sda1 not showing up

Hi, I'm trying to mount a usb drive but the path /dev/sda1 does not show up under /dev when I plug in the usb device. In fact I see no differences under /dev before and after I plugin my usb drive. Any ideas why the system is not recognizing the usb drive and how to fix? This is on a... (3 Replies)
Discussion started by: orahi001
3 Replies

10. Shell Programming and Scripting

executing script not showing up in ps -efx

Hi everyone, here's my problem. I have a script that runs in a infinite loop, repeating the same action in another script every 10 minutes. We need to monitor whether or not this loop script is running and be able to kill it should the need arise. However, the name of the script does not show up... (6 Replies)
Discussion started by: herman404
6 Replies
Login or Register to Ask a Question