Is it better/possible to pause the rsyncing of a very large directory?

beginners, best practices, rsync, stopping jobs


Thread Tools Search this Thread
# 1  
Old 10-10-2012
Is it better/possible to pause the rsyncing of a very large directory?

Possibly a dumb question, but I'm deciding how I'm going to do this. I'm currently rsyncing a 25TB directory (with several layers of sub directories most of which have video files ranging from 500 megs to 4-5 gigs), from one NAS to another using rsync -av. By the time I need to act ~15TB should have been moved. I need to stop the transfer for ~12 hours. Can I just ^z the process and come back and fg it (this is running in a screen session) or should I just ^c it, and kick it back off and let rsync figure out what's already been transferred on it's own?
# 2  
Old 10-10-2012
We have done large rsync runs that we had to stop for reasons unrelated to the copying.
As simple
kill [rsync pid]

works fine. rsync figures out where to pick up.

We always run rsync as a series of at jobs because we segment the operation to speed it up. Plus we are copying from multiple sources to multiple LUNs.
# 3  
Old 10-10-2012
Awesome. And there shouldn't be a significant (given the scale of what's happening already) time lost from rsync having to rebuild the file list initially? That took quite a while the first time, and I assume now it'll have to redo that and also do the comparison against what's already been transferred.
# 4  
Old 10-11-2012
On restart: rsync takes a list of source files and looks at file times. Then compares those times to the existing destination filetimes. If you already copied 8000 files and those 8000 files match what is in the new directory, rsync can figure that out in a few minutes, tops. It then goes on to copy the files it has not already done. That takes time. rsync has to read and hash every block, send it write it and verify using the hash.

It really sounds like you need to segment your operation if you want to max I/O throughput. Of course if this is production, then you cannot eat the box alive just for rsync.

Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Find Large Files Recursively From Specific Directory aimy Shell Programming and Scripting 7 04-27-2016 06:24 AM
Pause before exit cmccabe Shell Programming and Scripting 8 11-20-2014 10:18 PM
How to copy very large directory trees siegfried Shell Programming and Scripting 3 11-12-2012 05:56 AM
Need to delete large set of files (i.e) close to 100K from a directory based on the input file prash358 Shell Programming and Scripting 36 08-25-2012 04:15 PM
On CentOS, moving space from large free directory to another pkiula Red Hat 7 05-01-2012 08:39 AM
Empty directory, large size and performance bdx Red Hat 5 01-20-2012 01:21 PM
Using find in a directory containing large number of files shoaibjameel123 Shell Programming and Scripting 6 08-08-2011 06:39 AM
script to check large directory--help anshu ranjan Shell Programming and Scripting 14 04-23-2011 04:17 PM
How to pause a shell script McLennon Shell Programming and Scripting 4 03-02-2010 11:38 AM
How to pause a while loop while reading from a file sekhar gajjala AIX 2 03-01-2010 06:38 AM
Any Way to pause/unpause system execution in HP-UX 11.11 and 11.23? deckard HP-UX 3 05-28-2009 01:43 PM
pause() problems IdleProc UNIX for Dummies Questions & Answers 1 03-17-2009 07:21 PM
pause? where art thou? 01000101 Programming 3 05-15-2006 08:42 PM
how to pause another process? daneensign UNIX for Dummies Questions & Answers 1 02-14-2006 12:27 AM