Home Man
Today's Posts

This forum is closed for new posts. Please post beginner questions to learn unix and learn linux in the UNIX for Beginners Questions & Answers forum.

Is it better/possible to pause the rsyncing of a very large directory?

beginners, best practices, rsync, stopping jobs


Thread Tools Search this Thread
# 1  
Old 10-10-2012
Is it better/possible to pause the rsyncing of a very large directory?

Possibly a dumb question, but I'm deciding how I'm going to do this. I'm currently rsyncing a 25TB directory (with several layers of sub directories most of which have video files ranging from 500 megs to 4-5 gigs), from one NAS to another using rsync -av. By the time I need to act ~15TB should have been moved. I need to stop the transfer for ~12 hours. Can I just ^z the process and come back and fg it (this is running in a screen session) or should I just ^c it, and kick it back off and let rsync figure out what's already been transferred on it's own?
# 2  
Old 10-10-2012
We have done large rsync runs that we had to stop for reasons unrelated to the copying.
As simple
kill [rsync pid]

works fine. rsync figures out where to pick up.

We always run rsync as a series of at jobs because we segment the operation to speed it up. Plus we are copying from multiple sources to multiple LUNs.
# 3  
Old 10-10-2012
Awesome. And there shouldn't be a significant (given the scale of what's happening already) time lost from rsync having to rebuild the file list initially? That took quite a while the first time, and I assume now it'll have to redo that and also do the comparison against what's already been transferred.
# 4  
Old 10-10-2012
On restart: rsync takes a list of source files and looks at file times. Then compares those times to the existing destination filetimes. If you already copied 8000 files and those 8000 files match what is in the new directory, rsync can figure that out in a few minutes, tops. It then goes on to copy the files it has not already done. That takes time. rsync has to read and hash every block, send it write it and verify using the hash.

It really sounds like you need to segment your operation if you want to max I/O throughput. Of course if this is production, then you cannot eat the box alive just for rsync.

« Previous Thread | Next Thread »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Find Large Files Recursively From Specific Directory aimy Shell Programming and Scripting 7 04-27-2016 05:24 AM
How to copy very large directory trees siegfried Shell Programming and Scripting 3 11-12-2012 04:56 AM
Need to delete large set of files (i.e) close to 100K from a directory based on the input file prash358 Shell Programming and Scripting 36 08-25-2012 03:15 PM
On CentOS, moving space from large free directory to another pkiula Red Hat 7 05-01-2012 07:39 AM
Empty directory, large size and performance bdx Red Hat 5 01-20-2012 12:21 PM
Using find in a directory containing large number of files shoaibjameel123 Shell Programming and Scripting 6 08-08-2011 05:39 AM
script to check large directory--help anshu ranjan Shell Programming and Scripting 14 04-23-2011 03:17 PM
pause() problems IdleProc UNIX for Dummies Questions & Answers 1 03-17-2009 06:21 PM
how to pause another process? daneensign UNIX for Dummies Questions & Answers 1 02-13-2006 11:27 PM

All times are GMT -4. The time now is 07:40 AM.

Unix & Linux Forums Content Copyrightę1993-2018. All Rights Reserved.
Show Password