Is it better/possible to pause the rsyncing of a very large directory? | Unix Linux Forums | UNIX for Dummies Questions & Answers

  Go Back    


UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !!

Is it better/possible to pause the rsyncing of a very large directory?

UNIX for Dummies Questions & Answers


Tags
best practices, rsync, stopping jobs

Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 10-10-2012
DeCoTwc DeCoTwc is offline
Registered User
 
Join Date: Mar 2008
Last Activity: 22 June 2014, 2:17 AM EDT
Location: NYC
Posts: 162
Thanks: 7
Thanked 8 Times in 8 Posts
Is it better/possible to pause the rsyncing of a very large directory?

Possibly a dumb question, but I'm deciding how I'm going to do this. I'm currently rsyncing a 25TB directory (with several layers of sub directories most of which have video files ranging from 500 megs to 4-5 gigs), from one NAS to another using rsync -av. By the time I need to act ~15TB should have been moved. I need to stop the transfer for ~12 hours. Can I just ^z the process and come back and fg it (this is running in a screen session) or should I just ^c it, and kick it back off and let rsync figure out what's already been transferred on it's own?
Sponsored Links
    #2  
Old 10-10-2012
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 27 July 2014, 8:53 PM EDT
Location: NM
Posts: 10,155
Thanks: 269
Thanked 781 Times in 731 Posts
We have done large rsync runs that we had to stop for reasons unrelated to the copying.
As simple
Code:
kill [rsync pid]

works fine. rsync figures out where to pick up.

We always run rsync as a series of at jobs because we segment the operation to speed it up. Plus we are copying from multiple sources to multiple LUNs.
Sponsored Links
    #3  
Old 10-10-2012
DeCoTwc DeCoTwc is offline
Registered User
 
Join Date: Mar 2008
Last Activity: 22 June 2014, 2:17 AM EDT
Location: NYC
Posts: 162
Thanks: 7
Thanked 8 Times in 8 Posts
Awesome. And there shouldn't be a significant (given the scale of what's happening already) time lost from rsync having to rebuild the file list initially? That took quite a while the first time, and I assume now it'll have to redo that and also do the comparison against what's already been transferred.
    #4  
Old 10-10-2012
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 27 July 2014, 8:53 PM EDT
Location: NM
Posts: 10,155
Thanks: 269
Thanked 781 Times in 731 Posts
On restart: rsync takes a list of source files and looks at file times. Then compares those times to the existing destination filetimes. If you already copied 8000 files and those 8000 files match what is in the new directory, rsync can figure that out in a few minutes, tops. It then goes on to copy the files it has not already done. That takes time. rsync has to read and hash every block, send it write it and verify using the hash.


It really sounds like you need to segment your operation if you want to max I/O throughput. Of course if this is production, then you cannot eat the box alive just for rsync.
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Empty directory, large size and performance bdx Red Hat 5 01-20-2012 12:21 PM
Using find in a directory containing large number of files shoaibjameel123 Shell Programming and Scripting 6 08-08-2011 05:39 AM
script to check large directory--help anshu ranjan Shell Programming and Scripting 14 04-23-2011 03:17 PM
Slow pattern matching ls or rm in a large directory JT-KGY UNIX for Dummies Questions & Answers 11 01-22-2008 07:56 PM



All times are GMT -4. The time now is 08:20 AM.