Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Need help on rsync
Top Forums Shell Programming and Scripting Need help on rsync Post 302961155 by gandolf989 on Tuesday 24th of November 2015 01:28:50 PM
Old 11-24-2015
If you had several directories under the main directory about the same size, you could run several rsync commands at the same time. There is a chance that with a full 10GB connection you can get the files transferred in one night. It depends on how many files and how large each file is. IMHO, you would be better off if you have a smaller number of files that are large compared to a huge number of small files. You should use the compress flag, --compress to speed the transfer of files.

I might not even use cron for the transfer, I would probably just log in when I want the transfer to start and run the rsync with nohup and an & amersand at the end. You can then create a cron job to email the nohup.out file every hour if you want. But the transfer might finish in one night. Of course if you have millions of tiny files, it could take longer. Then just log in before the job is supposed to finish and if it is still running kill it.

But if you manually start it you can know the pid at the start, you can even script the cron job to kill the process at the beginning since you know the pid. Just don't leave the job running because the pid will get reused and you will have a job that is randomly killing any job with that pid.

If you are worried about overloading the network you can also try rsyncing to removable storage, then move the drive to the server where it needs to be. Another option would be that if you are using SAN, you can do a SAN snapshot to create another SAN volume with the same data.
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

rsync

I want to do rsync only for the difference in the last 30 days. How do I specify the "30 days" on the command line below? >rsync -avz prj# /rsource /destination Thanks for help (6 Replies)
Discussion started by: wz253
6 Replies

2. UNIX for Dummies Questions & Answers

rsync vs cp

Hi guys, I will be copying data from one FS to another. (only once) These are local Filesystems... Which tool would be more efficient to use in this case? There is a huge amount of data that needs to be copied... (1.5TB) John (1 Reply)
Discussion started by: jsy
1 Replies

3. Solaris

please help in rsync

Dear all, I m trying to rsync a file remotely but it is throwing following error. #/opt/sfw/bin/rsync -v -a -e ssh user@xx.xx.xx.xx:/export/home/naresh/utils.sh bash: rsync: command not found rsync: connection unexpectedly closed (0 bytes read so far) rsync error: error in... (10 Replies)
Discussion started by: naree
10 Replies

4. UNIX for Dummies Questions & Answers

rsync

i last night i copied a 400GB folder using rsync and ssh i did: rsync -r /mnt/500_GB ssh miguel@192.168.1.3:/mnt/1500_GB and it copied the folder fine all 400GB. The question is: If i put more files to that initial 400GB folder, which command can i run on my server for it to update the... (4 Replies)
Discussion started by: supermiguel
4 Replies

5. Shell Programming and Scripting

rsync

hi there I wonder if some-one can help. I am trying to use rsync on my mac to transfer a folder to a remote machine. I have logged into rysnc on my mac no problem and I'm trying to execute this command: rsync -a -e ssh /Users/myname/myfolder/sourcefolder/... (3 Replies)
Discussion started by: volterony
3 Replies

6. Shell Programming and Scripting

rsync

Am I correct in assuming that the following command: rsync -a /a /b does not make any changes on /a if there are files on /b that are newer (2 Replies)
Discussion started by: jgt
2 Replies

7. SuSE

rsync

I'm trying to use rsync to sync some data from my windows laptop to my linux openSUSE home sytem using cwRsync. The process has always worked until I got a new laptop and more or less at the same time decided to rebuild my linux box. on linux rsync --version rsync version 3.0.7 protocol... (2 Replies)
Discussion started by: giessenw
2 Replies

8. UNIX for Dummies Questions & Answers

rsync

Hi all, I'm new to the rsync world in hpux. Currently I have rsync going through the main public lan, but dude to traffic loads, I have been told that i need to use a dedicated rsync lan. I currently have 4 spare nics and the plan was to use one of these for the rsync, but I can't figure out... (2 Replies)
Discussion started by: linuxwanabe
2 Replies

9. Shell Programming and Scripting

Rsync Error: rsync: link_stat failed: No such file or directory (2)

I wish to copy all the files & folder under /web/Transfer_Files/data/ on mymac1 (Linux) to remote server mybank.intra.com (Solaris 10) /tmp/ location I am using Ansible tool synchronize module which triggers the unix rsync command as below:rsync --delay-updates -F --compress --archive --rsh=ssh... (2 Replies)
Discussion started by: mohtashims
2 Replies

LEARN ABOUT OPENDARWIN

queuedefs

queuedefs(4)							   File Formats 						      queuedefs(4)

NAME

       queuedefs - queue description file for at, batch, and cron

SYNOPSIS

       /etc/cron.d/queuedefs

DESCRIPTION

       The queuedefs file describes the characteristics of the queues managed by cron(1M). Each non-comment line in this file describes one queue.
       The format of the lines are as follows:

	      q.[njobj][nicen][nwaitw]

       The fields in this line are:

       q	The name of the queue. a is the default queue for jobs started by at(1); b is the default queue for jobs  started  by  batch  (see
		at(1)); c is the default queue for jobs run from a crontab(1) file.

       njob	The  maximum  number of jobs that can be run simultaneously in that queue; if more than njob jobs are ready to run, only the first
		njob jobs will be run, and the others will be run as jobs that are currently running terminate.  The default value is  100.

       nice	The nice(1) value to give to all jobs in that queue that are not run with a user ID of super-user.  The default value is 2.

       nwait	The number of seconds to wait before rescheduling a job that was deferred because more than njob jobs were running in  that  job's
		queue, or because the system-wide limit of jobs executing has been reached.  The default value is 60.

       Lines beginning with # are comments, and are ignored.

EXAMPLES

       Example 1: A sample file.

       #
       #
       a.4j1n
       b.2j2n90w

       This  file  specifies that the a queue, for at jobs, can have up to 4 jobs running simultaneously; those jobs will be run with a nice value
       of 1.  As no nwait value was given, if a job cannot be run because too many other jobs are running cron will wait 60 seconds before  trying
       again to run it.

       The  b  queue, for batch(1) jobs, can have up to 2 jobs running simultaneously; those jobs will be run with a nice(1) value of 2.  If a job
       cannot be run because too many other jobs are running, cron(1M) will wait 90 seconds before trying again to run it. All	other  queues  can
       have up to 100 jobs running simultaneously; they will be run with a nice value of 2, and if a job cannot be run because too many other jobs
       are running cron will wait 60 seconds before trying again to run it.

FILES

       /etc/cron.d/queuedefs	       queue description file for  at, batch, and  cron.

SEE ALSO

       at(1), crontab(1), nice(1), cron(1M)

SunOS 5.10							    1 Mar 1994							      queuedefs(4)
All times are GMT -4. The time now is 10:31 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy