Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Using rsync with xargs

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using rsync with xargs
# 1  
Old 07-02-2014
Using rsync with xargs
I want to run parallelise rsync with xargs. The plan is to
take separate directories and run rsync in parallel.

Code:
find . -mindepth 1 -maxdepth 1 print0 | xargs -n1 -P0 rsync -avz

Hoxever I am stuck on how to use rsync in this way

---------- Post updated at 04:44 AM ---------- Previous update was at 04:25 AM ----------

The following gets me into problems

Code:
find ./radon/ -mindepth 1 -maxdepth 1 -print0 | xargs --null -n1 -P0 rsync -avz '{}' rsynctest/

---------- Post updated at 04:48 AM ---------- Previous update was at 04:44 AM ----------

This seems to do better but not sure if it executes as intended

Code:
find ./radon/ -mindepth 1 -maxdepth 1 -print0 | xargs --null -n1 -P0 -I '{}' rsync -avz '{}' ./rsynctest

---------- Post updated at 06:26 AM ---------- Previous update was at 04:48 AM ----------

I did ps and found the same processes are being created

Code:
ps -elf | grep rsync
0 S chrisd    1976 32101  0  80   0 - 25327 wait   12:24 pts/7    00:00:00 xargs --null -n1 -P0 -I {} rsync -avz {} ./rsynctest
0 R chrisd    1980  1976 54  80   0 - 29388 -      12:24 pts/7    00:00:02 rsync -avz ./radon/trunk-radon ./rsynctest
0 R chrisd    1981  1976 72  80   0 - 29123 -      12:24 pts/7    00:00:02 rsync -avz ./radon/branches-radon ./rsynctest
1 S chrisd    1983  1980  0  80   0 - 29228 poll_s 12:24 pts/7    00:00:00 rsync -avz ./radon/trunk-radon ./rsynctest
1 S chrisd    1984  1981  0  80   0 - 28972 poll_s 12:24 pts/7    00:00:00 rsync -avz ./radon/branches-radon ./rsynctest
1 S chrisd    1989  1983 10  80   0 - 29321 -      12:24 pts/7    00:00:00 rsync -avz ./radon/trunk-radon ./rsynctest
1 S chrisd    1990  1984 13  80   0 - 29064 poll_s 12:24 pts/7    00:00:00 rsync -avz ./radon/branches-radon ./rsynctest
0 S chrisd    2002  1665  0  80   0 - 25801 pipe_w 12:24 pts/9    00:00:00 grep --color=auto rsync

---------- Post updated at 07:02 AM ---------- Previous update was at 06:26 AM ----------

I have written a bash script as follows, and I am still getting

Code:
ps -elf | grep rsync
0 S chrisd    2609  2579  0  80   0 - 123667 poll_s 12:49 pts/13  00:00:03 emacs /home/cdimech/Desktop/rsynctest.bash
0 R chrisd    2930     1 37  80   0 - 30150 -      12:57 pts/10   00:01:00 /usr/bin/rsync -avz /uniprg/chrisd/radon/trunk-radon /uniprg/chrisd/rsynctest
0 R chrisd    2931     1 74  80   0 - 29123 -      12:57 pts/10   00:02:00 /usr/bin/rsync -avz /uniprg/chrisd/radon/branches-radon /uniprg/chrisd/rsynctest
1 S chrisd    2936  2930  0  80   0 - 29976 poll_s 12:57 pts/10   00:00:00 /usr/bin/rsync -avz /uniprg/chrisd/radon/trunk-radon /uniprg/chrisd/rsynctest
1 S chrisd    2938  2931  0  80   0 - 28972 poll_s 12:57 pts/10   00:00:00 /usr/bin/rsync -avz /uniprg/chrisd/radon/branches-radon /uniprg/chrisd/rsynctest
1 S chrisd    2943  2936  8  80   0 - 30143 poll_s 12:57 pts/10   00:00:13 /usr/bin/rsync -avz /uniprg/chrisd/radon/trunk-radon /uniprg/chrisd/rsynctest
1 S chrisd    2949  2938 14  80   0 - 29064 poll_s 12:57 pts/10   00:00:23 /usr/bin/rsync -avz /uniprg/chrisd/radon/branches-radon /uniprg/chrisd/rsynctest
0 S chrisd    3016  1665  0  80   0 - 25802 pipe_w 13:00 pts/9    00:00:00 grep --color=auto rsync

Code:
#!/bin/bash

fls=`find /uniprg/chrisd/radon/ -mindepth 1 -maxdepth 1 | tr "\n" " "`
echo $fls
echo ""
for f in $fls; do
  #echo "$f"
  echo "rsync -avz $f /uniprg/chrisd/rsynctest &"
  /usr/bin/rsync -avz $f /uniprg/chrisd/rsynctest &
done

This User Gave Thanks to kristinu For This Post:
rbatte1
# 2  
Old 07-02-2014
Consider this guy's approach.

Unix Admin Guide: Solaris: rsync in parallel

xargs is not needed.
# 3  
Old 07-02-2014
I can just use the bash script then.

---------- Post updated at 07:51 AM ---------- Previous update was at 07:49 AM ----------

I also figured that rsync does create three processes by design.
# 4  
Old 07-02-2014
Can you use --files-from=FILE read list of source-file names from FILE

Generate a FILE with find results
Split the file using split command on n desired parts naming them from 0 to n, and putting total count into variable

For total count run iterate rsync --files-from=FILEn ... &

Of course, if you want to ran batches 5 x 5 FILES or you have alot of rsyncs which can hit the machine hard, you will have to include wait command. There are some quality examples of shell parallelism on this forums.

If you want to run like 5 rsyncs at the same time (single time script is ran), you will not need the extra overhead in coding.

Hope that helps
Regards
Peasant
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Xargs and rsync not sending all files

Hi all, I have a script issue I can't seem to work out. In a directory I have several folders and I want to send just the sapdata1 to sapdata14 folders and their contents but not sapdataXX/.snapshot the script is: #!/bin/bash # SETUP OPTIONS export SRCDIR="/scratch/doug/test/sapdata*"... (5 Replies)
Discussion started by: DougyC
5 Replies

2. Shell Programming and Scripting

Rsync Error: rsync: link_stat failed: No such file or directory (2)

I wish to copy all the files & folder under /web/Transfer_Files/data/ on mymac1 (Linux) to remote server mybank.intra.com (Solaris 10) /tmp/ location I am using Ansible tool synchronize module which triggers the unix rsync command as below:rsync --delay-updates -F --compress --archive --rsh=ssh... (2 Replies)
Discussion started by: mohtashims
2 Replies

3. UNIX for Dummies Questions & Answers

Xargs

Hi, can anyone tell me in detail ? what the following do in detail ? I am trying to get a largest number in a list Thanks Tao LARGEST=$(echo $* | xargs -n1 | sort -nr | tail -1) (3 Replies)
Discussion started by: ccp
3 Replies

4. Shell Programming and Scripting

xargs

I have dir with many files ( close to 4M ) . $ ls -la total 76392 drwxr-xr-x 10 oracle dba 512 Jun 06 14:39 . drwxr-xr-x 11 oracle dba 512 Dec 20 13:21 .. drwxr-xr-x 2 oracle dba 39074816 Jun 15 14:07 ad I am trying to delete them using... (8 Replies)
Discussion started by: talashil
8 Replies

5. Shell Programming and Scripting

xargs

Dear all , any suggest on xargs to combine from (1.txt and 2.txt) to output.txt ? thanks a lot. 1.txt 0123 BUM-5M BUM-5M 93490481 63839 0124 BUM-5M BUM-5M 112112 ... (3 Replies)
Discussion started by: samoptimus
3 Replies

6. Shell Programming and Scripting

Help with xargs

hi Could any one please tell me the option using which we can run multiple commands using xargs I have list of files, I want to run dos2unix and chmod at one shot on them I tried google n searched man pages but couldnt really find the solution , please help right now im doing this ls... (4 Replies)
Discussion started by: sunilmenhdiratt
4 Replies

7. Shell Programming and Scripting

Help in using xargs

Hi, I have a requirement to RCP the files from remote server to local server. Also the RCP has to run in parallel. However using 'xargs' retrives 2 file names during each loop. How do we restrict to only one file name using xargs and loop till remaining files. I use the below code for... (2 Replies)
Discussion started by: senthil3d
2 Replies

8. Shell Programming and Scripting

Using xargs

hi i just want to know that how do we use xargs command to find files which are greater than specified memory in a given directory (6 Replies)
Discussion started by: sumit the cool
6 Replies

9. UNIX for Advanced & Expert Users

xargs -P

I discovered that GNU's xargs has a -P option to allow its processes to run in parallel. Great! Is this a GNU thing, or is it supported by other platforms as well? (4 Replies)
Discussion started by: otheus
4 Replies

10. Shell Programming and Scripting

why we use xargs..

hi , can anyone help me by saying why we use xargs.. is it acing like a place holder..? thanks, Krips. (3 Replies)
Discussion started by: kripssmart
3 Replies
Login or Register to Ask a Question