Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Using rsync with xargs
Top Forums Shell Programming and Scripting Using rsync with xargs Post 302907849 by kristinu on Wednesday 2nd of July 2014 08:02:50 AM
Old 07-02-2014
Using rsync with xargs
I want to run parallelise rsync with xargs. The plan is to
take separate directories and run rsync in parallel.

Code:
find . -mindepth 1 -maxdepth 1 print0 | xargs -n1 -P0 rsync -avz

Hoxever I am stuck on how to use rsync in this way

---------- Post updated at 04:44 AM ---------- Previous update was at 04:25 AM ----------

The following gets me into problems

Code:
find ./radon/ -mindepth 1 -maxdepth 1 -print0 | xargs --null -n1 -P0 rsync -avz '{}' rsynctest/

---------- Post updated at 04:48 AM ---------- Previous update was at 04:44 AM ----------

This seems to do better but not sure if it executes as intended

Code:
find ./radon/ -mindepth 1 -maxdepth 1 -print0 | xargs --null -n1 -P0 -I '{}' rsync -avz '{}' ./rsynctest

---------- Post updated at 06:26 AM ---------- Previous update was at 04:48 AM ----------

I did ps and found the same processes are being created

Code:
ps -elf | grep rsync
0 S chrisd    1976 32101  0  80   0 - 25327 wait   12:24 pts/7    00:00:00 xargs --null -n1 -P0 -I {} rsync -avz {} ./rsynctest
0 R chrisd    1980  1976 54  80   0 - 29388 -      12:24 pts/7    00:00:02 rsync -avz ./radon/trunk-radon ./rsynctest
0 R chrisd    1981  1976 72  80   0 - 29123 -      12:24 pts/7    00:00:02 rsync -avz ./radon/branches-radon ./rsynctest
1 S chrisd    1983  1980  0  80   0 - 29228 poll_s 12:24 pts/7    00:00:00 rsync -avz ./radon/trunk-radon ./rsynctest
1 S chrisd    1984  1981  0  80   0 - 28972 poll_s 12:24 pts/7    00:00:00 rsync -avz ./radon/branches-radon ./rsynctest
1 S chrisd    1989  1983 10  80   0 - 29321 -      12:24 pts/7    00:00:00 rsync -avz ./radon/trunk-radon ./rsynctest
1 S chrisd    1990  1984 13  80   0 - 29064 poll_s 12:24 pts/7    00:00:00 rsync -avz ./radon/branches-radon ./rsynctest
0 S chrisd    2002  1665  0  80   0 - 25801 pipe_w 12:24 pts/9    00:00:00 grep --color=auto rsync

---------- Post updated at 07:02 AM ---------- Previous update was at 06:26 AM ----------

I have written a bash script as follows, and I am still getting

Code:
ps -elf | grep rsync
0 S chrisd    2609  2579  0  80   0 - 123667 poll_s 12:49 pts/13  00:00:03 emacs /home/cdimech/Desktop/rsynctest.bash
0 R chrisd    2930     1 37  80   0 - 30150 -      12:57 pts/10   00:01:00 /usr/bin/rsync -avz /uniprg/chrisd/radon/trunk-radon /uniprg/chrisd/rsynctest
0 R chrisd    2931     1 74  80   0 - 29123 -      12:57 pts/10   00:02:00 /usr/bin/rsync -avz /uniprg/chrisd/radon/branches-radon /uniprg/chrisd/rsynctest
1 S chrisd    2936  2930  0  80   0 - 29976 poll_s 12:57 pts/10   00:00:00 /usr/bin/rsync -avz /uniprg/chrisd/radon/trunk-radon /uniprg/chrisd/rsynctest
1 S chrisd    2938  2931  0  80   0 - 28972 poll_s 12:57 pts/10   00:00:00 /usr/bin/rsync -avz /uniprg/chrisd/radon/branches-radon /uniprg/chrisd/rsynctest
1 S chrisd    2943  2936  8  80   0 - 30143 poll_s 12:57 pts/10   00:00:13 /usr/bin/rsync -avz /uniprg/chrisd/radon/trunk-radon /uniprg/chrisd/rsynctest
1 S chrisd    2949  2938 14  80   0 - 29064 poll_s 12:57 pts/10   00:00:23 /usr/bin/rsync -avz /uniprg/chrisd/radon/branches-radon /uniprg/chrisd/rsynctest
0 S chrisd    3016  1665  0  80   0 - 25802 pipe_w 13:00 pts/9    00:00:00 grep --color=auto rsync

Code:
#!/bin/bash

fls=`find /uniprg/chrisd/radon/ -mindepth 1 -maxdepth 1 | tr "\n" " "`
echo $fls
echo ""
for f in $fls; do
  #echo "$f"
  echo "rsync -avz $f /uniprg/chrisd/rsynctest &"
  /usr/bin/rsync -avz $f /uniprg/chrisd/rsynctest &
done

This User Gave Thanks to kristinu For This Post:
rbatte1
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

why we use xargs..

hi , can anyone help me by saying why we use xargs.. is it acing like a place holder..? thanks, Krips. (3 Replies)
Discussion started by: kripssmart
3 Replies

2. UNIX for Advanced & Expert Users

xargs -P

I discovered that GNU's xargs has a -P option to allow its processes to run in parallel. Great! Is this a GNU thing, or is it supported by other platforms as well? (4 Replies)
Discussion started by: otheus
4 Replies

3. Shell Programming and Scripting

Using xargs

hi i just want to know that how do we use xargs command to find files which are greater than specified memory in a given directory (6 Replies)
Discussion started by: sumit the cool
6 Replies

4. Shell Programming and Scripting

Help in using xargs

Hi, I have a requirement to RCP the files from remote server to local server. Also the RCP has to run in parallel. However using 'xargs' retrives 2 file names during each loop. How do we restrict to only one file name using xargs and loop till remaining files. I use the below code for... (2 Replies)
Discussion started by: senthil3d
2 Replies

5. Shell Programming and Scripting

Help with xargs

hi Could any one please tell me the option using which we can run multiple commands using xargs I have list of files, I want to run dos2unix and chmod at one shot on them I tried google n searched man pages but couldnt really find the solution , please help right now im doing this ls... (4 Replies)
Discussion started by: sunilmenhdiratt
4 Replies

6. Shell Programming and Scripting

xargs

Dear all , any suggest on xargs to combine from (1.txt and 2.txt) to output.txt ? thanks a lot. 1.txt 0123 BUM-5M BUM-5M 93490481 63839 0124 BUM-5M BUM-5M 112112 ... (3 Replies)
Discussion started by: samoptimus
3 Replies

7. Shell Programming and Scripting

xargs

I have dir with many files ( close to 4M ) . $ ls -la total 76392 drwxr-xr-x 10 oracle dba 512 Jun 06 14:39 . drwxr-xr-x 11 oracle dba 512 Dec 20 13:21 .. drwxr-xr-x 2 oracle dba 39074816 Jun 15 14:07 ad I am trying to delete them using... (8 Replies)
Discussion started by: talashil
8 Replies

8. UNIX for Dummies Questions & Answers

Xargs

Hi, can anyone tell me in detail ? what the following do in detail ? I am trying to get a largest number in a list Thanks Tao LARGEST=$(echo $* | xargs -n1 | sort -nr | tail -1) (3 Replies)
Discussion started by: ccp
3 Replies

9. Shell Programming and Scripting

Rsync Error: rsync: link_stat failed: No such file or directory (2)

I wish to copy all the files & folder under /web/Transfer_Files/data/ on mymac1 (Linux) to remote server mybank.intra.com (Solaris 10) /tmp/ location I am using Ansible tool synchronize module which triggers the unix rsync command as below:rsync --delay-updates -F --compress --archive --rsh=ssh... (2 Replies)
Discussion started by: mohtashims
2 Replies

10. Shell Programming and Scripting

Xargs and rsync not sending all files

Hi all, I have a script issue I can't seem to work out. In a directory I have several folders and I want to send just the sapdata1 to sapdata14 folders and their contents but not sapdataXX/.snapshot the script is: #!/bin/bash # SETUP OPTIONS export SRCDIR="/scratch/doug/test/sapdata*"... (5 Replies)
Discussion started by: DougyC
5 Replies
All times are GMT -4. The time now is 08:46 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy