Help in using xargs


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help in using xargs
# 1  
Old 02-17-2010
Help in using xargs

Hi,
I have a requirement to RCP the files from remote server to local server.
Also the RCP has to run in parallel. However using 'xargs' retrives 2 file names during each loop. How do we restrict to only one file name using xargs and loop till remaining files.


I use the below code for that..

Code:
set -x
SOURCESERVER=ss01
SOURCEDIR="/home/oracle/senthil"
TARGETDIR="/home/oracle/senthil"
MAX_PARALLEL=2
nroffiles=`ssh $SOURCESERVER "ls $SOURCEDIR|wc -w"`
echo $nroffiles
setsize=$(( nroffiles/MAX_PARALLEL ))
echo $setsize
ssh -n $SOURCESERVER ls -1 $SOURCEDIR/ | xargs -n $setsize | while read workset; do
ssh -n $SOURCESERVER scp -rp $SOURCESERVER:$SOURCEDIR/$workset . &
done

Here's the runtime log. As you can see there are two files passed -(a&b
c&d) during each loop processing. How can we restrict this to one file at a time.

Code:
 
[oracle@ss02 senthil]$ ./pull_data.ksh
++ SOURCESERVER=ss01
++ SOURCEDIR=/home/oracle/senthil
++ TARGETDIR=/home/oracle/senthil
++ MAX_PARALLEL=2
+++ ssh ss01 'ls /home/oracle/senthil|wc -w'
++ nroffiles=5
++ echo 5
5
++ setsize=2
++ echo 2
2
++ ssh -n ss01 ls -1 /home/oracle/senthil/
++ xargs -n 2
++ read workset
++ ssh -n ss01 scp -rp ss01:/home/oracle/senthil/a b .
++ read workset
++ ssh -n ss01 scp -rp ss01:/home/oracle/senthil/c d .
++ read workset


Last edited by zaxxon; 02-17-2010 at 05:51 AM.. Reason: use code tags please, ty
# 2  
Old 02-17-2010
Code:
xargs -n1

# 3  
Old 02-18-2010
Hi,
I'm able to restrict the files one at a time using FOR loop. But still scp command is run serially for each files under $workset due to using FOR loop. I need to process scp in parallel for each workset and this parallelism should be taken dynamically based on the $setsize.

For example:

for workset 1
Code:
scp -rp ussfqaorc001:/home/oracle/senthil/a .
scp -rp ussfqaorc001:/home/oracle/senthil/b .

for workset 2
Code:
scp -rp ussfqaorc001:/home/oracle/senthil/c .
scp -rp ussfqaorc001:/home/oracle/senthil/d .

Code:
 
[oracle@usegainsqaorc001 senthil]$ ./pull_data.ksh
++ SOURCESERVER=ss01
++ SOURCEDIR=/home/oracle/senthil
++ TARGETDIR=/home/oracle/senthil
++ MAX_PARALLEL=2
+++ ssh ss01 'ls /home/oracle/senthil|wc -w'
++ nroffiles=4
++ echo 4
4
++ setsize=2
++ echo 2
2
++ ssh -n ss01 ls -1 /home/oracle/senthil/
++ xargs -n 2
++ read workset
++ for i in '$workset'
++ scp -rp ussfqaorc001:/home/oracle/senthil/a .
++ for i in '$workset'
++ scp -rp ussfqaorc001:/home/oracle/senthil/b .
++ read workset
++ for i in '$workset'
++ scp -rp ussfqaorc001:/home/oracle/senthil/c .
++ for i in '$workset'
++ scp -rp ussfqaorc001:/home/oracle/senthil/d .
++ read workset


Last edited by Scott; 02-18-2010 at 05:49 AM.. Reason: Code tags, PLEASE!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Xargs

Hi, can anyone tell me in detail ? what the following do in detail ? I am trying to get a largest number in a list Thanks Tao LARGEST=$(echo $* | xargs -n1 | sort -nr | tail -1) (3 Replies)
Discussion started by: ccp
3 Replies

2. Shell Programming and Scripting

Xargs

Hello, I need some help with xargs $ ls aaa bbb ccc ddd$ ls | xargs -I{} ls -la {} -rw-rw-r--. 1 xxx xx 0 May 30 20:04 aaa -rw-rw-r--. 1 xxx xx 0 May 30 20:04 bbb -rw-rw-r--. 1 xxx xx 0 May 30 20:04 ccc -rw-rw-r--. 1 xxx xx 0 May 30 20:04 dddit's possible to have output like this with... (3 Replies)
Discussion started by: vikus
3 Replies

3. Shell Programming and Scripting

Help with xargs

Using the bash shell I'm trying to either create a command for the command line or a script that will show netstat info for a given process name. Here is an example of what I'm trying to do:$ ps aux |grep catalina |grep -v grep | awk '{print $2}' 5132 $ netstat -nlp |grep 5132 (Not all processes... (11 Replies)
Discussion started by: axiopisty
11 Replies

4. Shell Programming and Scripting

xargs

Dear all , any suggest on xargs to combine from (1.txt and 2.txt) to output.txt ? thanks a lot. 1.txt 0123 BUM-5M BUM-5M 93490481 63839 0124 BUM-5M BUM-5M 112112 ... (3 Replies)
Discussion started by: samoptimus
3 Replies

5. Shell Programming and Scripting

Help with xargs

hi Could any one please tell me the option using which we can run multiple commands using xargs I have list of files, I want to run dos2unix and chmod at one shot on them I tried google n searched man pages but couldnt really find the solution , please help right now im doing this ls... (4 Replies)
Discussion started by: sunilmenhdiratt
4 Replies

6. Shell Programming and Scripting

Xargs and

Hello there, Let me show you a simple example of what I am trying to achieve: 1) I have an input text file with some lines: 1 a 2 b 3 c 2) And I want to run a command with these lines as arguments (+ arbitrary extra arguments). For example: $ command "1 a" "2 b" "3 c" "bye" I... (7 Replies)
Discussion started by: tokland
7 Replies

7. Shell Programming and Scripting

Using xargs

hi i just want to know that how do we use xargs command to find files which are greater than specified memory in a given directory (6 Replies)
Discussion started by: sumit the cool
6 Replies

8. UNIX for Advanced & Expert Users

xargs -P

I discovered that GNU's xargs has a -P option to allow its processes to run in parallel. Great! Is this a GNU thing, or is it supported by other platforms as well? (4 Replies)
Discussion started by: otheus
4 Replies

9. Shell Programming and Scripting

why we use xargs..

hi , can anyone help me by saying why we use xargs.. is it acing like a place holder..? thanks, Krips. (3 Replies)
Discussion started by: kripssmart
3 Replies

10. UNIX for Dummies Questions & Answers

Help with xargs

Hi there, I am trying to move around 3000 files from one directory to another. The mv command is complaining from too many arguments. I tried to use the xargs command but with no luck. Could some body provide help? Regards (4 Replies)
Discussion started by: JimJim
4 Replies
Login or Register to Ask a Question