Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help in using xargs

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help in using xargs
# 1  
Old 02-17-2010
Help in using xargs
Hi,
I have a requirement to RCP the files from remote server to local server.
Also the RCP has to run in parallel. However using 'xargs' retrives 2 file names during each loop. How do we restrict to only one file name using xargs and loop till remaining files.


I use the below code for that..

Code:
set -x
SOURCESERVER=ss01
SOURCEDIR="/home/oracle/senthil"
TARGETDIR="/home/oracle/senthil"
MAX_PARALLEL=2
nroffiles=`ssh $SOURCESERVER "ls $SOURCEDIR|wc -w"`
echo $nroffiles
setsize=$(( nroffiles/MAX_PARALLEL ))
echo $setsize
ssh -n $SOURCESERVER ls -1 $SOURCEDIR/ | xargs -n $setsize | while read workset; do
ssh -n $SOURCESERVER scp -rp $SOURCESERVER:$SOURCEDIR/$workset . &
done

Here's the runtime log. As you can see there are two files passed -(a&b
c&d) during each loop processing. How can we restrict this to one file at a time.

Code:
 
[oracle@ss02 senthil]$ ./pull_data.ksh
++ SOURCESERVER=ss01
++ SOURCEDIR=/home/oracle/senthil
++ TARGETDIR=/home/oracle/senthil
++ MAX_PARALLEL=2
+++ ssh ss01 'ls /home/oracle/senthil|wc -w'
++ nroffiles=5
++ echo 5
5
++ setsize=2
++ echo 2
2
++ ssh -n ss01 ls -1 /home/oracle/senthil/
++ xargs -n 2
++ read workset
++ ssh -n ss01 scp -rp ss01:/home/oracle/senthil/a b .
++ read workset
++ ssh -n ss01 scp -rp ss01:/home/oracle/senthil/c d .
++ read workset

Last edited by zaxxon; 02-17-2010 at 05:51 AM.. Reason: use code tags please, ty
# 2  
Old 02-17-2010
Code:
xargs -n1

# 3  
Old 02-18-2010
Hi,
I'm able to restrict the files one at a time using FOR loop. But still scp command is run serially for each files under $workset due to using FOR loop. I need to process scp in parallel for each workset and this parallelism should be taken dynamically based on the $setsize.

For example:

for workset 1
Code:
scp -rp ussfqaorc001:/home/oracle/senthil/a .
scp -rp ussfqaorc001:/home/oracle/senthil/b .

for workset 2
Code:
scp -rp ussfqaorc001:/home/oracle/senthil/c .
scp -rp ussfqaorc001:/home/oracle/senthil/d .

Code:
 
[oracle@usegainsqaorc001 senthil]$ ./pull_data.ksh
++ SOURCESERVER=ss01
++ SOURCEDIR=/home/oracle/senthil
++ TARGETDIR=/home/oracle/senthil
++ MAX_PARALLEL=2
+++ ssh ss01 'ls /home/oracle/senthil|wc -w'
++ nroffiles=4
++ echo 4
4
++ setsize=2
++ echo 2
2
++ ssh -n ss01 ls -1 /home/oracle/senthil/
++ xargs -n 2
++ read workset
++ for i in '$workset'
++ scp -rp ussfqaorc001:/home/oracle/senthil/a .
++ for i in '$workset'
++ scp -rp ussfqaorc001:/home/oracle/senthil/b .
++ read workset
++ for i in '$workset'
++ scp -rp ussfqaorc001:/home/oracle/senthil/c .
++ for i in '$workset'
++ scp -rp ussfqaorc001:/home/oracle/senthil/d .
++ read workset

Last edited by Scott; 02-18-2010 at 05:49 AM.. Reason: Code tags, PLEASE!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Xargs

Hi, can anyone tell me in detail ? what the following do in detail ? I am trying to get a largest number in a list Thanks Tao LARGEST=$(echo $* | xargs -n1 | sort -nr | tail -1) (3 Replies)
Discussion started by: ccp
3 Replies

2. Shell Programming and Scripting

Xargs

Hello, I need some help with xargs $ ls aaa bbb ccc ddd$ ls | xargs -I{} ls -la {} -rw-rw-r--. 1 xxx xx 0 May 30 20:04 aaa -rw-rw-r--. 1 xxx xx 0 May 30 20:04 bbb -rw-rw-r--. 1 xxx xx 0 May 30 20:04 ccc -rw-rw-r--. 1 xxx xx 0 May 30 20:04 dddit's possible to have output like this with... (3 Replies)
Discussion started by: vikus
3 Replies

3. Shell Programming and Scripting

Help with xargs

Using the bash shell I'm trying to either create a command for the command line or a script that will show netstat info for a given process name. Here is an example of what I'm trying to do:$ ps aux |grep catalina |grep -v grep | awk '{print $2}' 5132 $ netstat -nlp |grep 5132 (Not all processes... (11 Replies)
Discussion started by: axiopisty
11 Replies

4. Shell Programming and Scripting

xargs

Dear all , any suggest on xargs to combine from (1.txt and 2.txt) to output.txt ? thanks a lot. 1.txt 0123 BUM-5M BUM-5M 93490481 63839 0124 BUM-5M BUM-5M 112112 ... (3 Replies)
Discussion started by: samoptimus
3 Replies

5. Shell Programming and Scripting

Help with xargs

hi Could any one please tell me the option using which we can run multiple commands using xargs I have list of files, I want to run dos2unix and chmod at one shot on them I tried google n searched man pages but couldnt really find the solution , please help right now im doing this ls... (4 Replies)
Discussion started by: sunilmenhdiratt
4 Replies

6. Shell Programming and Scripting

Xargs and

Hello there, Let me show you a simple example of what I am trying to achieve: 1) I have an input text file with some lines: 1 a 2 b 3 c 2) And I want to run a command with these lines as arguments (+ arbitrary extra arguments). For example: $ command "1 a" "2 b" "3 c" "bye" I... (7 Replies)
Discussion started by: tokland
7 Replies

7. Shell Programming and Scripting

Using xargs

hi i just want to know that how do we use xargs command to find files which are greater than specified memory in a given directory (6 Replies)
Discussion started by: sumit the cool
6 Replies

8. UNIX for Advanced & Expert Users

xargs -P

I discovered that GNU's xargs has a -P option to allow its processes to run in parallel. Great! Is this a GNU thing, or is it supported by other platforms as well? (4 Replies)
Discussion started by: otheus
4 Replies

9. Shell Programming and Scripting

why we use xargs..

hi , can anyone help me by saying why we use xargs.. is it acing like a place holder..? thanks, Krips. (3 Replies)
Discussion started by: kripssmart
3 Replies

10. UNIX for Dummies Questions & Answers

Help with xargs

Hi there, I am trying to move around 3000 files from one directory to another. The mv command is complaining from too many arguments. I tried to use the xargs command but with no luck. Could some body provide help? Regards (4 Replies)
Discussion started by: JimJim
4 Replies
Login or Register to Ask a Question