How to achieve Parellelism for the function which has multiple cp commands


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to achieve Parellelism for the function which has multiple cp commands
# 1  
Old 02-25-2010
How to achieve Parellelism for the function which has multiple cp commands

Hi,

Here is my requirement.

Currently, there is a function which gets called in a for loop 2 times.

Code:
doCopy() {

cp  /src/a*.jar  /target/
cp /src/b*.jar /target/
cp /src/c*.jar /target

}

Since it is called sequentially from the for loop, I was asked to make parellel copy to speed up.

I've done some thing like
Code:
for i in 1, 2 do:
  doCopy &
  PIDARR[I]=&!
done
waitForAllCopiesFinish

In waitForAllCopiesFinish, I've a logic to wait till the PIDS in array are completed running and then I display "Completed"

But, my logic is not working. Whenever doCopy & is called, PID returns for some process called "bash" and there is some "cp" process parellely runs but I never see that PID returning. Not sure, how I get the PID for the "cp" process out of many "cp"s running in the system already.

Appreciate if you help me in this regard.

Thanks

Last edited by Scott; 02-25-2010 at 08:18 PM.. Reason: Please use code tags
# 2  
Old 02-25-2010
Hi.

In your example, you seem to be copying the same files twice.

When you run a just in the background, you can, with proper job control access the job using %1, %2, etc.

i.e.
Code:
$ sleep 10 &
[1]	15536
$ kill %1
[1] + Terminated               sleep 10 &

Or
Code:
$ sleep 1000 &
[1]	15561
$ wait %1
(yawn)

Or
Code:
$ sleep 10 &
[1]	15696
$ fg %1
sleep 10

etc.

So just
Code:
cp  /src/a*.jar  /target &
cp /src/b*.jar /target &
cp /src/c*.jar /target &

wait %1 %2 %3

might work for you.
# 3  
Old 02-25-2010
Hi,

Thanks for the quick response. Here is one correction to my post.

doCopy() function takes an argument which will be a target directory, i.e. copying the same files to different target each time when it is called.

So, Can you suggest me now?

Thanks in advance
# 4  
Old 02-25-2010
For example:
Code:
doCopy() {
  cp  /src/a*.jar "$1"
  cp /src/b*.jar "$1"
  cp /src/c*.jar "$1"
}

for i in 1 2; do
  doCopy /target_$i &
done

wait

# 5  
Old 02-25-2010
Hi Scott

Thank you. I will try this out and get back to you.

Regards

---------- Post updated at 08:42 PM ---------- Previous update was at 07:51 PM ----------

Hi Scott

Can I ask you one question?

I know that &! returns the last PID of the process that runs in the back ground, but with this, there are some issues like it may return some other PID which has just after your bkground job and before &!.

So. Do you think "wait %1 %2" always wait for the jobs that I run just in the background? or any chances of waiting for some other PIDs.

Thanks again,
# 6  
Old 02-26-2010
&! means nothing to me.

$! returns the PID of the last background job.

It's unlikely that %1 will return, or wait for, some PID other than that of the job to which it relates.
# 7  
Old 02-26-2010
The "jobs" command returns the list of all background jobs. See "man" for your shell. Usefully it returns nothing when all the jobs have finished.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Pass Multiple Commands and Open Multiple Xterms via PSS

Hello, I'm attempting to open multiple xterms and run a command as an SAP user via sudo using PSSH. So far, I'm able to run PSSH to a file of servers with no check for keys, open every xterm in to the servers in the file list, and SUDO to the SAP(ADM) user, but it won't do anything else... (11 Replies)
Discussion started by: icemanj
11 Replies

2. Shell Programming and Scripting

perform 3 awk commands to multiple files in multiple directories

Hi, I have a directory /home/datasets/ which contains a bunch (720) of subdirectories called hour_1/ hour_2/ etc..etc.. in each of these there is a single text file called (hour_1.txt in hour_1/ , hour_2.txt for hour_2/ etc..etc..) and i would like to do some text processing in them. Each of... (20 Replies)
Discussion started by: amarn
20 Replies

3. UNIX for Advanced & Expert Users

how to achieve this output ?

in AIX ,HP Unix and in SUN solaris, how to find the logs by date and time, I mean , Logs which have been generated or updated most recently , which might be present in folders and sub folders, I need to get the list of these Logs along with location path. Example: ======== folder... (3 Replies)
Discussion started by: sidharthmellam
3 Replies

4. Solaris

Help with executing multiple remote commands after multiple hops

Hi SSHers, I have embedded this below code in my shell script.. /usr/bin/ssh -t $USER@$SERVER1 /usr/bin/ssh $USER2@S$SERVER2 echo uptime:`/opt/OV/bin/snmpget -r 0 -t 60 $nodeName system.3.0 | cut -d: -f3-5` SSH to both these servers are public-key authenticated, so things run... (13 Replies)
Discussion started by: LinuxUser2008
13 Replies

5. Shell Programming and Scripting

Is there a simpler way to achieve this?

Hi all I have the following which is part of a larger interactive script for adding virtual hosts to Apache's configuration (it was built for non-technical administrators). I'm curious as to whether there is a simpler way of achieving the same thing. All it does is look into the... (3 Replies)
Discussion started by: mlott
3 Replies

6. Programming

ndd commands using function calls

Hi, Is there any function calls available ( for using in a C program ) to get the Ethernet Link status. ? I am looking for the status available from ndd /dev/hme link_status And how about plumbing and configuring an interface using C program ? BTW, Is all this documented... (3 Replies)
Discussion started by: shibz
3 Replies

7. Shell Programming and Scripting

How to achieve this?

hi to all, i have the input(text file) like the below... Header 1,2,3 4,5,6 7,8,9 Footer i need a output(text file) like the below... Header,1,2,3,Footer Header,4,5,6,Footer Header,7,8,9,Footer please help me to find out? (2 Replies)
Discussion started by: aaha_naga
2 Replies

8. Programming

How to achieve Serialization in Unix C, C++

How to achieve SERIALIZATION in Unix , C, C++ Write Objects directly to disk and read back ? (1 Reply)
Discussion started by: Sivaswami
1 Replies

9. UNIX for Dummies Questions & Answers

Do i need a script to achieve this?

Hi all, i have a following folder call 'zz'. This 'zz' folder is found in the following directories. I would like to delete the zz in the following directories /aa/zz /aa/bb/cc/zz /aa/bb/cc/dd/zz /aa/bb/cc/dd/ee/zz and keep the zz folder in the following directories /WW/zz ... (1 Reply)
Discussion started by: new2ss
1 Replies

10. Shell Programming and Scripting

is there a way to achieve this with compress?

I'm trying to figure out how I can get an equivalent output to a text file such as if I use gunzip -l filename in that gunzip will output the previous file size and the compressed size and the ratio... is there a way to achieve this with compress and zip? i am very new at shell... (3 Replies)
Discussion started by: nortypig
3 Replies
Login or Register to Ask a Question