Need Help With Parallel Processing


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need Help With Parallel Processing
# 1  
Old 08-05-2008
Need Help With Parallel Processing

Hi

I am looking for some kind of feature in unix that will help me write a script that can invoke multiple processes in parallel. And make sure that the multiple parallel processes complete successfully before I proceed to the next step.

Someone suggested something called timespid or timepid or something that sounds like that, and recommended that it might help me accomplish what Im looking for.

Can someone help me with this?

Thanks!
A Beginner
# 2  
Old 08-05-2008
If you just want to start processes you could use the background process mechanism:

command &

will start the process "command" but will continue processing of the program issuing it after it is started. You can find out the background jobs PID with the "jobs" internal shell command and look in a loop if all started background processes have ended or not.

The following is a sketch - note, that the output of "jobs" is a little different from one system to another. This was written on AIX 5.3, maybe you need to change the regexps a bit to make them fit.

Code:
#! /bin/ksh
typeset -i iPID=0

some_command &
iPID=$(jobs -l | sed -n 's/^\[[0-9]*\]  *+  *\([^ ]*\) .*/\1/p')

while [ $(ps -fe | grep -c $iPID) -gt 1 ] ; do
     # wait for process to end - process is still running
     sleep 1
done

print - "process has finished"

exit 0

If you need to exchange messages with the subprocesses you will have to use the Korn Shells Coprocess facility: a coprocess is a background process just like above, but one that can communicate with the main process through two file descriptors: &4 and &5. There are special commands ("read -p" and "print -p") to communicate between the processes. You might want to read some documentation first, because building such a coprocess is a bit complicated at the beginning.

I hope this helps.

bakunin
# 3  
Old 08-05-2008
It is straightforward to run multiple processe in parallel, just run them in the background using "&".

Code:
# invoke job 1 in the background
job1 & 
# invoke job 2 in the background
job2 &
# wait for background processes to complete
wait
# any additional processing

You can also wait for a specific process to finish before continuing, see the wait command on your shell's man page.
# 4  
Old 08-05-2008
bakunin, a much easier method: iPID=$(jobs -p).

Also easier: while ps -p $iPID > /dev/null.
# 5  
Old 08-06-2008
Also just get the PID by iPID=$! after starting the background process, and wait $iPID to wait on it without the need for a wasteful loop, with or without sleeps.
# 6  
Old 08-06-2008
Thanks for the quick responses

Smilie I will try these suggestions.

Is anyone aware of anything called timepid or timespid or something like that? I tried Google, but no useful results.
# 7  
Old 08-07-2008
I sort of assumed you meant waitpid
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parallel processing

I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished? (1 Reply)
Discussion started by: garethsays
1 Replies

2. Shell Programming and Scripting

Parallel processing - continued

Hi, I am taking up the cue from where I was left in my earlier post ( link given below ) https://www.unix.com/shell-programming-scripting/231107-implement-parallel-processing.html I actually wanted to know the significance of using the Unix "wait" , which returns the control from background to... (3 Replies)
Discussion started by: kumarjt
3 Replies

3. Shell Programming and Scripting

Implement parallel processing

Unix OS : Linux 2.6x Shell type : Korn Hi all , This is a requirement to incorporate parallel processing of a Unix code . I have two pieces of unix code , one of which will act as a parent process . This script will invoke multiple ( say four ) instances of the second script at one go... (13 Replies)
Discussion started by: kumarjt
13 Replies

4. Shell Programming and Scripting

Parallel processing in bash?

Hi Say I am interested in processing a big data set over shell, and each process individually takes a long time, but many such processes can be pipe-lined, is there a way to do this automatically or efficiently in shell? For example consider pinging a list addresses upto 5 times each. Of... (5 Replies)
Discussion started by: jamie_123
5 Replies

5. Shell Programming and Scripting

PARALLEL PROCESSING IN PERL

HI All, I have scenerio where I need to call sub modules through for loop for (i=0; i<30 ;i++) { .. .. .. subroutine 1; subroutine 2; } I want this to be run in parallel process1 { ... ... subroutine 1; subroutine 2; (0 Replies)
Discussion started by: gvk25
0 Replies

6. Shell Programming and Scripting

script parallel processing

How to write script which run multiple scripts parllely, i have script called A,which has to execute B,C,D,E scripts parllely.. (2 Replies)
Discussion started by: machpee
2 Replies

7. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

8. Shell Programming and Scripting

parallel processing

hi i am preparing a set of batches for a set of files sequentially There is a folder /xyz where all the files reside now all the files starting with 01 - will be appended for one below other to form a batch batch01 then all the files starting with 02 - will be appended for one below other to... (7 Replies)
Discussion started by: mad_man12
7 Replies

9. Shell Programming and Scripting

parallel processing

Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M (3 Replies)
Discussion started by: audippa
3 Replies

10. UNIX for Dummies Questions & Answers

How to do parallel processing??

Hi All, I am working on solaris 8 sparc machine with 2 cpu. I am trying to run my application which generates files. I run multiple instance of the application, but the results don't seem to show as if it were runing parallely. When i run the application once it takes 12 secs to generate a... (1 Reply)
Discussion started by: zing
1 Replies
Login or Register to Ask a Question