Parallel processing and error checking


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parallel processing and error checking
# 1  
Old 02-15-2016
Parallel processing and error checking

Hi,
I need to run multiple scripts in parallel. The scripts needs to ensure that x scripts are always running. I trigger 6 jobs in parallel, as soon as one finishes it triggers the next one, but id something fails it should stop processing any further jobs. I was able to get hold of a script which helped me in first part, but not sure how to handle the error tracking. I do not have GNU parallel Smilie

Code:
#!/bin/ksh
PROCS=6
SLEEP=20
while read LINE
do
     while true
     do
         NUM=$(jobs | wc -l)
         echo NUM=$NUM
         if [ $NUM -lt $PROCS ]
         then
             stuff "$LINE" &
             break
         else
             sleep $SLEEP
         fi
    done
done < textfile
# wait for ALL remaining processes
wait

So above ensures 6 processes are always running. I want a way to find out if one of the process is failed, then do not trigger anymore jobs.

Thanks
# 2  
Old 02-15-2016
Hi.

This is a feature of GNU parallel:
Code:
       --halt-on-error <0|1|2>
       --halt <0|1|2>
       -H <0|1|2> (-H will be retired 20120122)
                0  Do not halt if a job fails. Exit status will be the number
                   of jobs failed. This is the default.

                1  Do not start new jobs if a job fails, but complete the
                   running jobs including cleanup. The exit status will be the
                   exit status from the last failing job.

                2  Kill off all jobs immediately and exit without cleanup. The
                   exit status will be the exit status from the failing job.

See man parallel for details. If GNU parallel is not in your repository, see GNU Parallel - GNU Project - Free Software Foundation

Best wishes ... cheers, drl
This User Gave Thanks to drl For This Post:
# 3  
Old 02-16-2016
I have mentioned in my post I do not have GNU parallel

---------- Post updated at 01:20 PM ---------- Previous update was at 09:02 AM ----------

Is there a way to capture pids of each process triggered in array and check for the exit status. For non-0 exit status I will stop triggering jobs
# 4  
Old 02-16-2016
Quote:
Originally Posted by wahi80
I have mentioned in my post I do not have GNU parallel
If you are allowed to run your own perl script you are allowed to run GNU Parallel.

See: _: Excuses for not installing GNU Parallel
# 5  
Old 02-16-2016
HI,
I have worked with parallel in earlier projects. In my current project the Admins will not install GNU parallel. So is there a way out with the code I have written?

Thanks
# 6  
Old 02-16-2016
Quote:
Originally Posted by wahi80
In my current project the Admins will not install GNU parallel.
I am confused now. Did you read http://oletange.blogspot.dk/2013/04/...-parallel.html? And did you understand that you do *not* need admins to install GNU Parallel as long as you are allowed to run your own scripts (excuse #3)?

If yes: Please elaborate why none of the excuses apply to you.
# 7  
Old 02-16-2016
Run a controll shell that can write an error file:
Code:
errfile="errorfile"
rm "$errfile"
while read LINE && [ ! -f "$errfile" ]
do
...
  while [ ! -f "$errfile" ]
  do
...
    (stuff "$LINE" || echo "bad exit status" > "$errfile") &
...

This User Gave Thanks to MadeInGermany For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parallel processing

I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished? (1 Reply)
Discussion started by: garethsays
1 Replies

2. Shell Programming and Scripting

Parallel processing - continued

Hi, I am taking up the cue from where I was left in my earlier post ( link given below ) https://www.unix.com/shell-programming-scripting/231107-implement-parallel-processing.html I actually wanted to know the significance of using the Unix "wait" , which returns the control from background to... (3 Replies)
Discussion started by: kumarjt
3 Replies

3. Programming

Parallel processing pthread

Need urgent help on thread: the goal here is the separtemask will take each image and separate different contours and for each contour in the image it will call handleobject thread. So every for loop will call the handeobject thread. However, object index variable needs to be passed in each... (0 Replies)
Discussion started by: temursalin
0 Replies

4. Shell Programming and Scripting

PARALLEL PROCESSING IN PERL

HI All, I have scenerio where I need to call sub modules through for loop for (i=0; i<30 ;i++) { .. .. .. subroutine 1; subroutine 2; } I want this to be run in parallel process1 { ... ... subroutine 1; subroutine 2; (0 Replies)
Discussion started by: gvk25
0 Replies

5. Programming

Algorithms for Parallel Processing

Hey, I just wanted to know how many algorithms there are that cannot be accelerated by parallel processing. I know one such algorithm is Euclid's Algorithm (for GCF). does anyone know any other algorithms that cannot be accelerated by pp? if so please list the names and a general sentence of what... (2 Replies)
Discussion started by: azar.zorn
2 Replies

6. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

7. Shell Programming and Scripting

parallel processing

hi i am preparing a set of batches for a set of files sequentially There is a folder /xyz where all the files reside now all the files starting with 01 - will be appended for one below other to form a batch batch01 then all the files starting with 02 - will be appended for one below other to... (7 Replies)
Discussion started by: mad_man12
7 Replies

8. Shell Programming and Scripting

Need Help With Parallel Processing

Hi I am looking for some kind of feature in unix that will help me write a script that can invoke multiple processes in parallel. And make sure that the multiple parallel processes complete successfully before I proceed to the next step. Someone suggested something called timespid or... (6 Replies)
Discussion started by: imnewtothis23
6 Replies

9. Shell Programming and Scripting

parallel processing

Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M (3 Replies)
Discussion started by: audippa
3 Replies

10. UNIX for Dummies Questions & Answers

How to do parallel processing??

Hi All, I am working on solaris 8 sparc machine with 2 cpu. I am trying to run my application which generates files. I run multiple instance of the application, but the results don't seem to show as if it were runing parallely. When i run the application once it takes 12 secs to generate a... (1 Reply)
Discussion started by: zing
1 Replies
Login or Register to Ask a Question