waiting on jobs in bash, allowing limited parallel jobs at one time, and then for all to finish Post: 302536976

Sponsored Content

Top Forums Shell Programming and Scripting waiting on jobs in bash, allowing limited parallel jobs at one time, and then for all to finish Post 302536976 by srao on Wednesday 6th of July 2011 06:31:32 PM

07-06-2011

Registered User

waiting on jobs in bash, allowing limited parallel jobs at one time, and then for all to finish

Hello,

I am running GNU bash, version 3.2.39(1)-release (x86_64-pc-linux-gnu). I have a specific question pertaining to waiting on jobs run in sub-shells, based on the max number of parallel processes I want to allow, and then wait for the remaining sub-shell jobs to finish before the next step is executed in the pipeline (if I am making proper sense here)..
Essentially,my pseudo code looks like this:

Code:

MAX_PROCS=3
    for (( k = 0 ; $k < $kmerlen ; k += 1 ))
    do
    (
     ### Running a perl script here for each k (this script is a memory hog)...
    )&
    while [ $(ps -e | grep 'perlScriptAbove' | grep -v grep | wc -l) -gt ${MAX_PROCS} ] ; 
    do
             wait
    done

    done

###wait <- works fine without this wait, but I need all kmerlen jobs to finish first to proceed to the next part of the pipeline
## Run the rest of the pipeline...

The first wait statement in the while loop works fine spawning 3 jobs, but when I use the next wait statement, that property is lost, and the number of sub-shells spawned are equal to my kmerlen.

Thanks for any pointers that you can provide.

Last edited by Franklin52; 07-07-2011 at 03:26 AM.. Reason: Please use code tags for code and data samples, thank you

srao

View Public Profile for srao

Find all posts by srao

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Diff: Server n parallel jobs

Hi, Could any one please explain the difference between DataStage server edition jobs and DS parallel extender jobs...? In which scenarios or application areas do we use either of these jobs.? Regards Suresh

2. Shell Programming and Scripting

background jobs exit status and limit the number of jobs to run

i need to execute 5 jobs at a time in background and need to get the exit status of all the jobs i wrote small script below , i'm not sure this is right way to do it.any ideas please help. $cat run_job.ksh #!/usr/bin/ksh #################################### typeset -u SCHEMA_NAME=$1 ...

3. Shell Programming and Scripting

Help with script that submits jobs at a certain time.

Here's the line I type everyday. echo "submit daily 00559 00010 00011" | at 21:00 I dunno. I'm totally new to this. Basically I want this to be fool-proof for everyone else we've had people do things like: subit daily 00559 00010 00011 submit daly 00559 00010 00011 Submit daily (no...

4. Shell Programming and Scripting

Conditional execution and parallel jobs

how can i process jobs parallel with conditions below. Script1.ksh Script2.ksh Script3.ksh Script4.ksh Script5.ksh Script6.ksh Script7.ksh Script8.ksh Script9.ksh Script10.ksh After successful completion of Script1.ksh I need to run Script7.ksh. After successful...

5. Shell Programming and Scripting

Waiting for an arbitrary background process (limiting number of jobs running)

Hi, I'm trying to write a script to decompress a directory full of files. The decompression commands can run in the background, so that many can run at once. But I want to limit the number running at any one time, so that I don't overload the machine. Something like this: n=0 for i in *.gz...

6. Shell Programming and Scripting

jobs run parallel - server consumption?

I have a requirement where jobs/scripts need to be run in the background.The concern here is there are around 20 scripts which need to be run in the bg.Does running all the 20 scripts/job at the same time in bg consumes much sever-utilization. If so wot would be the efficient way to run the jobs...

7. Shell Programming and Scripting

Running jobs in parallel

I need to process 50 sqlplus scripts which are listed in a text file. I need to develop a shell script that'll read this file and run these sqlplus scripts. At any point of time, the number of sqlplus scripts running shouldn't exceed 6. If any of the sqlplus scripts completes successfully then...

8. Shell Programming and Scripting

How to WAIT for jobs in each group to finish?

I have the shell script to call a Perl routine and pass the Informatica WorkFlow name to it. Jobs in each group executes in background do not seem to wait at all. How do I make it to WAIT for the prior group to complete before execute the next group of jobs? Sample of the jobs flow: { ...

9. Shell Programming and Scripting

Shell script to run multiple jobs and it's dependent jobs

I have multiple jobs and each job dependent on other job. Each Job generates a log and If job completed successfully log file end's with JOB ENDED SUCCESSFULLY message and if it failed then it will end with JOB ENDED with FAILURE. I need an help how to start. Attaching the JOB dependency...

LEARN ABOUT PLAN9

queuedefs

queuedefs(4)							   File Formats 						      queuedefs(4)

NAME

       queuedefs - queue description file for at, batch, and cron

SYNOPSIS

       /etc/cron.d/queuedefs

DESCRIPTION

       The queuedefs file describes the characteristics of the queues managed by cron(1M). Each non-comment line in this file describes one queue.
       The format of the lines are as follows:

	      q.[njobj][nicen][nwaitw]

       The fields in this line are:

       q	The name of the queue. a is the default queue for jobs started by at(1); b is the default queue for jobs  started  by  batch  (see
		at(1)); c is the default queue for jobs run from a crontab(1) file.

       njob	The  maximum  number of jobs that can be run simultaneously in that queue; if more than njob jobs are ready to run, only the first
		njob jobs will be run, and the others will be run as jobs that are currently running terminate.  The default value is  100.

       nice	The nice(1) value to give to all jobs in that queue that are not run with a user ID of super-user.  The default value is 2.

       nwait	The number of seconds to wait before rescheduling a job that was deferred because more than njob jobs were running in  that  job's
		queue, or because the system-wide limit of jobs executing has been reached.  The default value is 60.

       Lines beginning with # are comments, and are ignored.

EXAMPLES

       Example 1: A sample file.

       #
       #
       a.4j1n
       b.2j2n90w

       This  file  specifies that the a queue, for at jobs, can have up to 4 jobs running simultaneously; those jobs will be run with a nice value
       of 1.  As no nwait value was given, if a job cannot be run because too many other jobs are running cron will wait 60 seconds before  trying
       again to run it.

       The  b  queue, for batch(1) jobs, can have up to 2 jobs running simultaneously; those jobs will be run with a nice(1) value of 2.  If a job
       cannot be run because too many other jobs are running, cron(1M) will wait 90 seconds before trying again to run it. All	other  queues  can
       have up to 100 jobs running simultaneously; they will be run with a nice value of 2, and if a job cannot be run because too many other jobs
       are running cron will wait 60 seconds before trying again to run it.

FILES

       /etc/cron.d/queuedefs	       queue description file for  at, batch, and  cron.

SEE ALSO

       at(1), crontab(1), nice(1), cron(1M)

SunOS 5.10							    1 Mar 1994							      queuedefs(4)