waiting on jobs in bash, allowing limited parallel jobs at one time, and then for all to finish Post: 302536976

Sponsored Content

Top Forums Shell Programming and Scripting waiting on jobs in bash, allowing limited parallel jobs at one time, and then for all to finish Post 302536976 by srao on Wednesday 6th of July 2011 06:31:32 PM

07-06-2011

Registered User

waiting on jobs in bash, allowing limited parallel jobs at one time, and then for all to finish

Hello,

I am running GNU bash, version 3.2.39(1)-release (x86_64-pc-linux-gnu). I have a specific question pertaining to waiting on jobs run in sub-shells, based on the max number of parallel processes I want to allow, and then wait for the remaining sub-shell jobs to finish before the next step is executed in the pipeline (if I am making proper sense here)..
Essentially,my pseudo code looks like this:

Code:

MAX_PROCS=3
    for (( k = 0 ; $k < $kmerlen ; k += 1 ))
    do
    (
     ### Running a perl script here for each k (this script is a memory hog)...
    )&
    while [ $(ps -e | grep 'perlScriptAbove' | grep -v grep | wc -l) -gt ${MAX_PROCS} ] ; 
    do
             wait
    done

    done

###wait <- works fine without this wait, but I need all kmerlen jobs to finish first to proceed to the next part of the pipeline
## Run the rest of the pipeline...

The first wait statement in the while loop works fine spawning 3 jobs, but when I use the next wait statement, that property is lost, and the number of sub-shells spawned are equal to my kmerlen.

Thanks for any pointers that you can provide.

Last edited by Franklin52; 07-07-2011 at 03:26 AM.. Reason: Please use code tags for code and data samples, thank you

srao

View Public Profile for srao

Find all posts by srao

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Diff: Server n parallel jobs

Hi, Could any one please explain the difference between DataStage server edition jobs and DS parallel extender jobs...? In which scenarios or application areas do we use either of these jobs.? Regards Suresh

2. Shell Programming and Scripting

background jobs exit status and limit the number of jobs to run

i need to execute 5 jobs at a time in background and need to get the exit status of all the jobs i wrote small script below , i'm not sure this is right way to do it.any ideas please help. $cat run_job.ksh #!/usr/bin/ksh #################################### typeset -u SCHEMA_NAME=$1 ...

3. Shell Programming and Scripting

Help with script that submits jobs at a certain time.

Here's the line I type everyday. echo "submit daily 00559 00010 00011" | at 21:00 I dunno. I'm totally new to this. Basically I want this to be fool-proof for everyone else we've had people do things like: subit daily 00559 00010 00011 submit daly 00559 00010 00011 Submit daily (no...

4. Shell Programming and Scripting

Conditional execution and parallel jobs

how can i process jobs parallel with conditions below. Script1.ksh Script2.ksh Script3.ksh Script4.ksh Script5.ksh Script6.ksh Script7.ksh Script8.ksh Script9.ksh Script10.ksh After successful completion of Script1.ksh I need to run Script7.ksh. After successful...

5. Shell Programming and Scripting

Waiting for an arbitrary background process (limiting number of jobs running)

Hi, I'm trying to write a script to decompress a directory full of files. The decompression commands can run in the background, so that many can run at once. But I want to limit the number running at any one time, so that I don't overload the machine. Something like this: n=0 for i in *.gz...

6. Shell Programming and Scripting

jobs run parallel - server consumption?

I have a requirement where jobs/scripts need to be run in the background.The concern here is there are around 20 scripts which need to be run in the bg.Does running all the 20 scripts/job at the same time in bg consumes much sever-utilization. If so wot would be the efficient way to run the jobs...

7. Shell Programming and Scripting

Running jobs in parallel

I need to process 50 sqlplus scripts which are listed in a text file. I need to develop a shell script that'll read this file and run these sqlplus scripts. At any point of time, the number of sqlplus scripts running shouldn't exceed 6. If any of the sqlplus scripts completes successfully then...

8. Shell Programming and Scripting

How to WAIT for jobs in each group to finish?

I have the shell script to call a Perl routine and pass the Informatica WorkFlow name to it. Jobs in each group executes in background do not seem to wait at all. How do I make it to WAIT for the prior group to complete before execute the next group of jobs? Sample of the jobs flow: { ...

9. Shell Programming and Scripting

Shell script to run multiple jobs and it's dependent jobs

I have multiple jobs and each job dependent on other job. Each Job generates a log and If job completed successfully log file end's with JOB ENDED SUCCESSFULLY message and if it failed then it will end with JOB ENDED with FAILURE. I need an help how to start. Attaching the JOB dependency...

LEARN ABOUT DEBIAN

condor_wait

condor_wait(1) General Commands Manual condor_wait(1)

Name
condor_wait Wait - for jobs to finish

Synopsis
condor_wait [-help -version]

condor_wait[-debug] [-wait seconds] [-num number-of-jobs] log-file[job ID]

Description
condor_waitwatches a user log file (created with the logcommand within a submit description file) and returns when one or more jobs from
the log have completed or aborted.

Because condor_waitexpects to find at least one job submitted event in the log file, at least one job must have been successfully submitted
with condor_submitbefore condor_waitis executed.

condor_waitwill wait forever for jobs to finish, unless a shorter wait time is specified.

Options
-help

Display usage information

-version

Display version information

-debug

Show extra debugging information.

-wait seconds

Wait no more than the integer number of seconds. The default is unlimited time.

-num number-of-jobs

Wait for the integer number-of-jobsjobs to end. The default is all jobs in the log file.

log file

The name of the log file to watch for information about the job.

job ID

A specific job or set of jobs to watch. If the job IDis only the job ClassAd attribute ClusterId , then condor_wait waits for all jobs
with the given ClusterId . If the job IDis a pair of the job ClassAd attributes, given by ClusterId . ProcId , then condor_wait waits
for the specific job with this job ID. If this option is not specified, all jobs that exist in the log file when condor_wait is invoked
will be watched.

General Remarks
condor_waitis an inexpensive way to test or wait for the completion of a job or a whole cluster, if you are trying to get a process outside
of Condor to synchronize with a job or set of jobs.

It can also be used to wait for the completion of a limited subset of jobs, via the -numoption.

Examples
condor_wait logfile

This command waits for all jobs that exist in logfile to complete.

condor_wait logfile 40

This command waits for all jobs that exist in logfile with a job ClassAd attribute ClusterId of 40 to complete.

condor_wait -num 2 logfile

This command waits for any two jobs that exist in logfile to complete.

condor_wait logfile 40.1

This command waits for job 40.1 that exists in logfile to complete.

condor_wait -wait 3600 logfile 40.1

This waits for job 40.1 to complete by watching logfile , but it will not wait more than one hour (3600 seconds).

Exit Status
condor_waitexits with 0 if and only if the specified job or jobs have completed or aborted. condor_waitreturns 1 if unrecoverable errors
occur, such as a missing log file, if the job does not exist in the log file, or the user-specified waiting time has expired.

Author
Condor Team, University of Wisconsin-Madison

See the Condor Version 7.8.2 Manualor http://www.condorproject.org/licensefor additional notices. condor-admin@cs.wisc.edu

September 2012 condor_wait(1)