Waiting for an arbitrary background process (limiting number of jobs running)


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Waiting for an arbitrary background process (limiting number of jobs running)
# 8  
Old 01-29-2010
You could put the procs in background and use the shell var $! to save the pid. Not sure if it'll help or if it would be as easy to implement as it sounds. If not, then you could try perl or any other high level scripting languages.
# 9  
Old 01-29-2010
I tried seeing what I could do with Perl, and I got this, which seems to work:
Code:
#!/usr/bin/perl
sub spawn {
    $pid = fork;
    unless ($pid) {
        exec "@_";
    }
    $pid
}
@pids = ();
for $i (1..20) {
    $cmd = "sleep $i; echo \$\$";
    $pid = spawn $cmd;
    print "$pid: '$cmd'\n";
    push @pids, $pid;
    if ($i > 10) {
        $pid = shift @pids;
        waitpid $pid, 0;
        printf "Process $pid ended\n";
    }
}
while (@pids) {
    $pid = shift @pids;
    waitpid $pid, 0;
    printf "Process $pid ended\n";
}

Does that look like a reasonable approach? It seems reasonably clean - although not as nice as my original ksh attempt (which had the disadvantage that it didn't work, of course Smilie)
# 10  
Old 01-29-2010
I'm not a perl expert, but you don't seem to loop through the all pids and check the values.

You should loop through all pids with non-blocking waitpid and it the process is not running - spawn a new one in place of the old one.

It seems to me that you are now waiting for the first pid in queue and when it finishes - sprawning another one. While quite good, the situation may be that from the first 10 pids the 2-9 have ended and 1 is working very long. You will end up with only one worker running for most of the time, which I guess you've tried to avoid.
# 11  
Old 01-29-2010
Ah, thanks! Yes, you're right that is an issue.

I'll have a look at fixing this, I can probably also do it with a SIGCHLD handler I guess.

Paul.
# 12  
Old 01-29-2010
I was wrong that shell can't do this. You can easily do this in Bash.

Code:
#!/bin/bash

N_WORKERS=10
WORK_COUNTER=15
C=$N_WORKERS
PIDS[1]=0
while [ $C != 0 ]; do
        sleep $(($RANDOM / 1000 + 4)) &
        PIDS[$C]=$!
        echo "Sprawned PID: ${PIDS[$C]}"
        WORK_COUNTER=$(($WORK_COUNTER - 1))
        C=$(($C - 1))
done

WORKER_I=$N_WORKERS
WORKERS_RUNNING=$N_WORKERS
while [ $WORKERS_RUNNING != 0 ]; do
        WORKER_I=$(($WORKER_I - 1))
        [ $WORKER_I ==  0 ] && WORKER_I=N_WORKERS
        [ ${PIDS[$WORKER_I]} == 0 ] && continue
        sleep 1

        if [ -z "`ps a | awk '{print $1}' | grep ${PIDS[$WORKER_I]}`" ]; then
                echo "Finished PID: ${PIDS[$WORKER_I]}"

                if [ $WORK_COUNTER != 0 ]; then
                        sleep $(($RANDOM / 1000 + 4)) &
                        PIDS[$WORKER_I]=$!
                        echo "Sprawned PID: ${PIDS[$WORKER_I]}"
                        WORK_COUNTER=$(($WORK_COUNTER - 1))
                else
                        PIDS[$WORKER_I]=0
                        WORKERS_RUNNING=$((WORKERS_RUNNING - 1))
                fi
        fi
done

I use random time sleeps to simulate different times of processing. Everything else should be quite self-explanatory.

Last edited by dpc.ucore.info; 01-29-2010 at 10:12 AM.. Reason: fix some lame syntax
# 13  
Old 01-29-2010
Ah! It never occurred to me that I could use ps to check if workers were still running. That looks pretty good. And it looks like it will work just as well in ksh, too.

I'll give it a try. Thanks

---------- Post updated at 07:20 PM ---------- Previous update was at 04:06 PM ----------

I've now got a pretty neat solution in Perl (currently only tested on cygwin, but I see no reason it shouldn't work properly on "real" Unix). For those who might be interested, this is the final result:
Code:
#!/usr/bin/perl

use POSIX ":sys_wait_h";

%pids = ();
$npids = 0;
$MAX_CHILDREN = 10;

$SIG{CHLD} = \&REAPER;

sub REAPER {
    my $child;
    while (($child = waitpid(-1, WNOHANG)) > 0) {
        # print "$child ($pids{$child}) died!!!\n";
        delete $pids{$child};
        $npids--;
        # print "Child died ($npids still running)\n";
    }
    $SIG{CHLD} = \&REAPER;
}

sub launch {
    my ($cmd) = @_;
    if ($npids >= $MAX_CHILDREN) {
        # print "Zzzz...\n";
        sleep;
    }
    my $pid = fork;
    unless ($pid) {
        exec $cmd;
    }
    $pids{$pid} = $cmd;
    $npids++;
}

sub waitall {
    while ($npids) {
        sleep;
    }
}

for $i (<*.gz>) {
    $cmd = "gzip -d \"$i\"";
    print "Launching $cmd\n";
    launch $cmd;
}

waitall;

Thanks to all who helped me with this! It's been an interesting exercise, and I learned a lot along the way Smilie

Paul.
# 14  
Old 02-01-2010
Code:
if ($npids >= $MAX_CHILDREN) {
        # print "Zzzz...\n";
        sleep;
    }
    my $pid = fork;
    unless ($pid) {

I think you missed an 'else' there.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Running process in the background

Hi, I have this simple c program that creates duplicate process with fork(): #include <sys/types.h> main() { if (fork() == 0) while(1); else while(1); } I tried running it in the background gcc -o test first.c test & And I got this list of running process: (4 Replies)
Discussion started by: uniran
4 Replies

2. AIX

Use of screen in running background jobs and how to use this

Hello, Please advise use of screen in running jobs in nohup background and how to use this Best regards, Vishal (1 Reply)
Discussion started by: Vishal_dba
1 Replies

3. Shell Programming and Scripting

command to see process running at background

Hi , I want to see all the background process that are running in unix box machine...please guide me is there any specific command for that..since I am executing some scripts at background..!!:confused: (1 Reply)
Discussion started by: nks342
1 Replies

4. Shell Programming and Scripting

waiting on jobs in bash, allowing limited parallel jobs at one time, and then for all to finish

Hello, I am running GNU bash, version 3.2.39(1)-release (x86_64-pc-linux-gnu). I have a specific question pertaining to waiting on jobs run in sub-shells, based on the max number of parallel processes I want to allow, and then wait... (1 Reply)
Discussion started by: srao
1 Replies

5. Shell Programming and Scripting

Waiting for the background process to finish

Hi, I have on shell script which internally calls more than one scripts which run in background. These scripts cannot be modified to run in foreground. eg. myscript.sh -> bulk_launcher.sh -> acq_launcher.sh -> bulk_loader.sh I want the calling shell script myscript.sh to wait till the... (7 Replies)
Discussion started by: AB10
7 Replies

6. Shell Programming and Scripting

Waiting for a background process to finish

Hello, I am a novice shell script programmer. And facing this problem any help is appreciated. I m writing a shell script and running few commands in it background as I have to run them simultaneously. Sample code : sql_prog & sql_prog & sql_prog & echo "Process Completed" Here... (2 Replies)
Discussion started by: vineetbhati
2 Replies

7. Shell Programming and Scripting

How to find the jobs running in background and stop

Hi All, I have requirement. I am running a job every 30mins. before starting the process, i need to check the process, if the process is still running then i need not trigger the process again, if it is not running then trigger the process again. I am using cron to trigger the shell script. Can... (7 Replies)
Discussion started by: srinivas_paluku
7 Replies

8. Shell Programming and Scripting

background jobs exit status and limit the number of jobs to run

i need to execute 5 jobs at a time in background and need to get the exit status of all the jobs i wrote small script below , i'm not sure this is right way to do it.any ideas please help. $cat run_job.ksh #!/usr/bin/ksh #################################### typeset -u SCHEMA_NAME=$1 ... (1 Reply)
Discussion started by: GrepMe
1 Replies

9. Shell Programming and Scripting

Issues with exit after running jobs in background

I have the following sample script to run a script the jobs with the same priority(in this case field3) in parallel; wait for the jobs to finish and run the next set of jobs in parallel.When all the lines are read exit the script. I have the following script which is doing evrything I want... (1 Reply)
Discussion started by: hyennah
1 Replies

10. UNIX for Dummies Questions & Answers

running process in background

I'm trying to install a solaris 9 patch cluster and when I try to use & to run in background it won't allow me to enter in my sudo password so it fails the install and sudo auth. Does Solaris not have screen like linux? If & will work what am I doing wrong? sudo ./install_cluster -q & is... (3 Replies)
Discussion started by: kingdbag
3 Replies
Login or Register to Ask a Question