UNIX for Dummies Questions & Answers

I've been working on some scripts in which I spawn some background processes. I'd like to be able to limit the number of processes, but have my script spawn additional processes as previous tasks finish. So, let's say I have 20 tasks to complete. Any given task could take from 1 to 10 minutes. I want to have only 3 of these background tasks going at a time, but when one completes I want another to start up until all 20 are finished. I've come up with what seems like a sloppy but effective way to do this involving "while" and "jobs", but I'd appreciate any suggestions on good ways to accomplish this general type of scripting task efficiently. How do you do this?

have them each establish any one of several specifically named lock-files...and then govern the available slots.

You could also do this quite easily with a makefile. Make has built-in mechanisms for handling multiple jobs. Here's a generic makefile that feeds ".f" files into your programs' standard input, and puts its standard output into ".g" files:

Code:

# File should be named Makefile
# Anywhere you see eight leading spaces, its actually a tab

# Tell it .f and .g are suffixes to convert
.SUFFIXES:
.SUFFIXES: .f .g

# Rule telling it how to create *.g from *.f
.f.g :
        ./myscript.sh < "$<" > "$@"

With Makefile in your current directory, when you run

Code:

make file.g

it will feed file.f into ./myscript.sh 's standard input, and create file.g from its standard output.

You can specify more than one target:

Code:

make 1.g 2.g 3.g 4.g 5.g 6.g 7.g 8.g 9.g

And you can tell it how many processes it should allow to run at once:

Code:

# Allow two simultaneous processes
make -j 2 1.g 2.g 3.g 4.g 5.g 6.g 7.g 8.g 9.g

Thanks for the replies. I've worked with lock files a little before, but never for process regulation. Good idea. The makefile option is something I've never tried, but it's really interesting. Looks like I have something new for the "to be studied" list. Again, many thanks for the suggestions.

Makefiles are more or less designed for process management. Its original, and still primary function is compiling source code but it can be applied to lots of problems as long as your program can be though of as converting one kind of data into another, repeatably(always the same input means always the same output.) Given more complicated rule relationships it can deduce the proper order to build things in, i.e. tell it how to turn .a into .b, .b into .c, and .c into .d, run 'make file.d', it will find file.a and go through all the steps to produce file.d from it.

Hi.

See also threads Parallel Execution of Programs

and

https://www.unix.com/unix-advanced-ex...2-xargs-p.html

cheers, drl

This will run 100 jobs with 10 of them running at any one time:

Code:

#!/bin/bash

set -bm

startjob() {
  if [ $count -lt $total_jobs ]; then
    yourjob.sh &
    count=$(($count+1))
  fi
}

max_parallel=10
total_jobs=100

trap 'startjob' SIGCHLD

count=0
started=0
while [ $started -lt $max_parallel ]; do
  startjob
  started=$(($started+1))
done

wait

You just have to be really careful to be sure the only child processes your script spawns after the trap command are your jobs. No "ls" commands, no "rm", no "expr", nothing. Only shell built-ins.