Shell Programming and Scripting

I am trying to implement the below using Ksh script on a Lx machine.

There is a file(input_file) with 100K records. For each of these records, certain script(process_rec) needs to be called with the record as input. Sequential processing is time-consuming and parallel processing would eat up resources.
I would like to ensure that at any given time there are exactly 10 processes running. The problem with using wait command is that it waits until all child processes are over.

For example - If first 10 records are being processed by 10 child processes, I would like that if any 1 of the child process completes, a new child process gets started processing the 11th record from the input_file.

How could this be achieved?

You can query the number of the shell's background jobs with the jobs command.

Code:

#!/bin/ksh
maxjobs=3
while read line <&3
do
  while [ `jobs | wc -l` -ge $maxjobs ]
  do
    sleep 5;
  done
  # now start the background job with the information from $line
  # for example, simply put them as argument:
  process_rec $line &
done 3< records.txt

About a week ago, i had written something like that, thanks to community here, working
edit-comment /* feeling utterly 'overdressed' compared to MadeInGermany's solution */

You would start it like: tui-psm -l 10 $(cat list.txt) or tui-psm -l 10 *
Assuming the background jobs are scripts that is.

It is now part of TUI, and would look like:




See more in post: https://www.unix.com/302921589-post18.html which is the 'last' post of [BASH] Script to manage background scripts (running, finished, exit code)

Hope this helps and is not seen as illegal advertisement

EDIT:
It now is also part of https://www.unix.com/shell-programmin...k-scripts.html