Visit Our UNIX and Linux User Community


Controlling the Number of Child processes


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Controlling the Number of Child processes
# 1  
Old 10-28-2014
Controlling the Number of Child processes

I am trying to implement the below using Ksh script on a Lx machine.

There is a file(input_file) with 100K records. For each of these records, certain script(process_rec) needs to be called with the record as input. Sequential processing is time-consuming and parallel processing would eat up resources.
I would like to ensure that at any given time there are exactly 10 processes running. The problem with using wait command is that it waits until all child processes are over.

For example - If first 10 records are being processed by 10 child processes, I would like that if any 1 of the child process completes, a new child process gets started processing the 11th record from the input_file.

How could this be achieved?
# 2  
Old 10-28-2014
You can query the number of the shell's background jobs with the jobs command.
Code:
#!/bin/ksh
maxjobs=3
while read line <&3
do
  while [ `jobs | wc -l` -ge $maxjobs ]
  do
    sleep 5;
  done
  # now start the background job with the information from $line
  # for example, simply put them as argument:
  process_rec $line &
done 3< records.txt

These 3 Users Gave Thanks to MadeInGermany For This Post:
# 3  
Old 10-28-2014
About a week ago, i had written something like that, thanks to community here, working Smilie
edit-comment /* feeling utterly 'overdressed' compared to MadeInGermany's solution */

You would start it like: tui-psm -l 10 $(cat list.txt) or tui-psm -l 10 *
Assuming the background jobs are scripts that is.

It is now part of TUI, and would look like:

Image


See more in post: https://www.unix.com/302921589-post18.html which is the 'last' post of [BASH] Script to manage background scripts (running, finished, exit code)

Hope this helps and is not seen as illegal advertisement Smilie

EDIT:
It now is also part of https://www.unix.com/shell-programmin...k-scripts.html

Last edited by sea; 01-07-2015 at 01:48 PM..

Previous Thread | Next Thread
Test Your Knowledge in Computers #812
Difficulty: Medium
In CSS, the rgba() function define colors using the red-green-blue-accent (RGBA) model.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Get all child processes of a process

is there a universal way of getting the children of a particular process? i'm looking for a solution that works across different OSes...linux, aix, sunos, hpux. i did a search online and i kept finding answers that were specific to Linux..i.e. pstree. i want to be able to specify a process... (2 Replies)
Discussion started by: SkySmart
2 Replies

2. Windows & DOS: Issues & Discussions

Controlling AIX processes remotely using a NET app on a Windows server?

I have a .NET application that remotely starts, stops, and gets status of Windows services and scheduled tasks. I would like to add the capability of starting, stopping, and getting status of remote AIX applications also. Based on some preliminary research, one option may be to use 3rd party .NET... (0 Replies)
Discussion started by: auser1
0 Replies

3. Programming

Controlling a child's stdin/stdout (not working with scp)

All, Ok...so I know I *should* be able to control a process's stdin and stdout from the parent by creating pipes and then dup'ing them in the child. And, this works with all "normal" programs that I've tried. Unfortunately, I want to intercept the stdin/out of the scp application and it seems... (9 Replies)
Discussion started by: DreamWarrior
9 Replies

4. Programming

How to limit the number of child processes

I need a mechanism to fork child processes and all child processes should connect to a server.but the number of child processes should be limited(for ex:50) Here's my pseudo, but I cant figure out how to limit the child process number. Should I use a semaphore? or what? for(;;)... (3 Replies)
Discussion started by: xyzt
3 Replies

5. UNIX for Advanced & Expert Users

killing all child processes

Hi, Is there a way I can kill all the child processes of a process, given its process id. Many thanks in advance. J. (1 Reply)
Discussion started by: superuser84
1 Replies

6. Programming

fork() and child processes

Hello, How many child processes are actually created when running this code ? #include <signal.h> #include <stdio.h> int main () { int i ; setpgrp () ; for (i = 0; i < 10; i++) { if (fork () == 0) { if ( i & 1 ) setpgrp () ; printf ("Child id: %2d, group: %2d\n", getpid(),... (0 Replies)
Discussion started by: green_dot
0 Replies

7. Shell Programming and Scripting

Parent/Child Processes

Hello. I have a global function name func1() that I am sourcing in from script A. I call the function from script B. Is there a way to find out which script called func1() dynamically so that the func1() can report it in the event there are errors? Thanks (2 Replies)
Discussion started by: yoi2hot4ya
2 Replies

8. Programming

Controlling child processes

Hello all, I am trying to create n child processes and control them from a parent process; say make child 3 print its pid and then child 5 do the same and some other stuff. Is there a way to accomplishing this after all the child processes are created via a call to fork(). Thank you, FG (23 Replies)
Discussion started by: forumGuy
23 Replies

9. UNIX for Dummies Questions & Answers

what are parent and child processes all about?

I don't follow what these are... this is what my text says... "When a process is started, a duplicate of that process is created. This new process is called the child and the process that created it is called the parent. The child process then replaces the copy for the code the parent... (1 Reply)
Discussion started by: xyyz
1 Replies

10. UNIX for Dummies Questions & Answers

Controlling processes knowing the PID's

Dear all, suppose that I start a process (named "father"). "father" starts in turns a process called "child" with an execv call (after a fork). In this way "father" will be notified if "chlid" crashes (SIGCHILD mechanism). The problem is: if "father" crashes, how can I do to be recreate a... (1 Reply)
Discussion started by: npalmentieri
1 Replies

Featured Tech Videos