I think I wrote something similar to this recently which is almost like a map-reduce version for a single host which is more robust for a shared host execution which takes care of cpu and available memory hogging.
Interfacing between the master and slave workers is not that great, as I used lock files to communicate between master and slave. Before the slave is about to start its destined work, it creates a lock file which is accessible by the master and it checks periodically to see if the lock file is still there. Once the worker completes the work, the lock file would be removed which indicates that worker is done with work, based on current load on the system a new process can or cannot be spawned.
Though I said, lock interfacing isn't that great, it works very well as the master is very pessimist and takes the veto to kill the worker if it detects that the worker has gone stale, not needed any more, parallel execution of work by more than 1 worker.
Is that the same problem that you are trying to solve?
---------- Post updated at 06:58 PM ---------- Previous update was at 06:53 PM ----------
Quote:
Originally Posted by pbillast
In these kind of situations you'll find the environment variable $! (pid of backgound process) and the manpage for the command wait very useful.
I recently made something to launch sql statements and gzip on the output files with a maximum number of subprocesses (I chose this maximum in function of the numbers of cpu cores)
basically in the script you start a subprocess
Code:
ClientDataEncrypt <args> &
BPID=$!
and thus keep track of the backgound pids (by storing them in an array).
from there you can limit the number of subprocesses.
(P.S. on Solaris 10 pgrep is very useful in these situations, especially if you want to be able to launch ClientDataEncrypt both manually and via the watcher and want the watcher daemon to know about it)
I hope this helps
Is the reason for storing PIDs to check periodically whether the sub process has completed execution or not? If so, there is a potential problem in it and yes it will happen in ultra busy nodes.
For ex: Lets take the pid stored is 'p1', by the time watcher scans the pid list, 'p1' can complete its processing, die and a new process with 'p1' can be spawned again, watcher unaware of this will think that 'p1' is still alive which is not true.
Basically, problem is due to expanded scope, if there is a boundary drawn using process group, this problem can be avoided. Even if 'p1' is respawned as part of some other process, it wont be part of the process group boundary that we are checking.
How do you capture the return code from a background process?
I am dumping data to a fifo and then processing it in a c program.
I need to know that the sql finished successfully to ensure no missing data. Thanks.
ex.
sqlplus user/password < get_data.sql > data_fifo.txt &
bin/process_data... (2 Replies)
Hello Friends,
sorry, i am not very familiar with Unix programming. Could you please help me on this?
We have to start different components from a startup script.
each components are started as below in the background in a startprocess function
$nohup $file $args >>$logFile 2>&1 &
... (0 Replies)
Hello Friends,
sorry, i am not very familiar with Unix programming. Could you please help me on this?
We have to start different components from a startup script.
each components are started as below in the background in a startprocess function
$nohup $file $args >>$logFile 2>&1 &
... (1 Reply)
I have script 3 scripts
1 parent
2 children
child1
child2
In the code below the 2 child processes fire almost Instantaneously in the background, Is that possible to know the status of pass/fail of each process "as it happens" ?
In the present scenario although Child2... (5 Replies)
I have script 3 scripts 1 parent (p1) and 2 children child1 and child2
I have script 3 scripts
1 parent
2 children
child1
child2
In the code below the 2 child processes fire almost Instantaneously in
the background, Is that possible to know the status of pass/fail of each
process... (12 Replies)
Ok guys so I have my first dummy shell almost done except for one tiny part: I do not know how to run a process in the background, from the code!
I already know how to do that in a normal shell:
$ program &
However, no clue when it comes to how to program that thing. :eek:
A very... (2 Replies)
Hi,
I have a schell script parent.ksh from which I am calling three background processes a.ksh,b.ksh and c.ksh. Once these three processes completes the next step in parent.ksh should execute. How to achieve this?
Please help me....
Thanks... (1 Reply)
Hi All,
Iam trying to get a file processed and some lines have spaces...the below is not working
Want to remove empty line
Want to remove lines that start with #
Avoid line with substring WHOA
When trying to get the substring from the var also Iam having trouble
file is like VAR=VALUE,... (13 Replies)
Hello
I have a file which has around 120 lines of commands.
I am trying to write a shell script like which reads the 'command' file and executes line by line with some additional (common argument) with maximum 6 commands active at a time. Each of these commands when executed takes time... (5 Replies)
Discussion started by: JackyShane_36
5 Replies
LEARN ABOUT OSF1
voliod
voliod(8) System Manager's Manual voliod(8)NAME
voliod - Start, stop, and report on Logical Storage Manager kernel daemons
SYNOPSIS
/sbin/voliod
/sbin/voliod [-f] set count
OPTIONS
The following option is used by voliod: Force the kill of the last I/O daemon. Without this option, the I/O daemons can only be reduced to
one.
DESCRIPTION
The voliod utility starts, stops, or reports on Logical Storage Manager I/O daemons. An I/O daemon is a process that provides a process
context for processing any work that needs to be done to process Logical Storage Manager I/O.
When invoked with no arguments, voliod prints the current number of volume I/O daemons on the standard output.
When invoked with the set keyword, the number of daemons specified by count will be created. If more volume I/O daemons exist than are
specified by count, then the excess processes will be terminated. If more than the maximum number are created (currently 64), the requested
number will be silently truncated to that maximum.
The number of daemons to create for general I/O handling depends on system load and usage. One daemon for each CPU on the system is gener-
ally adequate, unless volume recovery seems unusually slow.
Each I/O daemon starts in the background and creates an asynchronously-running kernel thread and becomes a volume I/O daemon. The voliod
utility does not wait for these threads to complete.
NOTES
LSM automatically sets the number of I/O daemons when the system starts, so it is usually not necessary to set or change the number of I/O
daemons with this command.
LSM I/O daemons cannot be killed directly through the use of signals.
The number of Logical Storage Manager I/O daemons currently running can only be determined by running voliod; I/O daemons do not appear in
the list of processes produced by the ps(1) command.
EXIT CODES
The voliod utility prints a diagnostic on the standard error, and exits if an error is encountered. If an I/O error occurs within a spawned
I/O daemon thread, then the I/O is not reflected in the exit status for voliod. Otherwise, voliod returns a nonzero exit status on errors.
Usage errors result in an exit status of 1 and a usage message. If the requested number of daemons cannot be created, then the exit status
is 2, and the number of daemons that were successfully started is reported. If any other error occurs, the exit status is 3.
FILES
The device used to report on and start volume I/O daemon kernel threads.
SEE ALSO fork(2), volintro(8), vold(8), voldctl(8) ,pthread(8)voliod(8)
04-17-2011
