I think I wrote something similar to this recently which is almost like a map-reduce version for a single host which is more robust for a shared host execution which takes care of cpu and available memory hogging.
Interfacing between the master and slave workers is not that great, as I used lock files to communicate between master and slave. Before the slave is about to start its destined work, it creates a lock file which is accessible by the master and it checks periodically to see if the lock file is still there. Once the worker completes the work, the lock file would be removed which indicates that worker is done with work, based on current load on the system a new process can or cannot be spawned.
Though I said, lock interfacing isn't that great, it works very well as the master is very pessimist and takes the veto to kill the worker if it detects that the worker has gone stale, not needed any more, parallel execution of work by more than 1 worker.
Is that the same problem that you are trying to solve?
---------- Post updated at 06:58 PM ---------- Previous update was at 06:53 PM ----------
Quote:
Originally Posted by pbillast
In these kind of situations you'll find the environment variable $! (pid of backgound process) and the manpage for the command wait very useful.
I recently made something to launch sql statements and gzip on the output files with a maximum number of subprocesses (I chose this maximum in function of the numbers of cpu cores)
basically in the script you start a subprocess
Code:
ClientDataEncrypt <args> &
BPID=$!
and thus keep track of the backgound pids (by storing them in an array).
from there you can limit the number of subprocesses.
(P.S. on Solaris 10 pgrep is very useful in these situations, especially if you want to be able to launch ClientDataEncrypt both manually and via the watcher and want the watcher daemon to know about it)
I hope this helps
Is the reason for storing PIDs to check periodically whether the sub process has completed execution or not? If so, there is a potential problem in it and yes it will happen in ultra busy nodes.
For ex: Lets take the pid stored is 'p1', by the time watcher scans the pid list, 'p1' can complete its processing, die and a new process with 'p1' can be spawned again, watcher unaware of this will think that 'p1' is still alive which is not true.
Basically, problem is due to expanded scope, if there is a boundary drawn using process group, this problem can be avoided. Even if 'p1' is respawned as part of some other process, it wont be part of the process group boundary that we are checking.
How do you capture the return code from a background process?
I am dumping data to a fifo and then processing it in a c program.
I need to know that the sql finished successfully to ensure no missing data. Thanks.
ex.
sqlplus user/password < get_data.sql > data_fifo.txt &
bin/process_data... (2 Replies)
Hello Friends,
sorry, i am not very familiar with Unix programming. Could you please help me on this?
We have to start different components from a startup script.
each components are started as below in the background in a startprocess function
$nohup $file $args >>$logFile 2>&1 &
... (0 Replies)
Hello Friends,
sorry, i am not very familiar with Unix programming. Could you please help me on this?
We have to start different components from a startup script.
each components are started as below in the background in a startprocess function
$nohup $file $args >>$logFile 2>&1 &
... (1 Reply)
I have script 3 scripts
1 parent
2 children
child1
child2
In the code below the 2 child processes fire almost Instantaneously in the background, Is that possible to know the status of pass/fail of each process "as it happens" ?
In the present scenario although Child2... (5 Replies)
I have script 3 scripts 1 parent (p1) and 2 children child1 and child2
I have script 3 scripts
1 parent
2 children
child1
child2
In the code below the 2 child processes fire almost Instantaneously in
the background, Is that possible to know the status of pass/fail of each
process... (12 Replies)
Ok guys so I have my first dummy shell almost done except for one tiny part: I do not know how to run a process in the background, from the code!
I already know how to do that in a normal shell:
$ program &
However, no clue when it comes to how to program that thing. :eek:
A very... (2 Replies)
Hi,
I have a schell script parent.ksh from which I am calling three background processes a.ksh,b.ksh and c.ksh. Once these three processes completes the next step in parent.ksh should execute. How to achieve this?
Please help me....
Thanks... (1 Reply)
Hi All,
Iam trying to get a file processed and some lines have spaces...the below is not working
Want to remove empty line
Want to remove lines that start with #
Avoid line with substring WHOA
When trying to get the substring from the var also Iam having trouble
file is like VAR=VALUE,... (13 Replies)
Hello
I have a file which has around 120 lines of commands.
I am trying to write a shell script like which reads the 'command' file and executes line by line with some additional (common argument) with maximum 6 commands active at a time. Each of these commands when executed takes time... (5 Replies)
Discussion started by: JackyShane_36
5 Replies
LEARN ABOUT REDHAT
postfix
POSTFIX(1) General Commands Manual POSTFIX(1)NAME
postfix - Postfix control program
SYNOPSIS
postfix [-c config_dir] [-D] [-v] command
DESCRIPTION
This command is reserved for the super-user. To submit mail, use the Postfix sendmail command.
The postfix command controls the operation of the Postfix mail system: start or stop the master daemon, do a health check, and other main-
tenance.
The postfix command sets up a standardized environment and runs the postfix-script shell script to do the actual work.
The following commands are implemented:
check Validate the Postfix mail system configuration. Warn about bad directory/file ownership or permissions, and create missing directo-
ries.
start Start the Postfix mail system. This also runs the configuration check described above.
stop Stop the Postfix mail system in an orderly fashion. Running processes are allowed to terminate at their earliest convenience.
Note: in order to refresh the Postfix mail system after a configuration change, do not use the start and stop commands in succes-
sion. Use the reload command instead.
abort Stop the Postfix mail system abruptly. Running processes are signaled to stop immediately.
flush Force delivery: attempt to deliver every message in the deferred mail queue. Normally, attempts to deliver delayed mail happen at
regular intervals, the interval doubling after each failed attempt.
reload Re-read configuration files. Running processes terminate at their earliest convenience.
The following options are implemented:
-c config_dir
Read the main.cf and master.cf configuration files in the named directory instead of the default configuration directory. Use this
to distinguish between multiple Postfix instances on the same host.
-D (with postfix start only)
Run each Postfix daemon under control of a debugger as specified via the debugger_command configuration parameter.
-v Enable verbose logging for debugging purposes. Multiple -v options make the software increasingly verbose.
ENVIRONMENT
The postfix command sets the following environment variables:
MAIL_CONFIG
Directory with Postfix configuration files.
MAIL_VERBOSE
This is set when the -v command-line option is present.
MAIL_DEBUG
This is set when the -D command-line option is present.
CONFIGURATION PARAMETERS
The following main.cf configuration parameters are made available as process environment variables with the same names:
command_directory
Directory with Postfix administrative commands.
daemon_directory
Directory with Postfix daemon programs.
config_directory
Directory with Postfix configuration files and with administrative shell scripts.
queue_directory
The directory with Postfix queue files, with local inter-process communication endpoints, and with files needed for daemon programs
that run in the optional chrooted environment.
mail_owner
The owner of Postfix queue files and of most Postfix processes.
setgid_group
The group for mail submission and queue management commands.
sendmail_path
The full pathname for the Postfix sendmail command.
newaliases_path
The full pathname for the Postfix newaliases command.
mailq_path
The full pathname for the Postfix mailq command.
manpage_directory
The directory for the Postfix on-line manual pages.
sample_directory
The directory for the Postfix sample configuration files.
readme_directory
The directory for the Postfix README files.
Other configuration parameters
import_environment
List of names of environment parameters that can be imported from non-Postfix processes.
FILES
$config_directory/postfix-script, administrative commands
$config_directory/main.cf, configuration parameters
$config_directory/master.cf, Postfix daemon processes
SEE ALSO master(8) Postfix master program
LICENSE
The Secure Mailer license must be distributed with this software.
AUTHOR(S)
Wietse Venema
IBM T.J. Watson Research
P.O. Box 704
Yorktown Heights, NY 10598, USA
POSTFIX(1)
04-17-2011
