Sponsored Content
Full Discussion: xargs -P
Top Forums UNIX for Advanced & Expert Users xargs -P Post 302284941 by drl on Friday 6th of February 2009 03:14:40 PM
Old 02-06-2009
Hi.

I was very surprised to learn about the additional capabilities of GNU/Linux xargs. It can be used as a throttle -- allow no more than n processes to be running at a time (from the processes it creates). More than that, xargs monitors the processes and if one of the n processes stops, xargs will pluck another of its arguments and fire off an additional process. Very nice, but not very well documented at all.

To see how this might work (if at all) on a non-GNU/Linux machine. I obtained the source from findutils - GNU Project - Free Software Foundation (FSF) onto an HP box, and compiled it. It took a long time, but it finished -- it compiles a few other things, e.g. the GNU version of find. The tests included in the source failed to run -- an item named runtest could not be located -- but I ran a small test of my own, essentially:
Code:
xargs -n 2 -P 4 $burn

where burn is a script that prints start, sleeps, prints stop. Omitting the script and some output, but showing, for the number of processes being 4, that whenever a process ends, another one is started, but never more than 4:
Code:
$ ./s1 data3

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: HP-UX, B.11.00, 9000/712
Distribution        : GenericSysName [HP Release B.11.00] (see /etc/issue)
GNU bash 2.05b.0

 Results:
 job  4 start  8 seconds at: 06:10:33
 job  3 start  6 seconds at: 06:10:33
 job  2 start  4 seconds at: 06:10:33
 job  1 start 50 seconds at: 06:10:33
 job  2 stop   4 seconds at: 06:10:37
 job  5 start 10 seconds at: 06:10:37
 job  3 stop   6 seconds at: 06:10:39
 job  6 start 12 seconds at: 06:10:39
 job  4 stop   8 seconds at: 06:10:41
 job  7 start 14 seconds at: 06:10:41
 job  5 stop  10 seconds at: 06:10:47
 job  8 start 16 seconds at: 06:10:48
 job  6 stop  12 seconds at: 06:10:51
 job  9 start 18 seconds at: 06:10:52
 job  7 stop  14 seconds at: 06:10:55
 job 10 start 20 seconds at: 06:10:56
 job  8 stop  16 seconds at: 06:11:04
 job  9 stop  18 seconds at: 06:11:10
 job 10 stop  20 seconds at: 06:11:16
 job  1 stop  50 seconds at: 06:11:23

So, for large-granularity parallel processing, this could be useful.

Of course, many people here could write a shell script to do much the same, but it is nice to have it done for you ... cheers, drl
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Help with xargs

Hi there, I am trying to move around 3000 files from one directory to another. The mv command is complaining from too many arguments. I tried to use the xargs command but with no luck. Could some body provide help? Regards (4 Replies)
Discussion started by: JimJim
4 Replies

2. Shell Programming and Scripting

why we use xargs..

hi , can anyone help me by saying why we use xargs.. is it acing like a place holder..? thanks, Krips. (3 Replies)
Discussion started by: kripssmart
3 Replies

3. Shell Programming and Scripting

Using xargs

hi i just want to know that how do we use xargs command to find files which are greater than specified memory in a given directory (6 Replies)
Discussion started by: sumit the cool
6 Replies

4. Shell Programming and Scripting

Help in using xargs

Hi, I have a requirement to RCP the files from remote server to local server. Also the RCP has to run in parallel. However using 'xargs' retrives 2 file names during each loop. How do we restrict to only one file name using xargs and loop till remaining files. I use the below code for... (2 Replies)
Discussion started by: senthil3d
2 Replies

5. Shell Programming and Scripting

Xargs and

Hello there, Let me show you a simple example of what I am trying to achieve: 1) I have an input text file with some lines: 1 a 2 b 3 c 2) And I want to run a command with these lines as arguments (+ arbitrary extra arguments). For example: $ command "1 a" "2 b" "3 c" "bye" I... (7 Replies)
Discussion started by: tokland
7 Replies

6. Shell Programming and Scripting

Help with xargs

hi Could any one please tell me the option using which we can run multiple commands using xargs I have list of files, I want to run dos2unix and chmod at one shot on them I tried google n searched man pages but couldnt really find the solution , please help right now im doing this ls... (4 Replies)
Discussion started by: sunilmenhdiratt
4 Replies

7. Shell Programming and Scripting

xargs

Dear all , any suggest on xargs to combine from (1.txt and 2.txt) to output.txt ? thanks a lot. 1.txt 0123 BUM-5M BUM-5M 93490481 63839 0124 BUM-5M BUM-5M 112112 ... (3 Replies)
Discussion started by: samoptimus
3 Replies

8. Shell Programming and Scripting

Help with xargs

Using the bash shell I'm trying to either create a command for the command line or a script that will show netstat info for a given process name. Here is an example of what I'm trying to do:$ ps aux |grep catalina |grep -v grep | awk '{print $2}' 5132 $ netstat -nlp |grep 5132 (Not all processes... (11 Replies)
Discussion started by: axiopisty
11 Replies

9. Shell Programming and Scripting

Xargs

Hello, I need some help with xargs $ ls aaa bbb ccc ddd$ ls | xargs -I{} ls -la {} -rw-rw-r--. 1 xxx xx 0 May 30 20:04 aaa -rw-rw-r--. 1 xxx xx 0 May 30 20:04 bbb -rw-rw-r--. 1 xxx xx 0 May 30 20:04 ccc -rw-rw-r--. 1 xxx xx 0 May 30 20:04 dddit's possible to have output like this with... (3 Replies)
Discussion started by: vikus
3 Replies

10. UNIX for Dummies Questions & Answers

Xargs

Hi, can anyone tell me in detail ? what the following do in detail ? I am trying to get a largest number in a list Thanks Tao LARGEST=$(echo $* | xargs -n1 | sort -nr | tail -1) (3 Replies)
Discussion started by: ccp
3 Replies
stopping(7)                                              Miscellaneous Information Manual                                              stopping(7)

NAME
stopping - event signalling that a job is stopping SYNOPSIS
stopping JOB=JOB INSTANCE=INSTANCE RESULT=RESULT [PROCESS=PROCESS] [EXIT_STATUS=STATUS] [EXIT_SIGNAL=SIGNAL] [ENV]... DESCRIPTION
The stopping event is generated by the Upstart init(8) daemon when an instance of a job begins stopping. The JOB environment variable con- tains the job name, and the INSTANCE environment variable contains the instance name which will be empty for single-instance jobs. If the job is stopping normally, the RESULT environment variable will be ok, otherwise if the job is stopping because it has failed it will be failed. When the job has failed, the process that failed will be given in the PROCESS environment variable. This may be pre-start, post-start, main, pre-stop or post-stop; it may also be the special value respawn to indicate that the job is stopping because it hit the respawn limit. Finally in the case of a failed job, one of either EXIT_STATUS or EXIT_SIGNAL may be given to indicate the cause of the stop. Either EXIT_STATUS will contain the exit status code of the process, or EXIT_SIGNAL will contain the name of the signal that the process received. The normal exit job configuration stanza can be used to prevent particular exit status values or signals resulting in a failed job, see init(5) for more information. If neither EXIT_STATUS or EXIT_SIGNAL is given for a failed process, it is because the process failed to spawn (for example, file not found). See the system logs for the error. init(8) will wait for all services started by this event to be running, all tasks started by this event to have finished and all jobs stopped by this event to be stopped before allowing the job to continue stopping. This allows jobs to depend on other jobs, safely stopping themselves before their dependency goes away. This event is typically combined with the started(7) event by services. Job configuration files may use the export stanza to export environment variables from their own environment into the stopping event. See init(5) for more details. EXAMPLE
A service that wishes to depend on another service might use: start on started apache stop on stopping apache A task that must be run before another task or service is stopped might use: start on stopping postgresql RESULT=ok SEE ALSO
starting(7) started(7) stopped(7) init(5) Upstart 2009-07-09 stopping(7)
All times are GMT -4. The time now is 09:29 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy