Since there've been a few requests for a method to execute commands on multiple CPUs (logical or physical), with various levels of shell-, make-, or Perl-based solutions, ranging from well-done to well-meant, and mostly specific to a certain problem, I've started to write a C-based solution (besides, I was bored). And here it is.
It's not complete by any means. It's not fail-safe, not even by far. It doesn't even have any documentation (yet). But it does what it's meant to do: run an arbitrary number of commands on and arbitrary number of CPUs.
How to install:
There's no installer yet. There's only a Makefile to build it, requiring GNU make. Just enter make (or gmake if needed), and everything should be compiled. The binary is called "parallel" until I come up with a better name.
How to run:
By default it will run 2 commands in at the same time. If you want to run more, specify the number on the command line, eg
to run 5 commands.
The commands to run are read from stdin, separated by carriage returns and/or newlines, and ended by EOF. Commands will be passed to /bin/sh for execution, so a bit of shell syntax is possible, as long as it fits on one line.
An example might be in order. To compress all files in the current directory using gzip, and using 8 CPUs, enter
Known problems:
Exit codes of individual commands are not propagated. As long as a child process could be created, everything's dandy.
Command input is not checked for a maximum length.
In theory it should run on any POSIX-compatible OS.
... and all the rest
If you find a bug & can correct it: please send me a patch (email is in the code)
If you want to write documentation: please send it to me, I'll include it.
If you find a good name for it: tell me.
It's under the 2-clause BSD license
Updates to the code will be posted here, unless there are a lot requests for some kind of repository.
And yes, it's classical fork/exec. However, as some posters expressed a need for a program to do just that (eg run multiple gzips in parallel without a CPU idling) I thought I'd try to fill that need.
Location: Saint Paul, MN USA / BSD, CentOS, Debian, OS X, Solaris
Posts: 2,288
Thanks Given: 430
Thanked 480 Times in 395 Posts
Hi. pludi.
I vaguely recall that xargs does something like that. The GNU/Linux version:
Code:
--max-procs=max-procs
-P max-procs
Run up to max-procs processes at a time; the default is 1. If
max-procs is 0, xargs will run as many processes as possible at
a time. Use the -n option with -P; otherwise chances are that
only one exec will be done.
-- excerpt from man xargs
Yes, ever tried to run GNU xargs on a non-Linux system? Last time I tried I had quite some dependencies to pull in too. Sometimes it's not available in binary form, so you'll have to compile it from source (including everything else in the findutils package), suddenly realizing that GNU code relies heavily on GNU libc. My code only needs a compiler (any standard C compiler will suffice) and a C library (again, any will suffice as long as POSIX is supported).
Also, in contrast to xargs, you're allowed to run more than 1 command in parallel. You could, for example, create gzip, bzip2, and LZMA compressed archives at the same time. Try that with xargs alone.
I have few very huge files ~ 2 Billion rows of 130 column(CDR data) in a folder, I have written shell script need to read on each file in a folder and will create a new files based on some logic.
But problem is it's taking time to create a new file due to the size , So i dont want to corrupt... (6 Replies)
Hi All,
We have a table that has to store around 80-100 million records. The table is partitioned by a column called Market Code. There are 30 partitions each corresponding to one MRKT_CD.
The source of this table is a join between 3-4 other tables. We are loading this table through SQLPLUS... (2 Replies)
Hi,
I am using Ubuntu 10.04 LTS 32-bit. I would like to execute a command in two or more servers (multiple remote hosts) in parallel. The commands "shmux" or "on" are not available in Ubuntu. Could anyone tell me what other equivalent commands available in Ubuntu to accomplish this task? (1 Reply)
I am haveing 2 scripts, 1st script calls 2ed script for each parameter.(parameters are kept in a different txt file)
1st script
for x in `cat Export_Tables_List.txt`
do
sh Exp_Table.sh $x &
done
echo -e "1) following tables are successfully exported : \n" > temp
cat... (1 Reply)
how can i process jobs parallel with conditions below.
Script1.ksh
Script2.ksh
Script3.ksh
Script4.ksh
Script5.ksh
Script6.ksh
Script7.ksh
Script8.ksh
Script9.ksh
Script10.ksh
After successful completion of Script1.ksh I need to run Script7.ksh.
After successful... (4 Replies)
Hi All,
At present I am using a UNIX Script which is running a set of JOBS. These JOBS are to be repeated for 20 times, means the same set of JOBS are repeated by just passing different arguments (From 1 to 20).
Is there any way by which I can execute them parallel?
At present its all... (4 Replies)
Hi All,
I am working on a project where I need to execute set of arguments (around 500) on a Simulator. If I execute this on one linux (RedHat 8.0) machine it will approximately takes 2~3 days. Since I am having more linux machines am thinking of executing these on different machines in... (7 Replies)
Hi,
Is there any way to run parallel jobs using make command?
I am using non-GNU make utility on AIX 4.3.
I want to run 2 jobs symultaneously using make utility.
Thanks.
Suman (0 Replies)
Hello,
I wish to run parallel process forked from one script.
Currently I doing is submit them in background.
For example:
---------------------------------------------------------------
#!/usr/bin/ksh
process1 &
process2 &
process3 &
.....
.....
#here I check for completion of... (4 Replies)