Multiprocessing Help


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Multiprocessing Help
# 8  
Old 11-01-2012
Hi.

Here is an example of GNU parallel running on an (old) OSX box. This call spawns as many processes as needed for each filename with a suffix of txt. There are also controls for CPUs as well as number of cores.
Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate multiple processes simultaneously, with "GNU parallel".
# See: http://www.gnu.org/software/parallel
# See: http://freecode.com/projects/align

pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C parallel tree

pl " Structure of directories:"
tree d1 
tree d2

pl " Results of parallel processes:"
# ls d1/* d2/* |
find d1 d2 -name '*.txt' |
parallel --jobs 0 --ungroup 'echo -n job {#}, process $$", wc = "; wc {}' 2>/dev/null |
align

exit 0

producing:
Code:
$ ./s1

Environment: LC_ALL = POSIX, LANG = POSIX
(Versions displayed with local utility "version")
OSX 10.3.9
GNU bash 2.05b.0
grep - ( /usr/bin/grep, 29 Aug 2008 )
parallel - ( local: ~/bin/parallel, 1 Nov 11:35 )
tree - ( local: RepRev 1.2, ~/bin/tree, 1 Nov 11:48 )

-----
 Structure of directories:
d1
`--a.txt
`--b.txt
`--binary-1.exe
`--c.txt
d2
`--frog-town.jpg
`--x.txt
`--y.txt

-----
 Results of parallel processes:
job 1, process 2620, wc =  4  16   70 d1/a.txt
job 2, process 2622, wc = 16  16  123 d1/b.txt
job 3, process 2624, wc = 26 265 1464 d1/c.txt
job 4, process 2626, wc =  4  16   70 d2/x.txt
job 5, process 2628, wc = 16  16  123 d2/y.txt

See URLs mentioned in comments if you do not have the items in your repository.

Best wishes ... cheers, drl
# 9  
Old 11-02-2012
Thanks everyone for the helpful responses! I was messing around some last night, I think where I will go from here is change my loop, which looks like this:
Code:
#Stage 1 Data Mode
#$runlist_size is number of files found in text file
#Analyze every other file starting with first
sub Stage1_DataMode
{
    for (my $i=0; $i<$runlist_size; $i=$i+2)
    {
        system "vaStage1 -Stage1_RunMode=data -config=${CONFIG}Stg1.config $VBFDIR$runlist[$i].cvbf ${OUTPUT}$runlist[$i].root";
    }
}

to this:
Code:
#Stage 1 Data Mode
#$runlist_size is number of files found in text file
#Analyze every other file starting with first
sub Stage1_DataMode
{
    for (my $i=0; $i<$runlist_size; $i=$i+2)
    {
        system "vaStage1 -Stage1_RunMode=data -config=${CONFIG}Stg1.config $VBFDIR$runlist[$i].cvbf ${OUTPUT}$runlist[$i].root &";
    }
}

Adding the "&" to send files to the background. Then I will do some testing to see how many files I should send through at once, and change the loop accordingly. I'll also do some experimenting with parallel, that looks like a nice tool.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

4 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

<< Threading inside multiprocessing using queues >>

Hi All, I am trying to achieve threading inside each process of multiprocessing. I have 2 queues one for multiprocess (process) & another inside each process. when i execute it got hung after below output. My goal here is to go through p_source queue & for each process picks up all t_source... (0 Replies)
Discussion started by: kamauv234
0 Replies

2. Shell Programming and Scripting

Multiprocessing in Python

Hi there, I have a code that can take in any function with two arguements and do processing. However, I would like to implement a feature whether it can limit a number of process running concurrently so as not take up too much resources. I have tried researching for pool.map however I am unable... (1 Reply)
Discussion started by: alvinoo
1 Replies

3. Programming

Multiprocessing multipointers

I have a complex problem..... I have to search files on directory "text files" then search on all of them for a word or sentence....the user inter my problem is,,,, if I want to create a child for each file...and point a file by pointer to search...and I don't know how much files i have in... (2 Replies)
Discussion started by: fwrlfo
2 Replies

4. UNIX for Dummies Questions & Answers

Multiprocessing under Linux

I'm writing C programs to be executed on a multi-processor UNIX (GNU/Linux, kernel 2.6.11) Do I need to add a special kind of code to somewhere or run a special utility to execute the program file to be executed by all processors? Or is it handled automatically by kernel? (1 Reply)
Discussion started by: rayne
1 Replies
Login or Register to Ask a Question