02-23-2010
Yeah, the example I gave was perhaps oversimplified and doesn't show the reason behind the complexity. My actual tee statement has 6 such >(cmd) commands given to tee and the amount of data being read from disk is hundreds of gigs of compressed logs. The reason I'm doing it this way is so that the data is read of disk once, decompressed once and sent to 6 different processing pipelines in parallel for analysis.
The reading from disk is what takes all the time on my machine so this saves having to do it six times. Plus it leverages the 4 core processor I have to do the log processing in parallel. Even with 6 things running in parallel the CPU is not maxed out, it's still the disk IO that dictates the pace of processing.
Without this method it was taking much longer to process these logs.
I don't have access to the source code for all the tools in each of the 6 analysis pipelines otherwise I suppose I could rewrite those tools to do the parallel processing internally.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I'm using PERL on windows NT to try to run an extract of data. I have multiple zip files in multiple locations. I am extracting "*.t" from zip files and subsequently adding that file to one zip file so when the script is complete I should have one zip file with a whole bunch of ".t" files in it.
... (2 Replies)
Discussion started by: dangral
2 Replies
2. Shell Programming and Scripting
I am attempting within a for-loop, to have my shell script (Solaris v8 ksh) wait until a copy file command to complete before continueing. The specific code is:
for files in $(<inputfile.lst)
do
mv directory/$files directory/$files
ksh -m -i bg %%
wait $!
done
I am shaky on the... (3 Replies)
Discussion started by: gozer13
3 Replies
3. Shell Programming and Scripting
Does anyone have an example of a korn shell scripts kicking of multiple background processes and then using the wait command to get the return code from those processes?
I want to write a program that kicks off multiple Oracle procedures and then wait for the return code before I procede.... (1 Reply)
Discussion started by: lesstjm
1 Replies
4. Programming
As far as I can tell, the bash wait command waits for a logical "AND" of all the child processes.
Assuming I am coding in C:
(1) What is the function I would use to create multiple bash child process running perl?
(2) What is the function I would use to reinvent the bash wait command so I... (4 Replies)
Discussion started by: siegfried
4 Replies
5. Shell Programming and Scripting
Did not use 'wait' yet.
How I understand by now the wait works only for child processes, started background.
Is there any other way to watch completion of any, not related process (at least, a process, owned by the same user?)
I need to start a background process, witch will be waiting... (2 Replies)
Discussion started by: alex_5161
2 Replies
6. Filesystems, Disks and Memory
Hi All,
Am finding performance of my SD card using hdparm.
hdparm -tT /dev/BlockDev0
/dev/BlockDev0:
Timing cached reads: 1118 MB in 2.00 seconds = 558.61 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate
ioctl for device
Timing buffered disk reads: 14... (0 Replies)
Discussion started by: amio
0 Replies
7. Shell Programming and Scripting
Hi, Is there any way to know the child process status as and when it finished. If i write like below
nohup sh a1.sh & ### has sleep 20 ;echo a1.sh
nohup sh a2.sh & ### has sleep 10 ;echo a2.sh
nohup sh a3.sh & ### has sleep 5 ;echo a3.sh
wait
This will wait till a1.sh ,a2.sh a3.sh... (0 Replies)
Discussion started by: patrickk
0 Replies
8. Shell Programming and Scripting
Let's say I start process A.sh, then start process B.sh. I call both of them in my C.sh
How can I make sure that B starts its execution only after A.sh finishes.
I have to do this in loop.Execution time of A.sh may vary everytime.
It is a parameterized script. (17 Replies)
Discussion started by: rafa_fed2
17 Replies
9. Shell Programming and Scripting
Hi All,
I have a question related to Shell scripting. In my shell script, I have following two commands in sequence:
sed 's/^/grep "^120" /g' $ORIGCHARGEDAMTLIST|sed "s;$;| cut -f$FIELD_NO1 -d '|' | awk '{ sum+=\$1} END {printf (\"%0.2f\\\n\", sum/100)}' >$TEMPFILE
mv $TEMPFILE $ORIGFILE... (3 Replies)
Discussion started by: angshuman
3 Replies
10. Shell Programming and Scripting
Hello,
im having bash script with
while ***
command1 &&
command2 &&
command3 &&
done
i want to ask how i can prevent overloading server, by waiting untill all commands complete? any low resources intensive command like "wait" - i dont know if exist? (2 Replies)
Discussion started by: postcd
2 Replies
voxbo(7) The VoxBo Manual voxbo(7)
NAME
voxbo - general information about the VoxBo toolbox
DESCRIPTION
VoxBo is a suite of tools for brain image analysis, especially fMRI analysis and lesion-behavior mapping. VoxBo includes over 100 tools,
including many for general-purpose image munging (e.g., vbim(1)), many that implement specific statistical procedures (e.g., vbtmap(1)),
and some that would likely be just as useful for non-imaging applications (vbbatch(1)).
VoxBo has supported cluster parallel processing since 1998. Although cluster installation is a little tricky, it's worth the effort if
you'd like to speed up your analyses by an order of magnitude or two. As of 2010, VoxBo also supports desktop parallel processing on
multi-core systems. We do not yet support GPU parallel processing.
All VoxBo tools support a common set of file formats. See voxbo-fileformats(7) for more information.
AVAILABILITY
Pre-built VoxBo binaries are available for Linux, OSX, and Cygwin, and via the NeuroDebian project (neuro.debian.net) for Debian deriva-
tives.
CAVEATS
VoxBo is provided with no warranty whatsoever.
AUTHOR
For an historical roster of the VoxBo development team, visit www.voxbo.org.
SEE ALSO
For detailed help on any VoxBo command-line program, run it with no arguments. For graphical programs, use the -h flag. For more help
with VoxBo, for information about the mailing list, or to report bugs, visit the web site at www.voxbo.org.
For information about VoxBo file formats, see voxbo-fileformats(7).
VoxBo January, 2011 (v1.8.5) voxbo(7)