Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: How do you wait for command substitution processes to complete?
Top Forums UNIX for Dummies Questions & Answers How do you wait for command substitution processes to complete? Post 302398051 by mrvwman on Tuesday 23rd of February 2010 06:26:19 PM
Old 02-23-2010
Yeah, the example I gave was perhaps oversimplified and doesn't show the reason behind the complexity. My actual tee statement has 6 such >(cmd) commands given to tee and the amount of data being read from disk is hundreds of gigs of compressed logs. The reason I'm doing it this way is so that the data is read of disk once, decompressed once and sent to 6 different processing pipelines in parallel for analysis.

The reading from disk is what takes all the time on my machine so this saves having to do it six times. Plus it leverages the 4 core processor I have to do the log processing in parallel. Even with 6 things running in parallel the CPU is not maxed out, it's still the disk IO that dictates the pace of processing.

Without this method it was taking much longer to process these logs.

I don't have access to the source code for all the tools in each of the 6 analysis pipelines otherwise I suppose I could rewrite those tools to do the parallel processing internally.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

PERL: wait for process to complete

I'm using PERL on windows NT to try to run an extract of data. I have multiple zip files in multiple locations. I am extracting "*.t" from zip files and subsequently adding that file to one zip file so when the script is complete I should have one zip file with a whole bunch of ".t" files in it. ... (2 Replies)
Discussion started by: dangral
2 Replies

2. Shell Programming and Scripting

Wait for Background Process to complete

I am attempting within a for-loop, to have my shell script (Solaris v8 ksh) wait until a copy file command to complete before continueing. The specific code is: for files in $(<inputfile.lst) do mv directory/$files directory/$files ksh -m -i bg %% wait $! done I am shaky on the... (3 Replies)
Discussion started by: gozer13
3 Replies

3. Shell Programming and Scripting

Wait Command

Does anyone have an example of a korn shell scripts kicking of multiple background processes and then using the wait command to get the return code from those processes? I want to write a program that kicks off multiple Oracle procedures and then wait for the return code before I procede.... (1 Reply)
Discussion started by: lesstjm
1 Replies

4. Programming

Howto spawn multiple child processes and wait?

As far as I can tell, the bash wait command waits for a logical "AND" of all the child processes. Assuming I am coding in C: (1) What is the function I would use to create multiple bash child process running perl? (2) What is the function I would use to reinvent the bash wait command so I... (4 Replies)
Discussion started by: siegfried
4 Replies

5. Shell Programming and Scripting

wait command - cat it wait for not-chile process?

Did not use 'wait' yet. How I understand by now the wait works only for child processes, started background. Is there any other way to watch completion of any, not related process (at least, a process, owned by the same user?) I need to start a background process, witch will be waiting... (2 Replies)
Discussion started by: alex_5161
2 Replies

6. Filesystems, Disks and Memory

hdparm + HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl

Hi All, Am finding performance of my SD card using hdparm. hdparm -tT /dev/BlockDev0 /dev/BlockDev0: Timing cached reads: 1118 MB in 2.00 seconds = 558.61 MB/sec HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device Timing buffered disk reads: 14... (0 Replies)
Discussion started by: amio
0 Replies

7. Shell Programming and Scripting

Wait command help

Hi, Is there any way to know the child process status as and when it finished. If i write like below nohup sh a1.sh & ### has sleep 20 ;echo a1.sh nohup sh a2.sh & ### has sleep 10 ;echo a2.sh nohup sh a3.sh & ### has sleep 5 ;echo a3.sh wait This will wait till a1.sh ,a2.sh a3.sh... (0 Replies)
Discussion started by: patrickk
0 Replies

8. Shell Programming and Scripting

Wait for one processes to complete in a shell script

Let's say I start process A.sh, then start process B.sh. I call both of them in my C.sh How can I make sure that B starts its execution only after A.sh finishes. I have to do this in loop.Execution time of A.sh may vary everytime. It is a parameterized script. (17 Replies)
Discussion started by: rafa_fed2
17 Replies

9. Shell Programming and Scripting

In Shell Script Does Second Command Wait For First Command To Complete

Hi All, I have a question related to Shell scripting. In my shell script, I have following two commands in sequence: sed 's/^/grep "^120" /g' $ORIGCHARGEDAMTLIST|sed "s;$;| cut -f$FIELD_NO1 -d '|' | awk '{ sum+=\$1} END {printf (\"%0.2f\\\n\", sum/100)}' >$TEMPFILE mv $TEMPFILE $ORIGFILE... (3 Replies)
Discussion started by: angshuman
3 Replies

10. Shell Programming and Scripting

Bash script parallel tasks and command to wait untill complete?

Hello, im having bash script with while *** command1 && command2 && command3 && done i want to ask how i can prevent overloading server, by waiting untill all commands complete? any low resources intensive command like "wait" - i dont know if exist? (2 Replies)
Discussion started by: postcd
2 Replies

LEARN ABOUT PLAN9

vpo

VPO(4)							   BSD Kernel Interfaces Manual 						    VPO(4)

NAME

     vpo -- parallel to SCSI interface driver

SYNOPSIS

     device vpo

     For one or more SCSI busses:
     device scbus

DESCRIPTION

     The vpo driver provide access to parallel port Iomega Zip and Jaz drives.

HARDWARE

     The vpo driver supports the following parallel to SCSI interfaces:

     o	 Adaptec AIC-7110 Parallel to SCSI interface (built-in to Iomega ZIP drives)
     o	 Iomega Jaz Traveller interface
     o	 Iomega MatchMaker SCSI interface (built-in to Iomega ZIP+ drives)

USAGE

     The driver should let you use a printer connected to the drive while transferring data.

     DOS and FreeBSD file systems are supported.  When mounting a DOS file system or formating a FreeBSD file system, check the slice of the disk
     with the fdisk(8) utility.

     In order to unixify a ZIP disk, put the following in /etc/disktab:

     zip|zip 100:
	     :ty=removable:se#512:nc#96:nt#64:ns#32:
	     :pa#196608:oa#0:ba#4096:fa#512:
	     :pb#196608:ob#0:bb#4096:fb#512:
	     :pc#196608:oc#0:bc#4096:fc#512:

     and use bsdlabel(8).

     If you have trouble with your driver, your parallel chipset may not run properly at the detected mode (NIBBLE, PS2 or EPP).  Tune the ppc(4)
     bootflags to force other modes.

SEE ALSO

     da(4), lpt(4), ppbus(4), ppc(4), scsi(4)

HISTORY

     The vpo manual page first appeared in FreeBSD 3.0.

AUTHORS

     This manual page was written by Nicolas Souchu.

BUGS

     During boot, the driver first tries to detect a classic ZIP, then a ZIP+.	The ZIP+ detection is intrusive and may send erroneous characters
     to your printer if the drive is not connected to your parallel port.

BSD
								 December 14, 2004							       BSD
All times are GMT -4. The time now is 06:31 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy