Sponsored Content
Top Forums UNIX for Dummies Questions & Answers How do you wait for command substitution processes to complete? Post 302398064 by mrvwman on Tuesday 23rd of February 2010 07:27:42 PM
Old 02-23-2010
I'm trying to achieve the following goals:
1. Only read the log files from disk once.
2. Only decompress the log files once.
3. Process the log files with multiple parallel processes.
4. wait for them all to complete
5. do something else.

The reason being is that disk io and decompression are expensive in comparison to log processing, so I only want to do them once. And running the processing in parallel leverages my quad core processor to further reduce run time.

I think I have what I need, but I'm open to any other suggestions for how to do this better.

With background jobs method you suggest I would have to :
1. create an array of 6 unique temporary pipe names
2. mkfifo for each of the 6 pipes
3. start sending the decompressed logs to 6 named pipes using tee
4. launch a the 6 processing pipelines in the background telling them to read from the named pipes.
5. Now I could use wait to wait for them all.
6. rm all the temporary pipes
7. continue my script

This would work too, but is this better than using 'tee >(cmd1) >(cmd2)...>(cmd6)' and waiting for them to echo something into one named pipe that I have to manually manage? At least 'tee' takes care of creating and destroying the named pipes for me.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

PERL: wait for process to complete

I'm using PERL on windows NT to try to run an extract of data. I have multiple zip files in multiple locations. I am extracting "*.t" from zip files and subsequently adding that file to one zip file so when the script is complete I should have one zip file with a whole bunch of ".t" files in it. ... (2 Replies)
Discussion started by: dangral
2 Replies

2. Shell Programming and Scripting

Wait for Background Process to complete

I am attempting within a for-loop, to have my shell script (Solaris v8 ksh) wait until a copy file command to complete before continueing. The specific code is: for files in $(<inputfile.lst) do mv directory/$files directory/$files ksh -m -i bg %% wait $! done I am shaky on the... (3 Replies)
Discussion started by: gozer13
3 Replies

3. Shell Programming and Scripting

Wait Command

Does anyone have an example of a korn shell scripts kicking of multiple background processes and then using the wait command to get the return code from those processes? I want to write a program that kicks off multiple Oracle procedures and then wait for the return code before I procede.... (1 Reply)
Discussion started by: lesstjm
1 Replies

4. Programming

Howto spawn multiple child processes and wait?

As far as I can tell, the bash wait command waits for a logical "AND" of all the child processes. Assuming I am coding in C: (1) What is the function I would use to create multiple bash child process running perl? (2) What is the function I would use to reinvent the bash wait command so I... (4 Replies)
Discussion started by: siegfried
4 Replies

5. Shell Programming and Scripting

wait command - cat it wait for not-chile process?

Did not use 'wait' yet. How I understand by now the wait works only for child processes, started background. Is there any other way to watch completion of any, not related process (at least, a process, owned by the same user?) I need to start a background process, witch will be waiting... (2 Replies)
Discussion started by: alex_5161
2 Replies

6. Filesystems, Disks and Memory

hdparm + HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl

Hi All, Am finding performance of my SD card using hdparm. hdparm -tT /dev/BlockDev0 /dev/BlockDev0: Timing cached reads: 1118 MB in 2.00 seconds = 558.61 MB/sec HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device Timing buffered disk reads: 14... (0 Replies)
Discussion started by: amio
0 Replies

7. Shell Programming and Scripting

Wait command help

Hi, Is there any way to know the child process status as and when it finished. If i write like below nohup sh a1.sh & ### has sleep 20 ;echo a1.sh nohup sh a2.sh & ### has sleep 10 ;echo a2.sh nohup sh a3.sh & ### has sleep 5 ;echo a3.sh wait This will wait till a1.sh ,a2.sh a3.sh... (0 Replies)
Discussion started by: patrickk
0 Replies

8. Shell Programming and Scripting

Wait for one processes to complete in a shell script

Let's say I start process A.sh, then start process B.sh. I call both of them in my C.sh How can I make sure that B starts its execution only after A.sh finishes. I have to do this in loop.Execution time of A.sh may vary everytime. It is a parameterized script. (17 Replies)
Discussion started by: rafa_fed2
17 Replies

9. Shell Programming and Scripting

In Shell Script Does Second Command Wait For First Command To Complete

Hi All, I have a question related to Shell scripting. In my shell script, I have following two commands in sequence: sed 's/^/grep "^120" /g' $ORIGCHARGEDAMTLIST|sed "s;$;| cut -f$FIELD_NO1 -d '|' | awk '{ sum+=\$1} END {printf (\"%0.2f\\\n\", sum/100)}' >$TEMPFILE mv $TEMPFILE $ORIGFILE... (3 Replies)
Discussion started by: angshuman
3 Replies

10. Shell Programming and Scripting

Bash script parallel tasks and command to wait untill complete?

Hello, im having bash script with while *** command1 && command2 && command3 && done i want to ask how i can prevent overloading server, by waiting untill all commands complete? any low resources intensive command like "wait" - i dont know if exist? (2 Replies)
Discussion started by: postcd
2 Replies
subst(n)						       Tcl Built-In Commands							  subst(n)

__________________________________________________________________________________________________________________________________________________

NAME
subst - Perform backslash, command, and variable substitutions SYNOPSIS
subst ?-nobackslashes? ?-nocommands? ?-novariables? string _________________________________________________________________ DESCRIPTION
This command performs variable substitutions, command substitutions, and backslash substitutions on its string argument and returns the fully-substituted result. The substitutions are performed in exactly the same way as for Tcl commands. As a result, the string argument is actually substituted twice, once by the Tcl parser in the usual fashion for Tcl commands, and again by the subst command. If any of the -nobackslashes, -nocommands, or -novariables are specified, then the corresponding substitutions are not performed. For example, if -nocommands is specified, command substitution is not performed: open and close brackets are treated as ordinary characters with no special interpretation. Note that the substitution of one kind can include substitution of other kinds. For example, even when the -novariables option is speci- | fied, command substitution is performed without restriction. This means that any variable substitution necessary to complete the command | substitution will still take place. Likewise, any command substitution necessary to complete a variable substitution will take place, even | when -nocommands is specified. See the EXAMPLES below. | If an error occurs during substitution, then subst will return that error. If a break exception occurs during command or variable substi- | tution, the result of the whole substitution will be the string (as substituted) up to the start of the substitution that raised the excep- | tion. If a continue exception occurs during the evaluation of a command or variable substitution, an empty string will be substituted for | that entire command or variable substitution (as long as it is well-formed Tcl.) If a return exception occurs, or any other return code is | returned during command or variable substitution, then the returned value is substituted for that substitution. See the EXAMPLES below. | In this way, all exceptional return codes are ``caught'' by subst. The subst command itself will either return an error, or will complete | successfully. EXAMPLES
When it performs its substitutions, subst does not give any special treatment to double quotes or curly braces (except within command sub- stitutions) so the script set a 44 subst {xyz {$a}} returns ``xyz {44}'', not ``xyz {$a}'' and the script | set a "p} q {r" | subst {xyz {$a}} | return ``xyz {p} q {r}'', not ``xyz {p} q {r}''. | When command substitution is performed, it includes any variable substitution necessary to evaluate the script. | set a 44 | subst -novariables {$a [format $a]} | returns ``$a 44'', not ``$a $a''. Similarly, when variable substitution is performed, it includes any command substitution necessary to | retrieve the value of the variable. | proc b {} {return c} | array set a {c c [b] tricky} | subst -nocommands {[b] $a([b])} | returns ``[b] c'', not ``[b] tricky''. | The continue and break exceptions allow command substitutions to prevent substitution of the rest of the command substitution and the rest | of string respectively, giving script authors more options when processing text using subst. For example, the script | subst {abc,[break],def} | returns ``abc,'', not ``abc,,def'' and the script | subst {abc,[continue;expr 1+2],def} | returns ``abc,,def'', not ``abc,3,def''. | Other exceptional return codes substitute the returned value | subst {abc,[return foo;expr 1+2],def} | returns ``abc,foo,def'', not ``abc,3,def'' and | subst {abc,[return -code 10 foo;expr 1+2],def} | also returns ``abc,foo,def'', not ``abc,3,def''. SEE ALSO
Tcl(n), eval(n), break(n), continue(n) KEYWORDS
backslash substitution, command substitution, variable substitution Tcl 7.4 subst(n)
All times are GMT -4. The time now is 08:44 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy