UNIX for Dummies Questions & Answers

When running a command using the >(cmd) syntax in bash how do you wait for the command to complete before moving on in your script?

Here is a simple example:

Code:

zcat largefile.gz | tee >(wc && echo “HELLO”) > /dev/null
# I tried wait, here but it doesn't wait for the process in the subshell.
echo "WORLD"

This produces the output:

Code:

WORLD
22245  175374 3598423
HELLO

My objective is to guarantee echo "WORLD" gets printed after all of the >() processes are done.

The bigger picture is that I have a lot of large log files to process through a variety of log parsers and using this syntax does have the positive effect of reading the files from disk once to process them in parallel in many sub processes taking advantage of multi-core CPUs. Everything works fine except for the fact that I can't tell when those processes are done.

I can think of one solution where I could have my log processing scripts write an "I'm Done" message to a file or pipe somewhere for the parent to loop listening for.

Is there a better way?

I hope We can make use of wait and sleep command here .

wait doesn't work, I tried that. sleep might work most of the time, but it isn't a guarantee that the work would be done after sleeping.

I wrote up the following to do what I want, it works but I guess I was hoping for a more elegant solution like a simple flag for wait or something.

Code:

#!/bin/bash

DONE_PIPE=/tmp/donepipe$$
mkfifo $DONE_PIPE
exec 7<>$DONE_PIPE

zcat /logs/billing.log.2009/09/01/billing.log.20090901_101924.96.17.157.45.gz | tee >(wc && echo "HELLO" && echo "DONE" >&7) > /dev/null
COUNTER=0
#wait for the processors to complete
while [ $COUNTER -lt 1 ]; do
    read LINE <&7
    echo $LINE
    let COUNTER=COUNTER+1
done

exec 7>&-
exec 7<&-

rm $DONE_PIPE

echo "WORLD"

Which produces the following output:

Code:

22245  175374 3598423
HELLO
DONE
WORLD

With the above code multiple >(cmd) arguments can be given to tee and each would echo a statement to the 7 filedescriptor when they're complete. The while loop condition would just be adjusted to wait for however many processes you have launched using >(cmd).

Hi.

You seem to be effectively doing:

Code:

zcat $FILE | wc ; echo "HELLO"

I suggest eliminating the tee, since you are throwing away the output from the zcat anyway ( >/dev/null ).

The FIFO that the shell creates for ">(" may be causing the problem. I did verify that the behaviour you posted is accurate -- at least it is with bash and ksh (and it surprised me). It seemed to work correctly with zsh, so you may wish to try that variant.

I didn't quite understand the bigger picture you described, so I cannot comment farther.

Best wishes ... cheers, drl

Yeah, the example I gave was perhaps oversimplified and doesn't show the reason behind the complexity. My actual tee statement has 6 such >(cmd) commands given to tee and the amount of data being read from disk is hundreds of gigs of compressed logs. The reason I'm doing it this way is so that the data is read of disk once, decompressed once and sent to 6 different processing pipelines in parallel for analysis.

The reading from disk is what takes all the time on my machine so this saves having to do it six times. Plus it leverages the 4 core processor I have to do the log processing in parallel. Even with 6 things running in parallel the CPU is not maxed out, it's still the disk IO that dictates the pace of processing.

Without this method it was taking much longer to process these logs.

I don't have access to the source code for all the tools in each of the 6 analysis pipelines otherwise I suppose I could rewrite those tools to do the parallel processing internally.

Like "drl" I don't get exactly what you are tying to achieve. I can't see why there is the complexity of redirecting channels.

It seems that you want to issue multiple command sets concurrently and wait until they have all completed.

IMHO forget using exec. It will drive you nuts with multiple concurrent processes.

Best approach is to background the command (or lots of commands).
[CODE]nohup command_name &[/CODE

Then check for background jobs.

Code:

jobs

When there are no background jobs running, all the background jobs for your user have completed.

I'll leave you to script processing the output (or not!) from the shell command "jobs". See "man" for your shell to read about "jobs".

I'm trying to achieve the following goals:
1. Only read the log files from disk once.
2. Only decompress the log files once.
3. Process the log files with multiple parallel processes.
4. wait for them all to complete
5. do something else.

The reason being is that disk io and decompression are expensive in comparison to log processing, so I only want to do them once. And running the processing in parallel leverages my quad core processor to further reduce run time.

I think I have what I need, but I'm open to any other suggestions for how to do this better.

With background jobs method you suggest I would have to :
1. create an array of 6 unique temporary pipe names
2. mkfifo for each of the 6 pipes
3. start sending the decompressed logs to 6 named pipes using tee
4. launch a the 6 processing pipelines in the background telling them to read from the named pipes.
5. Now I could use wait to wait for them all.
6. rm all the temporary pipes
7. continue my script

This would work too, but is this better than using 'tee >(cmd1) >(cmd2)...>(cmd6)' and waiting for them to echo something into one named pipe that I have to manually manage? At least 'tee' takes care of creating and destroying the named pipes for me.