Shell Programming and Scripting

Hello,

I have a script that is running multiple instances of an application in parallel.

Code:

# learn on f0
emergent -nogui -p $ScriptLoc/$PROJ fold_tag=f0 &
sleep 5

# learn on f1
emergent -nogui -p $ScriptLoc/$PROJ fold_tag=f1 &
sleep 5

# learn on f2
emergent -nogui -p $ScriptLoc/$PROJ fold_tag=f2 &
sleep 5

# learn on f3
emergent -nogui -p $ScriptLoc/$PROJ fold_tag=f3 &
sleep 5

# learn on f4
emergent -nogui -p $ScriptLoc/$PROJ fold_tag=f4

# wait 4 minutes to make sure that processing has finished for all folds
sleep 240

echo "statistics"  
./stats4A_04.sh $BATCH 0 &
./stats4A_04.sh $BATCH 1 &
./stats4A_04.sh $BATCH 2 &
./stats4A_04.sh $BATCH 3 &
./stats4A_04.sh $BATCH 4

There are short sleep instances between where each instance is called. This helps keep the output to the terminal from getting garbled as multiple programs are printing at the same time. When all the instances have returned a separate program is called to process the output.

The problem I am having is that the output processing app (stats4A) is getting called before all of the instances have finished. I think that if f4 finishes before the others, then the script proceeds to call the stats program.

This more or less calls for a mutex to keep the stats program from running until everything has finished. There is no way to guarantee that f4 will finish after the others. You can see that I am already waiting 4 minutes as a measure to address this, but it appears that does not always work.

The only thing I can think of at the moment would be to have a 0/1 variable for each instance. That variable would initialize to 0. I would call the instances in a function, instead of inline, and have the function set the variable to 1 after the instance has returned. Then I would need the stats calls to be in something like a while loop that would not call stats until all of the values were 1. In other words, something needs to loop until all the values are 1 and then move on to the stats calls.

This would be relatively easy in cpp, but I really have no idea in bash. Is there a standard way to do this in a shell script? I can't imagine I am the first person to run across this.

Suggestions would be appreciated.

LMHmedchem

Hi, the wait command should be sufficient. You could try that and run emergent -nogui -p $ScriptLoc/$PROJ fold_tag=f4 in the background as well and leave out the sleep 240... and likewise for the rest of the script with a second wait statement after the statistics scripts that run in the background..

So something like this:

Code:

# learn on f0
emergent -nogui -p $ScriptLoc/$PROJ fold_tag=f0 &
sleep 5

# learn on f1
emergent -nogui -p $ScriptLoc/$PROJ fold_tag=f1 &
sleep 5

# learn on f2
emergent -nogui -p $ScriptLoc/$PROJ fold_tag=f2 &
sleep 5

# learn on f3
emergent -nogui -p $ScriptLoc/$PROJ fold_tag=f3 &
sleep 5

# learn on f4
emergent -nogui -p $ScriptLoc/$PROJ fold_tag=f4 &

# wait 4 minutes to make sure that processing has finished for all folds
wait

echo "statistics"  
./stats4A_04.sh $BATCH 0 &
./stats4A_04.sh $BATCH 1 &
./stats4A_04.sh $BATCH 2 &
./stats4A_04.sh $BATCH 3 &
./stats4A_04.sh $BATCH 4 &
wait

I like to 'cat' on the inherited stdout and stderr:

Code:

( whatever ) 2>&1 | cat >>log_file

'wait' only waits on child(ren), not grandchild(ren) and beyond.

Scrutinizer,
Thanks, that will simplify some other things as well. These sets do not always have the same number of instances and it is a pain to add logic to decide which instance doesn't get the & at the end of the call. That makes this hard to do in a loop.

DGPickett,
It sounds like you are going to read a file to figure out when a command has returned. I'm not sure how that would get integrated into this script. What would you be looking for in the log file to know that what ever you were running was finished? I though about checking for the existence of the last output file written by each instance, but those files exist along time before the app is done writing to them. For reference, the script I posted takes about 12 hours to finish.

LMHmedchem

No! All the children inside () for all generations inherit stdout and stderr from (), and when the last of them exists, the pipe into cat gets EOF and cat terminates. Nothing has to be written (but centralized logging is good). None of the commands inside () has to redirect stdout or stderr to get their errors logged, so the scripts are shorter, far less cluttered.