Shell Programming and Scripting

Hi

Is there a way to count how many processes a script has started, count how many of these have finished, and make the script wait if their difference goes over a given threshold?

I am using a script to repeatedly execute a code (~100x) which converts 2 data files into one .plt which is in the format for Tecplot. These files are large, 300-1000MB.
The code reads an input file which is altered before each instance of the code is executed. I've done this by hand (never again), and could get about 5-6 instances going at a time (ie the 1st process would finish before I could start the 7th). (The input is read at the very beginning so I can alter it while code is running.)
My problem is once I scripted this process, MANY instances can get going at once, and the computer gets bogged down. I inserted a 'sleep' command, but this isn't robust enough, and depending on other loads on the computer the same problem can happen.

see script below

Thanks!
drbones

Code:

 1 #!/usr/bin/env tcsh
  2 foreach name (file_a_01*0)
  3 echo $name
  4
  5 set number=`echo $name | sed s/file_a_//`
  6 echo $number
  7
  8 set scrfile=`echo scr | sed s/r/r$number/`
  9 sed '2 s/[0-9]*/'$number'/' <code.in >new
 10 mv new code.in
 11 ./code.exe>$scrfile&
 12 sleep 20
 13 end

Hi.

You could use wait.

i.e.

Code:

% date; sleep 20 &; sleep 5 &; wait; date
[1] 29009
[2] 29010
Fri 14 May 2010 01:41:51 CEST
Fri 14 May 2010 01:42:11 CEST
[2]  + Done                          sleep 5
[1]  + Done                          ( date; sleep 20 )

(in a good shell (i.e. not csh), you have better options for job control!)

if your script is sufficiently self-aware...in a different shell than the one you've shown...you can go ahead and supply a self-referencing function to grep the ps for itself, to get a count and/or check for file opens (fuser) on the dependent files.

This was actually raised just recently too...as was the deprecated status of the C shell.

HTH

Thanks for the quick replies.

yes yes, I've heard all the bashing of csh (no pun intended), unfortunately I've been encouraged by my advisor (i'm a in a phd program) to use tcsh. I've only started using scripts over the past week, and instantly see their value. However, I've very new to unix and am not familiar enough with functions/terminology for the arguments against csh to be very convincing to me.

As for the wait command. I believe I could figure out how to start a few instances, but would have to wait for all of these to complete, before starting another batch. I'd like to get 5 going, then start the sixth as soon as the first finishes, etc. Am I misinterpreting the functionality of 'wait'?

As for grepping 'ps' this was something I had thought about, or even checking out 'uptime' (i'm working with multiple cores, and the code is parallel). I'm pretty sure I could somehow use grepped data to control a loop, which would be like a dynamic sleep command, but was looking for something more direct, this seems like a work-around.

Any other suggestions/advice?

curleb, you mentioned this was brought up recently. Could you direct me to the thread?

Thanks again
drbones

Speaking of bash... https://www.unix.com/unix-dummies-que...can-spawn.html