Shell Programming and Scripting

It looks like it should work just fine (although it creates an unneeded process every time through the loop) if you want to completely ignore some command input read from your input file, never run those commands through my_job, and do not want any record of the fact that you chose to ignore those input file command lines.

Every additional process you run, slows down your system for all users on your system and slows down your script (and reduces the number of input lines you'll be able to process). This is an expected side effect when you need to run a process. It is just wasteful when you create processes that are not needed!

For example, consider changing:

Code:

           NUM_PROCS=`ps -u root | grep my_job | wc -l`

to:

Code:

           NUM_PROCS=`ps -u root | grep -c my_job`

changing:

Code:

        cat /home/gmark/rje/COMMANDS.csv | grep "A[BL][AE]" > ${LOCAL_CSV}
        > ${REMOTE_CSV}
        LOCAL_CSV_SIZE=`wc -l ${LOCAL_CSV} | sed "s;^ *;;" | sed "s; .*;;"`

to:

Code:

        grep "A[BL][AE]" < /home/gmark/rje/COMMANDS.csv > ${LOCAL_CSV}
        > ${REMOTE_CSV}
        LOCAL_CSV_SIZE=`wc -l < ${LOCAL_CSV}`

etc.

You still haven't given any indication about what type of system you're using, how many CPUs (or cores) are in your system, and what else might be running on your system that needs run while you're running this stuff. Without that information, I have to assume that setting MAX_NUM_PROCS to 400 will run significantly slower than setting MAX_NUM_PROCS to the number of cores on your system -2.

I would also assume that waiting for a job to complete once in a while instead of running a loop that runs ps and grep throttled only by a sleep 1 would consume considerably fewer resources and actually complete more jobs. (But of course that would require you to look at some of the other suggestions that have been made suggesting other ways to limit the number of background jobs you're running simultaneously.)

Some of the coding decisions I've made are to allow rapid prototyping and debugging, such as "read" statements in the loop to allow me to step through it slowly. Eventually this will all be removed or changed.

I think the idea of putting a "wait" command in there occasionally is a great idea!

Inside "my_job", I'm basically making several database queries and then using these to decide whether to make a hardware operation, which might take minutes. "my_job" leaves some output data and status in a small file that I go around periodically to collect, and possibly spawn a new "my_job" to respond to.

Is there a possibility of the "my_job" just hanging that there is a more elegant way of checking and purging? I'd hate to have a "wait" command in line that has 20 jobs all of which will complete and then the 21st stops the system by hanging.

Is there some way of handling this, such as a "wait" that has a "time limit"?

@ Don, might this mean counting cpus or threads?

Code:

$ grep -i cores /proc/cpuinfo              
cpu cores	: 4
...

4 - 2 = 2 paralell running jobs?

Or if counting threads:

Code:

$ grep -i cores /proc/cpuinfo -c
8
$ grep cpu[0-99] /proc/stat -c
8

8 - 2 = 6 paralell running jobs?

EDIT:
@ gmark99
Did you try:

Code:

sleep 15
sleep 0.5

CPU's.

To get a count, count 'core id' lines.

Code:

$ grep -c 'core id' /proc/cpuinfo

2

$

Keep in mind, once the jobs are in the background, they are running, regardless if you sleep in the foreground, so that has no effect.
Furthermore, it cant hurt the give the cpu's a second break between the jobs it executes.

Also, you could implement that check in the loop itself rather than in the background as well.
In that case, you might even be able to 'leave out' the sleep.

On the other hand, if the 'purge-job' takes several minutes, as you say, just to check if a process is running or not, you have a serious performance issue anyway.
Which could be either too many jobs, or a badly coded parser.
(EDIT2: on rethinking, i'm very sure the issue is here, since with every loop you start a background job lasting several minutes and doing db and cpu checks)

EDIT:
@ Corona, ok thank you.
So for you this would mean no background jobs then?
While i could do 6?

With any at least less modern hardware one should be able to run at least one background job, or is one not?

I'm not sure how you get "no background jobs" from "two cores". Without timesharing, two cores can run two things.

We could both run 30 if we wanted to, they'd just have to timeshare. I'd have each core running 15 jobs at a little less than 1/15th speed. You'd have each core running 5 jobs at a little less than 1/5th speed.

For maximum efficiency, and assuming each process keeps itself busy 100% of the time, don't exceed the number of cores.

Also, there's things which running things in parallel won't help with -- I/O like deleting files, for example. Two processes deleting from one disk doesn't make it spin twice as fast.

In my case, "my_job" does some database queries and decides to call a function that will result in a cross-processor message and then wait for a result to be returned.

I would still benefit from a lot of jobs in parallel if they're just waiting for results from another machine, right? This waiting for outside resources is the basic reason for my wanting to run so many jobs in background.

Am I missing anything, or am I on the right track?