Keep a certain number of background processes running


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Keep a certain number of background processes running
# 1  
Old 01-27-2009
Keep a certain number of background processes running

I've got a bit of code I'm trying to work on...
What i want to happen is ... at all times have four parallel mysql dump and imports running.

I found the follow code snippet on the forum and modified it to work
by starting four concurrent processes but it waits until all four are done before starting the next four...

Code:
 
LIST='wmi_product wmi_patches employee actions jobs appversions os hardware detection_apps status wmi_networkadapterconfig wmi_logicaldisk'
let counter=0;
for i in $LIST
do
   echo "    Dumping and Importing table $i" " ( `date +'%x %X'` ) "
   $MYSQLDUMP -h$DBHOST -u$USER -p$PASSWORD -q --opt --single-transaction --add-drop-table $DB --tables $i | $MYSQL -u$DBuser -p$DBpass $DB > $ilog.txt &
    let counter=$counter+1
    echo "$counter%4" | bc
    if [ `echo "$counter%4" | bc` -eq 0 ] ; then
          wait
    fi
done;

Is there a way to do this?
Any help would be appreciated.
# 2  
Old 01-27-2009
I don't see any procedure to decrement your counter variable once inside the loop. So if one process stops, the counter should decrement by 1 and then you would launch another process.

But what confuses me, is how does your script ever get out of the 'if' test once you've reached four concurrent processes?

Last edited by rwuerth; 01-27-2009 at 01:08 PM.. Reason: correcting grammar & spelling.
# 3  
Old 01-27-2009
Also, you should probably set the counter by looking at how many MYSQL processes you have running, and not just counting how many have started. This way you automatically have the current number of processes needed.

So you could use (depending upon shell):

Code:
counter=`ps -f | grep -c $MYSQLDUMP`

# 4  
Old 01-27-2009
I had a similar problem - submit up to and including four background jobs, and when one or more finish, built back up to the max.

Code:
while read LINE                          # go through the output file from the view one line at a time
do

### Commented a lot of code - but basically, based on the LINE, 
###   set up the job with export statements

###   the secret is the jobs command gives a list of background jobs to this process
###   then the wc -l tells me how many

###   if it is >= 4, it goes to sleep for 30 seconds, then checks the count again

    # getting here, we want to submit the job to run, up to the limit of 4 jobs running at one time
    # See if there are 4 jobs running right now
      while [[ $( jobs | wc -l ) -ge 4 ]]
      do
        echo_f going to sleep now at $(date)
        sleep 30
      done

      /avidyn/scripts/avd_runit_002.ksh 1>/dev/null 2>&1 &   # parameters are passed by the export statements

done < /avidyn/tmp/avd_runit_001_jobs_to_run.txt

###  Not show after the done, is a "wait" command, to wait until all background jobs are finished.

# 5  
Old 03-12-2009
Here's what I wrote to accomplish this

Code:
 
echo $$ >/tmp/caller.pid
HOSTFROM=192.168.0.3
HOSTTO=192.168.0.2
DBFROM=hronline
DBTO=hronline
DBUSER=admin
DBPASSWD=password
TABLES1=`mysql -u$DBUSER -p$DBPASSWD -h $HOSTFROM -e "SELECT table_name as 'removebeforeuse' FROM information_schema.tables WHERE table_schema='$DBFROM';" |grep -v removebeforeuse`
TABLES=( ${TABLES1} )
CALLERPID=`cat /tmp/caller.pid`
COUNTER=0
while [ $COUNTER -lt ${#TABLES[@]} ]; do
    NUMPROC=` ps -ef | grep $CALLERPID | grep -v grep | grep -v bash | wc -l `
    if [ $NUMPROC -lt 5 ]; then
        echo Dumping table $COUNTER of ${#TABLES[@]} -- ${TABLES[$COUNTER]} Processes running concurrently: $NUMPROC
        mysqldump -v -u$DBUSER -p$DBPASSWD -q -h $HOSTFROM -e -C $DBFROM ${TABLES[$COUNTER]} | mysql -u$DBUSER -p$DBPASSWD -h  $HOSTTO $DBTO &
        sleep 3;
        let COUNTER=COUNTER+1
    fi
done
rm -f /tmp/caller.pid

# 6  
Old 03-13-2009
Another way you can try to check is:

1. When you launch a job in background, the pid is in $!
2. Append this to running comma-sep list of PIDS.
3. Instead of checking using ps -ef |grep etc. (which is prone to errors due to false match etc.+more resource intensive), do ps -o pid="" -p <pid_list>

e.g
Code:
$ ps -o pid="" -p 7693,3351,25875,9999 |wc -w
     2

So you have the count of processes alive in background in a more accurate and inexpensive manner.
# 7  
Old 03-13-2009
Quote:
Originally Posted by dgob123
Code:
 
    let counter=$counter+1
    echo "$counter%4" | bc
    if [ `echo "$counter%4" | bc` -eq 0 ] ; then
          wait
    fi

Is there a way to do this?
Replace by:
Code:
while [ $(ps ax|grep -ic $MYSQLDUMP) -gt 4 ] ;do sleep 3;done

While 4+1(grep) sleep 3 seconds Smilie , or you can use pgrep but that's another story.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How do I find the number of processes running on root?

Is there a certain man command I'm missing here? I searched in ps but I couldn't find something that would give me the number of processes running on root. I only want to see the number of processes, not the processes itself. (2 Replies)
Discussion started by: l3monz
2 Replies

2. Shell Programming and Scripting

Need help on background processes

Hi, I have a schell script parent.ksh from which I am calling three background processes a.ksh,b.ksh and c.ksh. Once these three processes completes the next step in parent.ksh should execute. How to achieve this? Please help me.... Thanks... (1 Reply)
Discussion started by: ravinunna
1 Replies

3. UNIX for Dummies Questions & Answers

How do you print the number of processes that each user is currently running in Unix?

Ok, so I know there's a way to do this, but I've been trying to find out all afternoon with no luck. I think it should print out something like this: 1 bin 2 daemon 6 duo Where the numbers on the left are the number of processes being run by the user whose name is listed on the right. Is... (4 Replies)
Discussion started by: Duo11
4 Replies

4. UNIX for Dummies Questions & Answers

Query on Running Multiple processes in background

HI All , Pardon me for asking some very basic questions, I would be grateful if someone can help. I am trying to execute a shell script which runs multiple processes in background. It includes various operations like copying , DB operations etc etc. Now problem is that the complete script... (6 Replies)
Discussion started by: gpta_varun
6 Replies

5. Shell Programming and Scripting

How to Control Number of Processes Running

Hi Is there a way to count how many processes a script has started, count how many of these have finished, and make the script wait if their difference goes over a given threshold? I am using a script to repeatedly execute a code (~100x) which converts 2 data files into one .plt which is in... (4 Replies)
Discussion started by: drbones
4 Replies

6. Shell Programming and Scripting

Background Processes

Ok guys so I have my first dummy shell almost done except for one tiny part: I do not know how to run a process in the background, from the code! I already know how to do that in a normal shell: $ program & However, no clue when it comes to how to program that thing. :eek: A very... (2 Replies)
Discussion started by: Across
2 Replies

7. Shell Programming and Scripting

Waiting for an arbitrary background process (limiting number of jobs running)

Hi, I'm trying to write a script to decompress a directory full of files. The decompression commands can run in the background, so that many can run at once. But I want to limit the number running at any one time, so that I don't overload the machine. Something like this: n=0 for i in *.gz... (15 Replies)
Discussion started by: p.f.moore
15 Replies

8. Solaris

About running processes in background

Hi, I need to establish a procedure that will start an application in background each time my remote Solaris server is (re)started. This would be a kind of daemon. I am no sysadmin expert, so I am looking for pointers. How should I proceed? What are the main steps? Thanks, JVerstry (9 Replies)
Discussion started by: JVerstry
9 Replies

9. Shell Programming and Scripting

Determine Number of Processes Running

I am pretty new to unix, and I have a project to do. Part of the project asks me to determine the number of processes running and assign it to a variable. I know how to every part of the project but determine the number of processes running. How can I get just the number of processes... (4 Replies)
Discussion started by: wayne1411
4 Replies

10. Shell Programming and Scripting

Running two processes in background

hi there, here's what i need in my korn-shell: ... begin korn-shell script ... nohup process_A.ksh ; nohup process_B.ksh & ... "other stuff" ... end lorn-shell script in plain english i want process A and process B to run in the background so that the script can continue doing... (6 Replies)
Discussion started by: jacob_gs
6 Replies
Login or Register to Ask a Question