UNIX for Advanced & Expert Users

Hi,
I have a unix script that basically has a while loop inside which it checks Oracle database for certain records. If it finds the records, it does some processing and then goes back to the while loop. If it doesnot find any matching records, then it sleeps for 30 seconds and then goes back to the while loop to again check for existence of a new records.
We are supposed to have only one instance of this script running. The script is kicked off at the beginning of the week with the command sh scriptname.ksh &
It doesnot need to be started again until after server reboot the following week.
To ensure that not more than one instance of the script runs, we have a monitoring program that alerts us whenever multiple instances of the script are running due to either multiple users mistakenly starting a new instance of the script or some other reaosn. After seeing occassional alerts from this monitoring system that multiple instances of this script were running, I decided to see why the monitoring system was alerting us on multiple instances. So I wrote a temporary script that runs in an infinite loop and runs the command ps -ef|grep scriptname |grep -v grep every 2 secs. After analyzing the output of this test daemon that I wrote, I found that once in a while (eveyr 15 or 20 minutes), ps -ef returns two instances of the script.
(1) One instance has parent pid equal to 1
(2) The 2nd instance of this script has pid equal to the parent pid of the 1st process
Within a couple of seconds after that, my daemon shows that the 2nd instance has died and again there is only one instance.
So why does this 2nd instance get triggered by the parent process momentarily thereby misleading our monitoring system to believe that there are multiple instances of the script running.
Here is an output of the ps -ef run by my test daemon that runs every 2 seconds. As you can see from the output below, at 5:44:04 pm Aug 5, you can see 2 instances of the script running (in bold). 1st instance is parent script which has pid 12369. 2nd instance is Child script which has pid 5713 and whose parent pid is the pid of the parent script 12369. I have no idea how the child script with pid 5713 got triggered . myscriptname.ksh doesnot call itself. As mentioned before, it just runs in a while loop waiting for database changes and does some processing if there are database changes but it never calls itself anywhere. AS you can see below, within 2 secs at 5:44:06 PM, the child instance of the script with pid 5713 suddenly dies and we are back to just 1 instance again (pid 12369). This happens occassionally (maybe once every 15 minutes) but I donot know why. If you can explain the reason for this behavior of the parent process momentarily invoking a child process which calls the same script as the parent script and which dies within 2 secs, your help would be highly appreciated.
The script is running on SUSE Linux v10.3

Code:

Sun Aug  5 17:44:00 EDT 2012
ownerid     12359     1  0 04:10 ?        00:00:08 sh myscriptname.ksh
Sun Aug  5 17:44:02 EDT 2012
ownerid     12359     1  0 04:10 ?        00:00:08 sh myscriptname.ksh
Sun Aug  5 17:44:04 EDT 2012
ownerid      5713 12359  0 17:44 ?        00:00:00 sh myscriptname.ksh
ownerid     12359     1  0 04:10 ?        00:00:08 sh myscriptname.ksh
Sun Aug  5 17:44:06 EDT 2012
ownerid     12359     1  0 04:10 ?        00:00:08 sh myscriptname.ksh
Sun Aug  5 17:44:08 EDT 2012
ownerid     12359     1  0 04:10 ?        00:00:08 sh myscriptname.ksh

thanks

Moderator's Comments:

edit by bakunin: Please use code tags next time for your code and data.

when you use a pipe | the shell creates a subshell.
maybe your script should touch a file in /tmp and refuse to run again if that file exists. or write out a pid file, which you can test with kill -0. using ps | grep for process management is, as you see, very inadequate

Most likely, it's running a command in a subshell. If you want a more accurate assessment, post the code.

Regards,
Alister

After timing the momentary occurrance of these multiple process instance, I localized this problem to one line in the script where it calls an Informatica ETL repository commands using Informatica's UNIX command-line utility PMCMD.
I have the following in my script.

Code:

while: 
do
script code before pmcmd command
............
............
informaticapath/pmcmd startworkflow -sv ETL_SERVICENAME -d ETL_DOMAINNAME -u ETL_USERID -p ETL_PASSWORD -f ETL_FOLDERNAME -wait ETL_WORKFLOWNAME

script code after the pmcmd command
...............
..............
done

To summarize what is going on above, each time the pmcmd command is called within the while loop, takes about 5 minutes to complete processing about 1000 messages. After it has completed it goes back to the while loop and if there are more new messages to be processed, gets invoked again and so on. Note that the -wait option in PMCMD tells pmcmd to return to the shell only after it completes executing the ETL workflow. So until pmcmd completes, the shell will not execute the subsequent line in the script.

What I noticed is everytime pmcmd finishes executing and returns control to the shell to execute the subsequent line in the shell script, briefly for a second I see 2 instances of this script running as shown in the sample below (a parent process which is the currently running script and a child process that momentarily appears and then vanishes). Within a second the child process that just appeared, vanishes.

Code:

ownerid 5713 12359 0 17:44 ? 00:00:00 sh myscriptname.ksh
ownerid 12359 1 0 04:10 ? 00:00:08 sh myscriptname.ksh

Based on the -wait option to the PMCMD command it returns control to the shell only after it has finished executing the ETL WORKFLOW specified in the pmcmd command. After completion, at the time of returning control to the shell, why is the child process momentarily created ? Is this done by pmcmd ? Any inputs would be appreciated.

thanks

Moderator's Comments:

edit by bakunin: same as above, please use CODE-tags

I don't understand how you expect us to give you a clue, beyond what has already been said about subshells, when we haven't seen the code. Since it is a loop, the culprit could lurk before or after pmcmd.

Regards,
Alister

I just removed everything before and after the pmcmd and this is all I have now. And it still spawns a new process momentarily for a second just after the pmcmd exits

Code:

#!/usr/bin/ksh
while:
do

informaticapath/pmcmd startworkflow -sv ETL_SERVICENAME -d ETL_DOMAINNAME -u ETL_USERID -p ETL_PASSWORD -f ETL_FOLDERNAME -wait ETL_WORKFLOWNAME

done

PLease post what hardware you have (processors, memory etc.) and what Operating System you are running.
The ps command is unreliable (by design) on overloaded systems, but it usually misses processes or returns nothing rather than doubling up. We can't really comment withous seeing the script.

The construct ps -ef | grep scriptname | grep -v grep is horrendously inefficient and the most likely command to be rejected by the kernel on a heavily-loaded system.

Why not get the script to create a "PID file" just containing the PID of the script. Delete the file when the script ends. This is is technique used by many system scripts. You put the PID in the file in case you need to check whether that single PID is running (rather than trawling the entire process list). For general interlock, just check whether the file exists or not. Prevention is better than cure.