Help scripting to start, check, and restart processes


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help scripting to start, check, and restart processes
# 1  
Old 06-21-2011
Help scripting to start, check, and restart processes

Here it goes from my unexperienced point of view. I am using CentOS 5.6. I have a Java based server that needs to be running 24/7/365. To begin from the machine the server is on rebooting; I SSH in to a shell, cd to the server dir, screen -S server1, and execute ./exec (listed below) in the screen. This runs the Java server and restarts the it's process if/when stopped.

exec
Code:
#!/bin/bash
while true; do
java (*all variables*)
wait
done

I would like to automate this starting process with a script using a crontab job. This script(s) would need to do what I do manually to start the server, perform checks to make sure the 3 processes are always running (screen, exec, java) and be able to restart them in their appropriate environment (command line or screen).

I have tried the psybnc psybncchk script with some adjustments to try and do some of this, but my knowledge of scripting is very minimal.

cbchk
Code:
CBPATH=/home/*user*/*dir*

if test -r $CBPATH/cb.pid; then
    CBPID=$(cat $CBPATH/cb.pid)
    if $(kill -CHLD $CBPID >/dev/null 2>&1)
    then
	exit 0
    fi
fi
cd $CBPATH
SCREEN -S server1
./exec

When I kill all of the processes, then run the above script, the shell screen gets spammed with arrows and the server's starting info displays inside the never ending arrows. When I attach to the screen, Java is continually stating errors nonstop. I've also tried adding " & echo $! > cb.pid" after the ./exec to automate getting the PID, but that makes the screen unusable for some reason.

While exec is running, it's keeping the Java server running, but when all of the processes are killed, I'm not sure how to script the starting sequence and have it make sure everything stays running. Any and all help will be greatly appreciated. Thank you.

Edit: Edited for clarity, hopefully.

Last edited by MacG32; 06-21-2011 at 07:34 PM..
# 2  
Old 06-21-2011
And this has to stay inside of a screen session because it's interactive, or just a way to view the output?

running your "exec" script in the background is not sufficient?

shell$ ./exec &

Then you want a crontab to make sure 'exec' is running?
This User Gave Thanks to neutronscott For This Post:
# 3  
Old 06-21-2011
Quote:
Originally Posted by neutronscott
And this has to stay inside of a screen session because it's interactive, or just a way to view the output?

running your "exec" script in the background is not sufficient?

shell$ ./exec &

Then you want a crontab to make sure 'exec' is running?
The screen is interactive and the only way to view the output.

The exec script covers keeping the Java server running only if it doesn't consume all available memory and the OS doesn't kill all of the processes.

The exec script is executed in the screen to run, view, and interact with the server with ./exec .

When the machine the server is on is rebooted, I would like a script to cd to the server dir, start a screen with screen -S server1, then execute the exec script within the screen with ./exec . Then have the script check to make sure all of the processes are running. If not, restart them.

Sorry for not being able to explain everything very precise or easily.
# 4  
Old 06-21-2011
Having screen as part of the equation at first seemed difficult for me. Then I read the manual...

Firstly, we create the crontab-able script which checks if the screen session is started.

Code:
#!/bin/sh
# chkscreen: checks if a screen session is running.
SESSION="server1"
DAEMON="screen -d -m -S $SESSION /home/mute/test/java-daemon.sh"

# does the session exist?
screen -r $SESSION -ls -q 2>&1 >/dev/null

if [ $? -le 10 ]; then
        echo "Restarting $DAEMON"
        $DAEMON
fi

Then java-daemon.sh is your while looping deal, which hopefully doesn't die since it'd restart your java faster than waiting for cron job.

Code:
#!/bin/sh
# java-daemon.sh: keeps restarting a buggy server... ;)

while true; do
./java
done

So then you'll place chkscreen inside of /etc/init.d/rc.local to start with the server, and inside of crontab.

mute@geek:~/test$ crontab - <<__EOF__
> */15 * * * * /home/mute/test/chkscreen
> __EOF__

Guaranteed to work on my machine!

Sorry I'm no guru myself but unless someone else replies this is what you're stuck with Smilie
This User Gave Thanks to neutronscott For This Post:
# 5  
Old 06-22-2011
Thank you very much neutronscott for your help, this has worked like a charm. Looks like I was way off. Thanks again. Smilie
# 6  
Old 06-23-2011
Quote:
Originally Posted by MacG32
Thank you very much neutronscott for your help, this has worked like a charm. Looks like I was way off. Thanks again. Smilie
There's just many different ways it can be solved. Instead of keeping a PID file (my original trial) I just ask screen if the session is running. It makes it so elegant, eh? Smilie

Thanks for the Thanks! Smilie

---------- Post updated 06-23-11 at 09:50 AM ---------- Previous update was 06-22-11 at 02:13 PM ----------

Oh man, I couldn't sleep last night because I was thinking I didn't test all cases. So first thing today I run 'chkscreen' while attached to 'server1' session and I was right. bug Smilie

Sorry. Using '-r' checks if there is a session you can attach to, which is false if you're already attached.

So change

Code:
# does the session exist?
screen -r $SESSION -ls -q 2>&1 >/dev/null

to read

Code:
# does the session exist?
screen -S $SESSION -ls -q 2>&1 >/dev/null

And it shouldn't start extra servers when you're viewing the screen anymore. Sorry.
This User Gave Thanks to neutronscott For This Post:
# 7  
Old 06-23-2011
In my case, the -r only produces 1 additional chkscreen and java-daemon.sh process, and only 1 Java server process. When I changed the -r to -S, it kept making chkscreen and java-daemon.sh processes and no additional Java server processes. That's why I was trying to get and use the PIDs, using the second script in the OP.

If Java uses all of the machine's memory, then the screen will be terminated. I'll no longer be attached to it, so the check for being or not being attached may not be the approach needed. I'll try and use what you've kindly given me and integrate something to get and check the PIDs.

No need to be sorry and lose sleep over this. It'll get worked out eventually and I'll post the final tweaked results then. Below is what I'm using. Thanks a million for all of your help neutronscott.

chkscr
Code:
#!/bin/sh
# chkscr: checks if a screen session is running.

SESSION="server1"
DAEMON="screen -d -m -S $SESSION /home/*server*/*local*/jd.sh"

# does the session exist?
screen -r $SESSION -ls -q 2>&1 >/dev/null

if [ $? -le 10 ]; then
        echo "Restarting $DAEMON"
        $DAEMON
fi

jd.sh
Code:
#!/bin/sh
# jd.sh: restarts a stopped server.

while true; do
java *server variables*
wait
done

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

Service restart and check if running

Hello, I'l like to create a script that restart a service (/etc/init.d/httpd restart) and also check if after restart the service is actually running. Sometimes it happen that at the first try the service fails to restart. Thanks (2 Replies)
Discussion started by: bazzola
2 Replies

2. Emergency UNIX and Linux Support

Check hung process and restart

Hi all I have networker running on a RHEL 5.7 and over time it hangs. So the solution backup team proposed is to check if the process is hung, to stop and start it. Unfortunately for me, the rc script only allows three commands, start, stop and status (no restart option) so I managed to set... (15 Replies)
Discussion started by: hedkandi
15 Replies

3. UNIX for Dummies Questions & Answers

Stop/Start vs. Restart

Is there any functional difference between: issuing separate stop/start commands like this; super (handler) (instance) stop super (handler) (instance) start versus issuing a single recycle command like this; super (handler) (instance) restart (3 Replies)
Discussion started by: Newbix
3 Replies

4. Solaris

How to start/stop processes

Please anyone tell me In my last interview the HR asks me how to monitor, start,stop & kill the various processes and subprocesses. Please anyone explain me clearly. It's my personal request (3 Replies)
Discussion started by: suneelieg
3 Replies

5. Shell Programming and Scripting

guarantee to start before restart...

Hi All, is there a way or script that i can check my AIX 5.3 OS will restart before i made restart? is there a script that can check all the startup files are ok before restarting. it is because i was stuck last time when i restart my PC because some startup files were missing:o. (2 Replies)
Discussion started by: malcomex999
2 Replies

6. Shell Programming and Scripting

Start Stop Restart

I'm wondering how I should make a script that can start, stop, and restart another script. What I need to be able to do, is start and stop a perl script from the command line. The easiest way of doing this seems to be to have another script, starting and stopping the other script. I have BASH,... (7 Replies)
Discussion started by: Bakes
7 Replies

7. UNIX for Advanced & Expert Users

Log 'syslog start/stop/restart' messages

How can I tell my syslog.conf to log "syslog start/stop/restart" messages on a Solaris box? (1 Reply)
Discussion started by: SunnyK
1 Replies

8. AIX

How to start/stop/restart NFS on AIX

Hi, Very new to aix How to start/stop/restart NFS on AIX thanks, (2 Replies)
Discussion started by: jredx
2 Replies

9. Shell Programming and Scripting

need help: shell script to restart apache when no. of processes keeps growing

I need a shell script to kill apache and restart it, in case the number of processes keeps growing. The logic is like the below, but I don't know how to get the number and neither the syntax. Could somebody kindly help? if no_of_processes (ps ax ¦ grep httpd) > 200 then killall httpd... (14 Replies)
Discussion started by: _joshua_
14 Replies

10. Shell Programming and Scripting

How can I make a program start up automatically after the computer restart/startup?

hi all How can I make a program start up automatically after the computer restart/startup in fedora? something like: ... Establish a shell then run some of command code. Thanks for Help!! (1 Reply)
Discussion started by: munna_dude
1 Replies
Login or Register to Ask a Question