Shell Programming and Scripting

Here it goes from my unexperienced point of view. I am using CentOS 5.6. I have a Java based server that needs to be running 24/7/365. To begin from the machine the server is on rebooting; I SSH in to a shell, cd to the server dir, screen -S server1, and execute ./exec (listed below) in the screen. This runs the Java server and restarts the it's process if/when stopped.

exec

Code:

#!/bin/bash
while true; do
java (*all variables*)
wait
done

I would like to automate this starting process with a script using a crontab job. This script(s) would need to do what I do manually to start the server, perform checks to make sure the 3 processes are always running (screen, exec, java) and be able to restart them in their appropriate environment (command line or screen).

I have tried the psybnc psybncchk script with some adjustments to try and do some of this, but my knowledge of scripting is very minimal.

cbchk

Code:

CBPATH=/home/*user*/*dir*

if test -r $CBPATH/cb.pid; then
    CBPID=$(cat $CBPATH/cb.pid)
    if $(kill -CHLD $CBPID >/dev/null 2>&1)
    then
	exit 0
    fi
fi
cd $CBPATH
SCREEN -S server1
./exec

When I kill all of the processes, then run the above script, the shell screen gets spammed with arrows and the server's starting info displays inside the never ending arrows. When I attach to the screen, Java is continually stating errors nonstop. I've also tried adding " & echo $! > cb.pid" after the ./exec to automate getting the PID, but that makes the screen unusable for some reason.

While exec is running, it's keeping the Java server running, but when all of the processes are killed, I'm not sure how to script the starting sequence and have it make sure everything stays running. Any and all help will be greatly appreciated. Thank you.

Edit: Edited for clarity, hopefully.

And this has to stay inside of a screen session because it's interactive, or just a way to view the output?

running your "exec" script in the background is not sufficient?

shell$ ./exec &

Then you want a crontab to make sure 'exec' is running?

The screen is interactive and the only way to view the output.

The exec script covers keeping the Java server running only if it doesn't consume all available memory and the OS doesn't kill all of the processes.

The exec script is executed in the screen to run, view, and interact with the server with ./exec .

When the machine the server is on is rebooted, I would like a script to cd to the server dir, start a screen with screen -S server1, then execute the exec script within the screen with ./exec . Then have the script check to make sure all of the processes are running. If not, restart them.

Sorry for not being able to explain everything very precise or easily.

Having screen as part of the equation at first seemed difficult for me. Then I read the manual...

Firstly, we create the crontab-able script which checks if the screen session is started.

Code:

#!/bin/sh
# chkscreen: checks if a screen session is running.
SESSION="server1"
DAEMON="screen -d -m -S $SESSION /home/mute/test/java-daemon.sh"

# does the session exist?
screen -r $SESSION -ls -q 2>&1 >/dev/null

if [ $? -le 10 ]; then
        echo "Restarting $DAEMON"
        $DAEMON
fi

Then java-daemon.sh is your while looping deal, which hopefully doesn't die since it'd restart your java faster than waiting for cron job.

Code:

#!/bin/sh
# java-daemon.sh: keeps restarting a buggy server... ;)

while true; do
./java
done

So then you'll place chkscreen inside of /etc/init.d/rc.local to start with the server, and inside of crontab.

mute@geek:~/test$ crontab - <<__EOF__
> */15 * * * * /home/mute/test/chkscreen
> __EOF__

Guaranteed to work on my machine!

Sorry I'm no guru myself but unless someone else replies this is what you're stuck with

Thank you very much neutronscott for your help, this has worked like a charm. Looks like I was way off. Thanks again.

There's just many different ways it can be solved. Instead of keeping a PID file (my original trial) I just ask screen if the session is running. It makes it so elegant, eh?

Thanks for the Thanks!

---------- Post updated 06-23-11 at 09:50 AM ---------- Previous update was 06-22-11 at 02:13 PM ----------

Oh man, I couldn't sleep last night because I was thinking I didn't test all cases. So first thing today I run 'chkscreen' while attached to 'server1' session and I was right. bug

Sorry. Using '-r' checks if there is a session you can attach to, which is false if you're already attached.

So change

Code:

# does the session exist?
screen -r $SESSION -ls -q 2>&1 >/dev/null

to read

Code:

# does the session exist?
screen -S $SESSION -ls -q 2>&1 >/dev/null

And it shouldn't start extra servers when you're viewing the screen anymore. Sorry.

In my case, the -r only produces 1 additional chkscreen and java-daemon.sh process, and only 1 Java server process. When I changed the -r to -S, it kept making chkscreen and java-daemon.sh processes and no additional Java server processes. That's why I was trying to get and use the PIDs, using the second script in the OP.

If Java uses all of the machine's memory, then the screen will be terminated. I'll no longer be attached to it, so the check for being or not being attached may not be the approach needed. I'll try and use what you've kindly given me and integrate something to get and check the PIDs.

No need to be sorry and lose sleep over this. It'll get worked out eventually and I'll post the final tweaked results then. Below is what I'm using. Thanks a million for all of your help neutronscott.

chkscr

Code:

#!/bin/sh
# chkscr: checks if a screen session is running.

SESSION="server1"
DAEMON="screen -d -m -S $SESSION /home/*server*/*local*/jd.sh"

# does the session exist?
screen -r $SESSION -ls -q 2>&1 >/dev/null

if [ $? -le 10 ]; then
        echo "Restarting $DAEMON"
        $DAEMON
fi

jd.sh

Code:

#!/bin/sh
# jd.sh: restarts a stopped server.

while true; do
java *server variables*
wait
done