UNIX for Dummies Questions & Answers

Is it possible for a group of servers to monitor each other and then send an alert if one of them is no longer 'alive'?

Or if its easier have one server that monitors the other five and then sends an alert.

If so how would this be done?

Thanks

It sort of depends what you want to use to alert, how many alerts you will tollerate in the event of a problem, and how vital it is that you get the alert.

Having each server watch the other 4 will mean that it's very likely you will hear about it if one of them fails, but you will get 4 messages about one machine. If you have a network interruption, you might get 20 messages as each server tells you the other 4 are dead. This can get quite bad if you plan to scale this up.

Having just one server watch the other 4 means that if that one server fails, you won't know. Further more, if it fails then another goes, you won't hear about that either. On the up-side, if one server fails, you only get one message. If the network fails, you only get 4 messages.

You can pick something halfway between (eg have two server watch everything else, or have two servers watch each other and 1 or 2 other servers) to find the balance you need.

A quick framework of what you'd probably want to run:

Code:

#!/bin/sh
while true
do
  for host in $HOSTLIST
  do
    if ping $host
    then
      echo "`date` $host ok" >> log
    else
      echo "`date` $host NOT OK" >> log
      echo "Hi, `hostname` here to tell you that its all gone Pete Tong on $host" | mail_prog_of_your_choice
      echo "`hostname` says: $host has left the building" | your_favourite_pager_or_sms_gateway
      /cool/noises/play_alarming_sounds aaawuuuuga.au
    fi
  done
  sleep $INTERVAL
done

You can try to implement a tool that will not only solve this problem, but also has potential to address many other areas associated with building a high availability infrastructure. The tool I'm referring to is called "heartbeat".

Heartbeat is a publicly available package written by Alan Robertson. Heartbeat monitors system for certain event and can take actions based on those events. It can look to see if a particular service is running and if it isn't it can send an alert and/or automatically restart the service on a different machine. It also has the capability of transferring ownership of a shared IP address between nodes in a "heartbeat cluster".

Just do a google search for configuration details. It takes a bit to configure but it is a really cool solution for any HA environment.

Vassilios

Thanks.

I'll have a look at this.