It sort of depends what you want to use to alert, how many alerts you will tollerate in the event of a problem, and how vital it is that you get the alert.
Having each server watch the other 4 will mean that it's very likely you will hear about it if one of them fails, but you will get 4 messages about one machine. If you have a network interruption, you might get 20 messages as each server tells you the other 4 are dead. This can get quite bad if you plan to scale this up.
Having just one server watch the other 4 means that if that one server fails, you won't know. Further more, if it fails then another goes, you won't hear about that either. On the up-side, if one server fails, you only get one message. If the network fails, you only get 4 messages.
You can pick something halfway between (eg have two server watch everything else, or have two servers watch each other and 1 or 2 other servers) to find the balance you need.
A quick framework of what you'd probably want to run:
Code:
#!/bin/sh
while true
do
for host in $HOSTLIST
do
if ping $host
then
echo "`date` $host ok" >> log
else
echo "`date` $host NOT OK" >> log
echo "Hi, `hostname` here to tell you that its all gone Pete Tong on $host" | mail_prog_of_your_choice
echo "`hostname` says: $host has left the building" | your_favourite_pager_or_sms_gateway
/cool/noises/play_alarming_sounds aaawuuuuga.au
fi
done
sleep $INTERVAL
done