AIX

I'd take a different starting point. Instead of looking for possible failures I'd define what resources have to be up and running to say that the cluster node is ok. If you keep in mind that the final goal of a cluster is to garantee the availability of a service and not the detection of errors your script may look a bit different, while the list of resources is pretty much what XrAy wrote.

Yes, exactly. There are more possible failure modes than one brain can imagine, but a very finite list of things your cluster is supposed to be providing and resources it uses to run.

Make sure you are pining the Persistent IP and NOT the Service IP, because Service IP will jump between the nodes, whereas persistent IP is hard bounded to the node.

You can check the cluster services, and the cluster state, and write a wrapper script to send an email if any of those goes south.
And ofcourse taking into consideration all the valuable suggestions given by forum members.

The backbone of the highavailability is the nodes in a cluster checking each other so you should look into heartbeats. They are usually implemented by sharing a disk space, like a concurrent accessable VG, which is rather small and have the nodes write in there some bits and by the freshness of it the nodes can decide who is still up and alive.
Additionally there is heartbeating via network interfaces. Some even use or used serial interfaces etc.
This is an important part of HACMP/PowerHA and other Cluster Technologies.

Have a look here:
Heartbeating in HACMP - AIX 6.1 Information Center