Quote:
Originally Posted by
Tharsan
Ya actually the final aim is not to use the IBM cluster software (HACMP, powerHA)
So, basically you want to rewrite HACMP, yes? Why not, but be warned: there is a reason good cluster software is not coming in the dozens.
Let us see: a cluster is a device for making some "service" available even in cases of machines failing. So, what is a
service?
A "service" is an application you can reach under a certain network address, therefore you need:
one (or more) network addresses,
some filesystems with data,
some processes serving said service.
This, bound into a group, is called a
resource group in HACMP terminology.
You need also some device (say, a script, or whatever) telling you when the service is failing. Just checking some processes is problematic, because it could happen in some big software package that a certain process has to stop and another has to start as part of the normal operation. Therefore you need for every resource group a customised way of telling everything is good or not - a so-called
application monitor. In its simplest form it will indeed check some processes, but it can be much more sophisticated than that.
This was the "internal" supervision, taking place on one node. You also need an "external" supervision, where the passive node checks if the active node is still alive. This is done via
heartbeats, but is not always easy to tell, because if the service is not reachable via, say, network, this could mean that the node is failing or the connecting network is failing. Taking over in the first case corrects the problem while doing so in the second will achieve nothing. HACMP therefore uses network hearbeats, serial heartbeats and through shared disks (classically SCSI or SSA, nowadays FC networks) in parallel.
The cluster state which has to be avoided at all costs is the "split brain" condition: both nodes thinking they are primary and the other is failing. For this to avoid you need some means of shutting down a node as fast as possible.
shutdown will be too slow,
halt -q will be better and something like
cat /etc/hosts > /dev/kmem (not possible any more since AIX 5.3 ML 1) would be best (fastest). Because you need to be able to trigger it from outside HACMP has the
DMS (dead-man-switch), a kernel-extension which takes down the system real fast under certain conditions. While most of HACMP consists of scripts calling other scripts, this part is kernel-software. You will have to create such a thing too.
So far, off the top of my head. There is probably much more to say than what came to my mind right now, so just ask. I suggest reading the IBM redbooks about HACMP. Implementing a cluster software is a laudable effort, because even if you fail you will get to appreciate the problems it poses. If you even succeed, all the better.
I hope this helps.
bakunin