Crash dump and Panic message : RSCT Dead Man Switch Timeout for HACMP; halting non-responsive node
Dear all
i have two aix system
-Model : P770
-OS version: AIX 6.1
-patch level : 6100-07-04-1216
-ha version : HACMP v 6.1.0.8
-host : A, B
last Wednesday, my B system suddenly went down with crash dump. after 1 minute, A system went down with crash dump. I checked the dump of A-system using kdb command and found the following: PANIC MESSAGES:
RSCT Dead Man Switch Timeout for HACMP; halting non-responsive node
So, I was convinced that the phrase was the cause of the down of A-system. Is my judgment correct?
And I looked for what dead man switch is. but it was not well understood.
Why did the dead man switch bring down A-system?
hi Neo
Thank you for your reply.
as you mention, errpt log on the A-system node as follow,
when I checked the logs, the dead man switch was triggered by the heavy io, which seems to cause the system to crash.
please confirm it for me.
Could this be that whatever is supplying the common disk used to keep heartbeat failed? That way, both nodes would be unable to keep updating the shared disk and the usual response is to terminate all services to avoid getting in the way, i.e. to panic/abort. We've had a Oracle RAC database cluster do this before. not pretty, but it is the best course of action to avoid damage.
Hi Neo and rbattel
my system are using GPFS(archive directory) and raw devices with Oracle RAC on the hacmp.
I think the reason why A-system has down itself is that B-system is down due to system bug and all oracle sessions of B-system node are moved to A-system, which causes huge IO on A-system. The sync time of the A-system was slowed down, and as a result, the system was down when the dead man switch limit was reached.
You lost all heartbeats from node 1 to node 2 - thats the reason for the crash. This might happen when your system is simply too busy - but since you should have both heartbeat on disk and heartbeat via network, you should think that there is time enough to send at least one every couple of seconds, Your cluster heartbeat settings might be too tight - giving it more time for the heartbeat might help preventing this issue in the future.
Just out of curiosity - using GPFS and HACMP and RAC on the same systems appears to me to be a completely unnecessary setup, as you are running essentially 3 different cluster products on a system when RAC alone would suffice. Why ?
Hi Experts,
I have configured HP-UX Service Guard cluster and it dumps crash every time i reboot a cluster node. Can anyone please help me to prevent these unnecessary crash dumps at the time of rebooting SG cluster node?
Thanks in advance.
Vaishey (2 Replies)
Hi
I had an active passive cluster. Node A went down and all resource groups moved to Node B.
Now we brought up Node A. What is the procedure to bring everything back to Node A.
Node A #lssrc -a | grep cl
clcomdES clcomdES 323782 active
clstrmgrES cluster... (9 Replies)
Hi Guys,
I have to design a multinode hacmp cluster and am not sure if the design I am thinking of makes any sense.
I have to make an environment that currently resides on 5 nodes more resilient but I have the constrain of only having 4 frames. In addition the business doesnt want to pay for... (7 Replies)
Hi.
I have started heartbeat on two redhat servers. Using eth0.
Before I start heartbeat I can ping the two server to each other.
Once I start heartbeat both the server become active as they both have warnings that the other node is dead.
Also I am not able to ping each other. After stopping... (1 Reply)
Hi Guys,
I have two nodes clustered. Each node is AIX 5.2 & they are clustered with HACMP 5.2. The mode of the cluster is Active/Passive which mean one node is the Active node & have all resource groups on it & the 2nd node is standby.
Last Monday I noted that all resource groupes have been... (2 Replies)
hi friends,
i know that when there is a crash then that memory image is
put into /var/adm/crash
but if the system hangs up and if i have access to console of
that machine then how can i take the crash dump manully.
thanks (2 Replies)