I have troubles making clstat work. All the "usual suspects" have been covered but still no luck. The topology is a two-node active/passive with only one network-interface (it is a test-setup). The application running is SAP with DB/2 as database. We do not use SmartAssists or other gadgets.
Here are the OS and HACMP-versions:
cldump works and all other cluster services are working as expected too. Alas, calling clstat:
I followed this procedure and double-checked everything mentioned there:
I also made sure the services are up and snmpd is the correct one:
The loopback-addresses for IPv6 are there in the /etc/hosts:
In the cited document it is mentioned to remove the comments in /etc/snmpdv3.conf as a last-ditch effort which i did. The services were restarted as described there and finally the whole system rebooted. I also did a cluster verification and synchronisation (in fact several times, before and after the reboot).
To be honest i am out of ideas what i still could do.
I know nothing about AIX, but if the implementation of the snmp protocol is anything like elsewhere(so there may be some huge faults in my understanding), consider:
Are there required MIB lists missing as a startup parameter for snmpd?
'Failed to retrieve' can alternatively be rendered as 'do not know how'. MIB lists provide the know how. Or. It can mean 'permission denied'. So I assume permissions strings have not been changed from default. And your UDP stack/ports are all up correctly?
I
Are there required MIB lists missing as a startup parameter for snmpd?
Thank you, Jim.
In fact all the MIB settings are in place (this is basically what the mentioned entries in /etc/snmpdv3.conf do) and the quoted line with the snmpinfo command proves that snmp is up and working as expected. I could have (and in fact - have) started snmpwalk instead and it shows the whole MIB tree for HACMP being in place. The listing is quite long so i didn't post it but in fact it is there.
In addition, if SNMP would not be configured correctly in respect to HACMP then the cldump should also not work, but does so. This is why i believe that SNMP is not the problem here but it is the common problem if clstat is not working so i posted the respective info beforehand.
As it is, no. This is a test cluster for the latest AIX/PowerHA version and its integration with SAP.
Quote:
Originally Posted by igalvarez
If not, which level of AIX do you have?
See post #1, the output of the "oslevel" command.
Quote:
Originally Posted by igalvarez
It's an 'forever' old issue on hacmp.powerHA
Not to my knowledge. I have about 50 other clusters in my environment (mostly HACMP 6.x and 5.x, but also a few on 7.x, OS versions are 6.1-7.1.3), and "clstat" is working on any of them. I use to check cluster statii with "cldump" so i commonly do not use clstat, but i would like to understand why it is not working - just out of curiosity.
Quote:
Originally Posted by igalvarez
Did you check on support if there any efix?
I would do so but right now i do not even understand where the problem is. If i could point to a certain fileset as the culprit i would try to get an update/efix/whatever or open an PMR, but i am not sure if there is anything left i could do before. There is no point in issuing a software call only to learn that "just do this, that and that to make it work as expected".
Quote:
Originally Posted by igalvarez
I remember we solved this issue with an APAR.
I'll be thankful if you could tell what the issue was because right now i don't even understand where the problem is.
we have got this error from time to time in our old AIX 6.1 (powerHA 6.1 GLVM) clusters. In deed last week we had to upgrade nodes from AIX 6.1TL6 to TL9 because a problem with clstat/cldump. But this is not your problem..
The steps we use here for all powerHA 6.1 clusters, sure are the same on your link above, are:
Really sorry I can not help in this case...
Last edited by igalvarez; 10-29-2014 at 09:21 AM..
Finally i found a "solution" to my problem: install a even newer version. As it seems the version i used was somewhat differently abled, as i believe the politically correct euphemism for "buggy" is. (A big THANK YOU goes to IBM for letting me do the beta-testing of software i thought to have purchased. I only bought a cluster-software but got a built-in adventure game at no cost.)
Here is what i did: first, install the latest AIX release (AIX 7.1, TL3 SP3):
This i did on both nodes. I am not sure if this was necessary, but together with the other changes (see below) it did the job. Next was to update the cluster software itself:
After this (and of course a reboot) i did a final cluster verification, then started the cluster without any problems. SNMP (and, as far as i can see, everything else) was working as expected. All in all it took me about 25 minutes per node, most of it can be done in parallel if the cluster can be stopped. Plan about 30-60 minutes for the whole update if you have the resources ready from the NIM server and everything else working.
HACMP two-node cluster with mirrored LVM.
HACMP two-node cluster with two SAN storages mirrored using LVM. Configured 2 disk heartbeat networks - 1 per each SAN storage. While performing redundancy tests. Once one of SAN storage is down - cluster is going to ERROR state. What are the guidelines... (2 Replies)
Hi all,
I remember way back in some old environment, having the HA cluster services not being started automatically at startup, ie. no entry in /etc/inittab.
I remember reason was (taken a 2 node active/passive cluster), to avoid having a backup node being booted, so that it will not... (4 Replies)
Hi,
A customer I'm supporting once upon a time broke their 2 cluster node database servers so they could use the 2nd standby node for something else. Now sometime later they want to bring the 2nd node back into the cluster for resilance. Problem is there are now 3 VG's that have been set-up... (1 Reply)
As i have updated a lot of HACMP-nodes lately the question arises how to do it with minimal downtime. Of course it is easily possible to have a downtime and do the version update during this. In the best of worlds you always get the downtime you need - unfortunately we have yet to find this best of... (4 Replies)
Hi
I'm a little rusty with HACMP, but wanted to find out if it is possible to remove a disk heartbeat network from a running HACMP cluster.
Reason is, I need to migrate all the SAN disk, so the current heartbeat disk will be disappearing. Ideally, I'd like to avoid taking the cluster down to... (2 Replies)
Hi,
I have a IBM Power series machine that has 2 VIOs and hosting 20 LPARS.
I have two LPARs on which GPFS is configured (4-5 disks)
Now these two LPARs need to be configured for HACMP (PowerHA) as well.
What is recommended? Is it possible that HACMP can be done on this config or do i... (1 Reply)
Hi all,
I was wondering if someone direct me in how to Make system backup for 2 nodes HACMP cluster ( system image ) .
What are the consideration for this task (3 Replies)
Hi
What is the procedure to upgrade the MQ from 6 to 7 in aix hacmp cluster. Do i need to bring down the cluster
services running in both the nodes and then give #smitty installp in both the nodes separately. Please assist... (0 Replies)
This post just as a follow-up for thread https://www.unix.com/aix/115548-hacmp-5-4-aix-5300-10-not-working.html: there was a bug in the clcomdES that would cause the Two-Node-Cluster-Configuration-Assistant to fail even with a correct TCP/IP adapter setup. That affected HACMP 5.4.1 in combinatin... (0 Replies)
Hello,
I was wondering if I have 3 nodes (A, B, C) all configured to startup with HACMP, but I would like to configure HACMP in such a way:
1) Node B should startup first. After the cluster successfully starts up and mounts all the filesystems, then
2) Node A, and Node C should startup !
... (4 Replies)