[Howto] Update AIX in HACMP cluster-nodes


 
Thread Tools Search this Thread
Operating Systems AIX [Howto] Update AIX in HACMP cluster-nodes
# 1  
Old 03-09-2013
[Howto] Update AIX in HACMP cluster-nodes

As i have updated a lot of HACMP-nodes lately the question arises how to do it with minimal downtime. Of course it is easily possible to have a downtime and do the version update during this. In the best of worlds you always get the downtime you need - unfortunately we have yet to find this best of worlds.

The following procedure is proven to work with AIX 5.3, 6.x and 7.x and associated HACMP/PowerHA versions. It needs only one takeover, so the downtime is from somewhere from under a minute to some minutes, depending on the nature of your resource group(s).

Communications in HACMP happens via RSCT and for a cluster to work the version of the RSCT-packages have to be in sync. Fortunately it is easy to update the RSCT independent of the rest of the OS. This is what this procedure depends on. We will consider a dual-node cluster with an active and a standby-system (rotating cluster), but the procedure can easily be adapted to other cluster-architectures.
  • Stop the clustermanager on the standby-node. This will end the cluster-communication. The remaining node will be on its own.

  • Update the RSCT-packages on both nodes. It won't matter that the communication path over the RSCT-daemons will be disrupted, because there is nobody to communicate with anyways.

  • Optional step: If you are of the well and truly paranoid type (like me) you can now restart the clustermanager on the standby-node and do a cluster-synchronization. I never experienced any problems when i tried this procedure in a test-environment and skipped this step, but i still feel better to do it when working on a PROD-system.

  • Stop the clustermanager on the standby-system again and update the rest of AIX and/or HACMP. Because you made sure the RSCT-daemons are already updated and at a equal version it won't do any harm if the versions of the other packages are different.

  • Once the standby-system has finished the update restart cluster-services and move the resource-group to the standby-system. This takeover will be your downtime.

  • Update now the remaining node after shutting down cluster-services. After the update finished restart cluster-services and do a cluster-synchronization. You are finished.

I hope this helps.

bakunin
# 2  
Old 03-11-2013
So, is the HACMP as a whole never down, just degraded to fewer nodes? It seems like with a HA cluster, you can update hosts in rotation and return them to the pool, so it is only down on host at a time.
# 3  
Old 03-11-2013
Quote:
Originally Posted by DGPickett
So, is the HACMP as a whole never down, just degraded to fewer nodes?
Yes. Exactly this is the point.

Quote:
Originally Posted by DGPickett
It seems like with a HA cluster, you can update hosts in rotation and return them to the pool, so it is only down on host at a time.
Yes and no. The point is that the HA-communication is done via RSCT and the versions of the RSCT packages have to be consistent throughout the cluster at any time. This is why you have to split up the cluster into single nodes at one point (precisely the point where you update the RSCT). During this phase communication would not be possible. But as each node is single at this time it doesn't recognize this inability to communicate.

I hope this helps.

bakunin
# 4  
Old 03-12-2013
It is sad the HA version n+ cannot discover and talk to version n as well as, when available, version n+. Backward compatability is a pretty common theme in the industry for many decades. They were sloppy in their requirements? No message version in the messaging?
# 5  
Old 03-12-2013
Quote:
Originally Posted by bakunin
The following procedure is proven to work with AIX 5.3, 6.x and 7.x and associated HACMP/PowerHA versions. It needs only one takeover, so the downtime is from somewhere from under a minute to some minutes, depending on the nature of your resource group(s).

Communications in HACMP happens via RSCT and for a cluster to work the version of the RSCT-packages have to be in sync. Fortunately it is easy to update the RSCT independent of the rest of the OS.
... snip ...
I hope this helps.

bakunin
Looks good - however, have you also verified this with an update to SystemMirror (aka PowerHA v7?). As I understand it, SystemMirror is not using (only?) RSCT - but is using CAA (Cluster Aware AIX) for communication, topology and heartbeats. -- I do not do much with SystemMirror so I am asking - anyone - just to be sure someone does not get surprised when working with/updating to SystemMirror.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. AIX

HACMP - two nodes - take too long to sync

HI Admin, I have running 2 node HACMP cluster- AIX 6.1. I just set it up. It does sync completely without any errors. But it take too long to sync. More than 30 mins... any reasons? Where can I start looking into ? Same network, same Subnet (1 Reply)
Discussion started by: snchaudhari2
1 Replies

2. Red Hat

RedHat Cluster: Nodes won't see each other

Hi All; I try to build a Redhat Cluster (CentOS 6) on vmware. But each node sees the other down like: # clustat Cluster Status for mycluster @ Wed Apr 8 11:01:38 2015 Member Status: Quorate Member Name ID Status ------ ---- ... (1 Reply)
Discussion started by: Meacham12
1 Replies

3. UNIX for Advanced & Expert Users

Arbitrator for 2 nodes ocfs cluster

Is there any way to create a arbitrary node for ocfs2 on a virtual machine (others are physical servers) so it won't go panic when one of physical server goes down? This is for load balanced application servers. Any setting example or tips? Thanks. (0 Replies)
Discussion started by: malayo
0 Replies

4. AIX

Re-cluster 2 HACMP 5.2 nodes

Hi, A customer I'm supporting once upon a time broke their 2 cluster node database servers so they could use the 2nd standby node for something else. Now sometime later they want to bring the 2nd node back into the cluster for resilance. Problem is there are now 3 VG's that have been set-up... (1 Reply)
Discussion started by: elcounto
1 Replies

5. Solaris

What is the procedure to reboot cluster nodes

Hi we have 2 solaris 10 servers in veritas cluster. also we have oracle cluster on the database end. now we have a requirement to reboot both the servers as it has been running for more than a year. can any one tell what is the procedure to bring down the cluster services in both the nodes... (7 Replies)
Discussion started by: newtoaixos
7 Replies

6. AIX

Make system backup for 2 nodes HACMP cluster

Hi all, I was wondering if someone direct me in how to Make system backup for 2 nodes HACMP cluster ( system image ) . What are the consideration for this task (3 Replies)
Discussion started by: h@foorsa.biz
3 Replies

7. Shell Programming and Scripting

Run script on HACMP nodes?

Hello All, Anybody knows how can I run script on the AIX HACMP offline node, without logon a offline node? I would like to run a script on the online node and at same time or after the online node on the offline node. Any IDEA? :confused: (3 Replies)
Discussion started by: kalaso
3 Replies

8. Emergency UNIX and Linux Support

Rebooting 3 to 1 Cluster nodes.

hello Gurus, My current set up is 3 to 1 Cluster (SUN Cluster 3.2) running oracle database. Task is to reboot the servers. My query is about the procedure to do the same. My understanding is suspend the databases to avoid switchover. Then execute the command scshutdown to down the cluster... (4 Replies)
Discussion started by: EmbedUX
4 Replies

9. AIX

Aix hacmp cluster question (oracle & sap)

Hello, I was wondering if I have 3 nodes (A, B, C) all configured to startup with HACMP, but I would like to configure HACMP in such a way: 1) Node B should startup first. After the cluster successfully starts up and mounts all the filesystems, then 2) Node A, and Node C should startup ! ... (4 Replies)
Discussion started by: filosophizer
4 Replies

10. UNIX for Advanced & Expert Users

Howto upgrade AIX to Level Update 4320-02_AIX_ML

Hi, Installing a new version of PROGRESS DB software 9.1c requires us to upgrade AIX to Level Update 4320-02_AIX_ML ( http://techweb.progress.com/kbase/19779 ) How would a do this savely. This is what i have already. 1) take a system backup. 2) find the level we are on right now. #... (1 Reply)
Discussion started by: progressdll
1 Replies
Login or Register to Ask a Question