Topics: LVM, PowerHA / HACMP, System Admin
VGDA out of sync
With HACMP, you can run into the following error during a verification/synchronization:
WARNING: The LVM time stamp for shared volume group: testvg is inconsistent
with the time stamp in the VGDA for the following nodes: host01
To correct the above condition, run verification & synchronization with
"Automatically correct errors found during verification?" set to either 'Yes'
or 'Interactive'. The cluster must be down for the corrective action to run.
This can happen when you've added additional space to a logical volume/file system from the command line instead of using the smitty hacmp menu. But you certainly don't want to take down the entire HACMP cluster to solve this message.
First of all, you don't. The cluster will fail-over nicely anyway, without these VGDA's being in sync. But, still, it is an annoying warning, that you would like to get rid off.
Have a look at your shared logical volumes. By using the lsattr command, you can see if they are actually in sync or not:
host01 # lsattr -Z: -l testlv -a label -a copies -a size -a type -a strictness -FvalueWell, there you have it. One host reports testlv having a size of 806 LPs, the other says it's 809. Not good. You will run into this when you've used the extendlv and chfs commands to increase the size of a shared file system. You should have used the smitty menu.
/test:1:809:jfs2:y:
host02 # lsattr -Z: -l testlv -a label -a copies -a size -a type -a strictness -Fvalue
/test:1:806:jfs2:y:
The good thing is, HACMP will sync the VGDA's if you do some kind of logical volume operation through the smitty hacmp menu. So, either increase the size of a shared logical volume through the smitty menu with just one LP (and of course, also increase the size of the corresponding file system); Or, you can create an additional shared logical volume through smitty of just one LP, and then remove it again afterwards.
When you've done that, simply re-run the verification/synchronization, and you'll notice that the warning message is gone. Make sure you run the lsattr command again on your shared logical volumes on all the nodes in your cluster to confirm.
HACMP automatically runs a verification every night, usually around mid-night. With a very simple command you can check the status of this verification run:
# tail -10 /var/hacmp/log/clutils.log 2>/dev/null|grep detected|tail -1If this shows a returncode of 0, the cluster verification ran without any errors. Anything else, you'll have to investigate. You can use this command on all your HACMP clusters, allowing you to verify your HACMP cluster status every day.
With the following smitty menu you can change the time when the auto-verification runs and if it should produce debug output or not:
# smitty clautover.dialogYou can check with:
# odmget HACMPclusterBe aware that if you change the runtime of the auto-verification that you have to synchronize the cluster afterwards to update the other nodes in the cluster.
# odmget HACMPtimersvc
Topics: Fun, PowerHA / HACMP↑
HACMP humor
This is NOT a cluster snapshot, but a snapshot of a cluster.
(This is a very inside joke. Just ignore it if you don't get it.)
Official IBM sites:
- IBM PowerHA SystemMirror for AIX
- HACMP Documentation Library
- PowerHA Cluster Manager (HACMP) - Support for Power Systems
PowerHA is the new name for HACMP, which is short for High Availability Cluster Multi-Processing, a product of IBM. PowerHA / HACMP runs on AIX (and also on Linux) and its purpose is to provide high availability to systems, mainly for hardware failures. It can automatically detect system or network failures and can provide the capability to recover system hardware, applications, data and users while keeping recovery time to an absolute minimum. This is useful for systems that need to be online 24 hours a day, 365 days per year; for organizations that can't afford to have systems down for longer than 15 minutes. It's not completely fault-tolerant, but it is high available.
Compared to other cluster software, PowerHA / HACMP is highly robust, allows for large distances between nodes of a single cluster and allows up to 32 nodes in a cluster. Previous version of PowerHA / HACMP have had a reputation of having a lot of "bugs". From version 5.4 onward PowerHA / HACMP has seen a lot of improvements.
IBM's HACMP exists for over 15 years. It's not actually an IBM product; IBM bought it from CLAM, which was later renamed to Availant and then renamed to LakeViewTech and nowadays is called Vision Solutions. Until August 2006, all development of HACMP was done by CLAM. Nowadays, IBM does its own development of PowerHA / HACMP in Austin, Poughkeepsie and Bangalore.
Competitors of PowerHA / HACMP are Veritas Cluster and Echo Cluster. The last one, Echo Cluster, is a product of Vision Solutions mentioned above and tends to be easier to set-up and meant for simpler clusters. Veritas is only used by customers that use it already on other operating systems, like Sun Solaris and Windows Server environments, and don't want to invest into yet another clustering technology.
| AIX 5.1 | AIX 5.2 | AIX 5.3 | AIX 6.1 | AIX 7.1 | AIX 7.2 | Release Date | End Of Support |
|
| HACMP 5.1 | Yes | Yes | Yes | No | No | No | 7/11/2003 | 9/1/2006 |
| HACMP 5.2 | Yes | Yes | Yes | No | No | No | 7/16/2004 | 9/30/2007 |
| HACMP 5.3 | No | ML4+ | ML2+ | Yes | No | No | 8/12/2005 | 9/30/2009 |
| HACMP 5.4.0 | No | TL8+ | TL4+ | No | No | No | 7/28/2006 | 9/30/2011 |
| HACMP 5.4.1 | No | TL8+ | TL4+ | Yes | Yes | No | 9/11/2007 | 9/30/2011 |
| PowerHA 5.5 | No | No | TL7+ | tl2 sp1+ | Yes | No | 11/14/2008 | 4/30/2012 |
| PowerHA 6.1 | No | No | TL9+ | tl2 sp1+ | Yes | No | 10/20/2009 | 4/30/2015 |
| PowerHA 7.1.0 | No | No | No | tl6+ | Yes | No | 9/10/2010 | 9/30/2014 |
| PowerHA 7.1.1 | No | No | No | tl7 sp2+ | tl1 sp2+ | No | 9/10/2010 | 4/30/2015 |
| PowerHA 7.1.2 | No | No | No | tl8 sp1+ | tl2 sp1+ | No | 10/3/2012 | 4/30/2016 |
| PowerHA 7.1.3 | No | No | No | tl9 sp1+ | tl3 sp1+ | No | 10/7/2013 | 4/30/2018 |
| PowerHA 7.2.0 | No | No | No | tl9 sp5+ | tl3 sp5+ tl4 sp1+ | tl0 sp1+ | 12/4/2015 | 4/30/2019 |
| PowerHA 7.2.1 | No | No | No | No | tl3+ | tl0 sp1+ | 12/16/2016 | 4/30/2020 |
| PowerHA 7.2.2 | No | No | No | No | tl4+ | tl0 sp1+ | 12/15/2017 | tbd |
Source: PowerHA for AIX Version Compatibility Matrix
With HACMP clusters documentation is probably the most important issue. You cannot properly manage an HACMP cluster if you do not document it. Document the precise configuration of the complete cluster and document any changes you've carried out. Also document all management procedures and stick to them! The cluster snapshot facility is an excellent way of documenting your cluster.
Next step: get educated. You have to know exactly what you're doing on an HACMP cluster. If you have to manage a production cluster, getting a certification is a necessity. Don't ever let non-HACMP-educated UNIX administrators on your HACMP cluster nodes. They don't have a clue of what's going on and probably destroy your carefully layed-out configuration.
Geographically separated nodes are important! Too many cluster nodes just sit on top of each other in the same rack. What if there's a fire? Or a power outage? Having an HACMP cluster won't help you if both nodes are on a single location, use the same power, or the same network switches.
Put your HACMP logs in a sensible location. Don't put them in /tmp knowing that /tmp gets purged every night....
Test, test, test and test your cluster over again. Doing take-over tests every half year is best practice. Document your tests, and your test results.
Don't assume that your cluster is high available after installing the cluster software. There are a lot of other things to consider in your infrastructure to avoid single points of failures, like: No two nodes sharing the same I/O drawer; Power redundancy; No two storage or network adapters on the same SCSI backplane or bus; Redundancy in SAN HBA's; Application monitoring in place.
The traditional method for making an Oracle database capable of 7*24 operation is by means of creating an HACMP cluster in an Active-Standby configuration. In case of a failure of the Active system, HACMP lets the standby system take over the resources, start Oracle and thus resumes operation. This takeover is done with a downtime period of aprox. 5 to 15 minutes, however the impact on the business applications is more severe. It can lead to interruptions up to one hour in duration.
Another way to achieve high availability of databases, is to use a special version of the Oracle database software called Real Application Cluster, also called RAC. In a RAC cluster multiple systems (instances) are active (sharing the workload) and provide a near always-on database operation. The Oracle RAC software relies on IBM's HACMP software to achieve high availability for hardware and the operating system platform AIX. For storage it utilizes a concurrent filesystem called GPFS (General Parallel File System), a product of IBM. Oracle RAC 9 uses GPFS and HACMP. With RAC 10 you no longer need HACMP and
GPFS.
HACMP is used for network down notifications. Put all network adapters of 1 node on a single switch and put every node on a different switch. HACMP only manages the public and private network service adapters. There are no standby, boot or management adapters in a RAC HACMP cluster. It just uses a single
hostname; Oracle RAC and GPFS do not support hostname take-over or IPAT (IP Address take-over). There are no disks, volume groups or resource groups defined in an HACMP RAC cluster. In fact, HACMP is only necessary for event handling for Oracle RAC.
Name your HACMP RAC clusters in such away, that you can easily recognize the cluster as a RAC cluster, by using a naming convention that starts with RAC_.
On every GPFS node of an Oracle RAC cluster a GPFS daemon (mmfs) is active. These daemons need to communicate with each other. This is done via the public network, not via the private network.
Cache Fusion
Via SQL*Net an Oracle block is read in memory. If a second node in an HACMP RAC cluster requests the same block, it will first check if it already has it stored locally in its own cache. If not, it will use a private dedicated network to ask if another node has the block in cache. If not, the block will be read from disk. This is called Cache Fusion or Oracle RAC interconnect.
This is why on RAC HACMP clusters, each node uses an extra private network adapter to communicate with the other nodes, for Cache Fusion purposes only. All other communication, including the communication between the GPFS daemons on every node and the communication from Oracle clients, is done via the public network adapter. The throughput on the private network adapter can be twice as high as on the public network adapter.
Oracle RAC will use its own private network for Cache Fusion. If this network is not available, or if one node is unable to access the private network, then the private network is no longer used, but the public network will be used instead. If the private network returns to normal operation, then a fallback to the private network will occur. Oracle RAC uses cllsif of HACMP for this purpose.
Topics: Monitoring, PowerHA / HACMP, Security↑
HACMP 5.4: How to change SNMP community name from default "public" and keep clstat working
HACMP 5.4 supports changing the default community name from "public" to something else. SNMP is used for clstatES communications. Using the "public" SNMP community name, can be a security vulnerability. So changing it is advisable.
First, find out what version of SNMP you are using:
# ls -l /usr/sbin/snmpd(In this case, it is using version 3).
lrwxrwxrwx 1 root system 9 Sep 08 2008 /usr/sbin/snmpd -> snmpdv3ne
Make a copy of your configuration file. It is located on /etc.
/etc/snmpd.conf <- Version 1Edit the file and replace wherever public is mentioned for your new community name. Make sure to use not more that 8 characters for the new community name.
/etc/snmpdv3.conf <- Version 3
Change subsystems and restart them:
# chssys -s snmpmibd -a "-c new"Test using your locahost:
# chssys -s hostmibd -a "-c new"
# chssys -s aixmibd -a "-c new"
# stopsrc -s snmpd
# stopsrc -s aixmibd
# stopsrc -s snmpmibd
# stopsrc -s hostmibd
# startsrc -s snmpd
# startsrc -s hostmibd
# startsrc -s snmpmibd
# startsrc -s aixmibd
# snmpinfo -m dump -v -h localhost -c new -o /usr/es/sbin/cluster/hacmp.defs nodeTableIf the command hangs, something is wrong. Check the changes you made.
If everything works fine, perform the same change in the other node and test again. Now you can test from one server to the other using the snmpinfo command above.
If you need to backout, replace with the original configuration file and restart subsystems. Note in this case we use double-quotes. There is no space.
# chssys -s snmpmibd -a ""Okay, now make the change to clinfoES and restart and both nodes:
# chssys -s hostmibd -a ""
# chssys -s aixmibd -a ""
# stopsrc -s snmpd
# stopsrc -s aixmibd
# stopsrc -s snmpmibd
# stopsrc -s hostmibd
# startsrc -s snmpd
# startsrc -s hostmibd
# startsrc -s snmpmibd
# startsrc -s aixmibd
# chssys -s clinfoES -a "-c new"Wait a few minutes and you should be able to use clstat again with the new community name.
# stopsrc -s clinfoES
# startsrc -s clinfoES
Disclaimer: If you have any other application other than clinfoES that uses snmpd with the default community name, you should make changes to it as well. Check with your application team or software vendor.
You can tweak the Dead Man Switch settings for HACMP. First have a look at the current setting by running:
# lssrc -ls topsvcsA system usually has at least 2 heartbeats: 1 through the network: net_ether_01, with a sensitivity of 10 missed beats x 1 second interval x 2 = 20 seconds for it to fail. The other heartbeat is usually the disk heartbeat, diskhb_0, with a sensitivity of 4 missed beats x 2 second interval x 2 = 16 seconds.
Basically, if the other node has failed, HACMP will know if all the heartbeating has failed, thus after 20 seconds.
You can play around with the HACMP detection rates: Set it to normal:
# /usr/es/sbin/cluster/utilities/claddnim -oether -r2Ethernet heartbeating fails after 20 seconds. If you want to set it to slow: Use "-r3" instead of "-r2", and it fails after 48 seconds. Set it to fast: Use -r1, which will fail it after 10 seconds.
To give you some more time, you can use a grace period:
# claddnim -oether -g 15This will give you 15 seconds of grace time, which is the time within a network fallover must be taken care of.
You will have to synchronize the cluster after making any changes using claddnim:
# /usr/es/sbin/cluster/utilities/cldare -rt -V 'normal'
