Topics: AIX, PowerHA / HACMP, System Admin

NFS mounts on HACMP failing

When you want to mount an NFS file system on a node of an HACMP cluster, there are a couple of items you need check, before it will work:

  • Make sure the hostname and IP address of the HACMP node are resolvable and provide the correct output, by running:
    # nslookup [hostname]
    # nslookup [ip-address]
    
  • The next thing you will want to check on the NFS server, if the node names of your HACMP cluster nodes are correctly added to the /etc/exports file. If they are, run:
    # exportfs -va
  • The last, and tricky item you will want to check is, if a service IP label is defined as an IP alias on the same adapter as your nodes hostname, e.g.:
    # netstat -nr
    Routing tables
    Destination   Gateway       Flags  Refs  Use    If  Exp  Groups
    
    Route Tree for Protocol Family 2 (Internet):
    default       10.251.14.1   UG      4    180100 en1  -     -
    10.251.14.0   10.251.14.50  UHSb    0         0 en1  -     -
    10.251.14.50  127.0.0.1     UGHS    3    791253 lo0  -     -
    
    The example above shows you that the default gateway is defined on the en1 interface. The next command shows you where your Service IP label lives:
    # netstat -i
    Name  Mtu   Network   Address         Ipkts   Ierrs Opkts
    en1   1500  link#2    0.2.55.d3.75.77 2587851 0      940024
    en1   1500  10.251.14 node01          2587851 0      940024
    en1   1500  10.251.20 serviceip       2587851 0      940024
    lo0   16896 link#1                    1912870 0     1914185
    lo0   16896 127       loopback        1912870 0     1914185
    lo0   16896 ::1                       1912870 0     1914185
    
    As you can see, the Service IP label (in the example above called "serviceip") is defined on en1. In that case, for NFS to work, you also want to add the "serviceip" to the /etc/exports file on the NFS server and re-run "exportfs -va". And you should also make sure that hostname "serviceip" resolves to an IP address correctly (and of course the IP address resolves to the correct hostname) on both the NFS server and the client.

Topics: AIX, System Admin

MD5 for AIX

If you need to run an MD5 check-sum on a file on AIX, you will notice that there's not md5 or md5sum command available on AIX. Instead, use the following command to do this:

# csum -h MD5 [filename]
Note: csum can't handle files larger than 2 GB.

Topics: AIX, Performance, System Admin

Nmon analyser - A free tool to produce AIX performance reports

Searching for an easy way to create high-quality graphs that you can print, publish to the Web, or cut and paste into performance reports? Look no further. The nmon_analyser tool takes files produced by the NMON performance tool, turns them into Microsoft Excel spreadsheets, and automatically produces these graphs.

You can download the tool here:
http://www.ibm.com/developerworks/aix/library/au-nmon_analyser/

Topics: AIX, Backup & restore

NFS mksysb script

Here's a script you can use to run mksysb backups of your clients to a NFS server. It is generally a good idea to set up a NIM server and also use this NIM server as a NFS server. All your clients should then be configured to create their mksysb backups to the NIM/NFS server, using the script that you can download here: nimbck.ksh.

By doing this, the latest mksysb images are available on the NIM server. This way, you can configure a mksysb resource on the NIM server (use: smitty nim_mkres) pointing to the mksysb image of a server, for easy recovery.

Topics: AIX, Backup & restore, NIM, System Admin

How to unconfigure items after mksysb recovery using NIM

There will be a situation where you want to test a mksysb recovery to a different host. The major issue with this is, that you bring up a server within the same network, that is a copy of an actual server that's already in your network. To avoid running into 2 exactly the same servers in your network, here's how you do this:

First make sure that you have a separate IP address available for the server to be recovered, for configuration on your test server. You definitely don't want to bring up a second server in your network with the same IP configuration.

Make sure you have a mksysb created of the server that you wish to recover onto another server. Then, create a simple script that disables all the items that you don't want to have running after the mksysb recovery, for example:

# cat /export/nim/cust_scripts/custom.ksh
#!/bin/ksh

# Save a copy of /etc/inittab
cp /etc/inittab /etc/inittab.org

# Remove unwanted entries from the inittab
rmitab hacmp 2>/dev/null
rmitab tsmsched 2>/dev/null
rmitab tsm 2>/dev/null
rmitab clinit 2>/dev/null
rmitab pst_clinit 2>/dev/null
rmitab qdaemon 2>/dev/null
rmitab sddsrv 2>/dev/null
rmitab nimclient 2>/dev/null
rmitab nimsh 2>/dev/null
rmitab naviagent 2>/dev/null

# Get rid of the crontabs
mkdir -p /var/spool/cron/crontabs.org
mv /var/spool/cron/crontabs/* /var/spool/cron/crontabs.org/

# Disable start scripts
chmod 000 /etc/rc.d/rc2.d/S01app

# copy inetd.conf
cp /etc/inetd.conf /etc/inetd.conf.org
# take out unwanted items
cat /etc/inetd.conf.org | grep -v bgssd > /etc/inetd.conf

# remove the hacmp cluster configuration
if [ -x /usr/es/sbin/cluster/utilities/clrmclstr ] ; then
        /usr/es/sbin/cluster/utilities/clrmclstr
fi

# clear the error report
errclear 0

# clean out mail queue
rm /var/spool/mqueue/*
The next thing you need to do, is to configure this script as a 'script resource' in NIM. Run:
# smitty nim_mkres
Select 'script' and complete the form afterwards. For example, if you called it 'UnConfig_Script':
# lsnim -l UnConfig_Script
UnConfig_Script:
   class       = resources
   type        = script
   comments    = 
   Rstate      = ready for use
   prev_state  = unavailable for use
   location    = /export/nim/cust_scripts/custom.ksh
   alloc_count = 0
   server      = master
Then, when you are ready to perform the actual mksysb recovery using "smitty nim_bosinst", you can add this script resource on the following line:
Customization SCRIPT to run after installation [UnConfig_Script]

Topics: AIX, Backup & restore, NIM, System Admin

Using the image_data resource to restore a mksysb without preserving mirrors using NIM

Specify using the 'image_data' resource when running the 'bosinst' command from the NIM master:

From command line on the NIM master:

# nim -o bos_inst -a source=mksysb -a lpp_source=[lpp_source] -a spot=[SPOT] -a mksysb=[mksysb] -a image_data=mksysb_image_data -a accept_licenses=yes server1
Using smit on the NIM master:
# smit nim_bosinst
Select the client to install. Select 'mksysb' as the type of install. Select a SPOT at the same level as the mksysb you are installing. Select an lpp_source at the same level than the mksysb you are installing.

NOTE: It is recommended to use an lpp_source at the same AIX Technology Level, but if using an lpp_source at a higher level than the mksysb, the system will be updated to the level of the lpp_source during installation. This will only update Technology Levels.

If you're using an AIX 5300-08 mksysb, you cannot use an AIX 6.1 lpp_source. This will not migrate the version of AIX you are running to a higher version. If you're using an AIX 5300-08 mksysb and allocate a 5300-09 lpp_source, this will update your target system to 5300-09.
         Install the Base Operating System on Standalone Clients

Type or select values in entry fields.
Press Enter AFTER making all desired changes.

[TOP]                                         [Entry Fields]
* Installation Target                       server1
* Installation TYPE                         mksysb
* SPOT                                      SPOTaix53tl09sp3
  LPP_SOURCE                               [LPPaix53tl09sp3]
  MKSYSB                                    server1_mksysb

  BOSINST_DATA to use during installation  []
  IMAGE_DATA to use during installation    [server1_image_date]

Topics: AIX, Backup & restore, NIM, System Admin

Creating an image_data resource without preserving mirrors for use with NIM

Transfer the /image.data file to the NIM master and store it in the location you desire. It is a good idea to place the file, or any NIM resource for that matter, in a descriptive manor, for example: /export/nim/image_data. This will ensure you can easily identify your "image_data" NIM resource file locations, should you have the need for multiple "image_data" resources.

Make sure your image.data filenames are descriptive also. A common way to name the file would be in relation to your clientname, for example: server1_image_data.

Run the nim command, or use smitty and the fast path 'nim_mkres' to define the file that you have edited using the steps above:

From command line on the NIM master:

# nim -o define -t image_data -a server=master -a location=/export/nim/image_data/server1_image_data -a comments="image.data file with broken mirror for server1" server1_image_data
NOTE: "server1_image_data" is the name given to the 'image_data' resource.

Using smit on the NIM master:
# smit nim_mkres
Select 'image_data' as the Resource Type. Then complete the following screen:
                       Define a Resource

Type or select values in entry fields.
Press Enter AFTER making all desired changes.

                               [Entry Fields]
* Resource Name            [server1_image_data]
* Resource Type             image_data
* Server of Resource       [master]
* Location of Resource     [/export/nim/image_data/server1_image_data]
  Comments                 []

  Source for Replication   []
Run the following command to make sure the 'image_data' resource was created:
# lsnim -t image_data
The command will give output similar to the following:
# lsnim -t image_data
server1_image_data     resources       image_data
Run the following command to get information about the 'image_data' resource:
# lsnim -l server1_image_data
server1_image_data:
   class       = resources
   type        = image_data
   Rstate      = ready for use
   prev_state  = unavailable for use
   location    = /export/nim/image_data/server1_image_data
   alloc_count = 0
   server      = master

Topics: AIX, Backup & restore, NIM, System Admin

How to edit an image.data file to break a mirror

Create a new image.data file by running the following command:

# cd /
# mkszfile
Edit the image.data file to break the mirror, by running the following command:
# vi /image.data
What you are looking for are the "lv_data" stanzas. There will be one for every logical volume associated with rootvg.

The following is an example of an lv_data stanza from an image.data file of a mirrored rootvg. The lines that need changing are marked bold:
lv_data:
VOLUME_GROUP= rootvg
LV_SOURCE_DISK_LIST= hdisk0 hdisk1
LV_IDENTIFIER= 00cead4a00004c0000000117b1e92c90.2
LOGICAL_VOLUME= hd6
VG_STAT= active/complete
TYPE= paging
MAX_LPS= 512
COPIES= 2
LPs= 124
STALE_PPs= 0
INTER_POLICY= minimum
INTRA_POLICY= middle
MOUNT_POINT=
MIRROR_WRITE_CONSISTENCY= off
LV_SEPARATE_PV= yes
PERMISSION= read/write
LV_STATE= opened/syncd
WRITE_VERIFY= off
PP_SIZE= 128
SCHED_POLICY= parallel
PP= 248
BB_POLICY= non-relocatable
RELOCATABLE= yes
UPPER_BOUND= 32
LABEL=
MAPFILE= /tmp/vgdata/rootvg/hd6.map
LV_MIN_LPS= 124
STRIPE_WIDTH=
STRIPE_SIZE=
SERIALIZE_IO= no
FS_TAG=
DEV_SUBTYP=
Note: There are two disks in the 'LV_SOURCE_DISK_LIST', THE 'COPIES' value reflects two copies, and the 'PP' value is double that of the 'LPs' value.

The following is an example of the same lv_data stanza after manually breaking the mirror. The lines that have been changed are marked bold. Edit each 'lv_data' stanza in the image.data file as shown below to break the mirrors.
lv_data:
VOLUME_GROUP= rootvg
LV_SOURCE_DISK_LIST= hdisk0
LV_IDENTIFIER= 00cead4a00004c0000000117b1e92c90.2
LOGICAL_VOLUME= hd6
VG_STAT= active/complete
TYPE= paging
MAX_LPS= 512
COPIES= 1
LPs= 124
STALE_PPs= 0
INTER_POLICY= minimum
INTRA_POLICY= middle
MOUNT_POINT=
MIRROR_WRITE_CONSISTENCY= off
LV_SEPARATE_PV= yes
PERMISSION= read/write
LV_STATE= opened/syncd
WRITE_VERIFY= off
PP_SIZE= 128
SCHED_POLICY= parallel
PP= 124
BB_POLICY= non-relocatable
RELOCATABLE= yes
UPPER_BOUND= 32
LABEL=
MAPFILE= /tmp/vgdata/rootvg/hd6.map
LV_MIN_LPS= 124
STRIPE_WIDTH=
STRIPE_SIZE=
SERIALIZE_IO= no
FS_TAG=
DEV_SUBTYP=
Note: The 'LV_SOURCE_DISK_LIST' has been reduced to one disk, the 'COPIES' value has been changed to reflect one copy, and the 'PP' value has been changed so that it is equal to the 'LPs' value.

Save the edited image.data file. At this point you can use the edited image.data file to do one of the following: You can now use your newly edited image.data file to create a new mksysb to file, tape, or DVD.

E.g.: To file or tape: place the edited image.data file in the / (root) directory and rerun your mksysb command without using the "-i" flag. If running the backup through SMIT, make sure you set the option "Generate new /image.data file?" to 'no' (By default it is set to 'yes').

To DVD: Use the -i flag and specify the [/location] of the edited image.data file. If running through SMIT specify the edited image.data file location in the "User supplied image.data file" field.

Within NIM you would create an 'image_data' resource for use with NIM to restore a mksysb without preserving mirrors.

Note: If you don't want to edit the image.data file manually, here's a script that you can use to have it updated to a single disk for you, assuming your image_data file is called /image.data:
cat /image.data | while read LINE ; do
  if [ "${LINE}" = "COPIES= 2" ] ; then
    COPIESFLAG=1
    echo "COPIES= 1"
  else
    if [ ${COPIESFLAG} -eq 1 ] ; then
      PP=`echo ${LINE} | awk '{print $1}'`
      if [ "${PP}" = "PP=" ] ; then
        PPNUM=`echo ${LINE} | awk '{print $2}'`
        ((PPNUMNEW=$PPNUM/2))
        echo "PP= ${PPNUMNEW}"
        COPIESFLAG=0
      else
        echo "${LINE}"
      fi
    else
      echo "${LINE}"
    fi
  fi
done > /image.data.1disk

Topics: AIX, Backup & restore, System Admin

How to restore an image.data file from an existing mksysb file

Change the /tmp directory (or a directory where you would like to store the /image.data file from the mksysb image) and restore the /image.data file from the mksysb:

# cd /tmp
# restore -xqvf [/location/of/mksysb/file] ./image.data
If you want to list the files in a mksysb image first, you can run the following command:
# restore -Tqvf [/location/of/mksysb/file]

Topics: AIX, Backup & restore, System Admin

How to restore an image.data file from tape

Restoring from tape:

First change the block size of the tape device to 512:

# chdev -l rmt0 -a block_size=512
Check to make sure the block size of the tape drive has been changed:
# tctl -f /dev/rmt0 status
You will receive output similar to this:
rmt0 Available 09-08-00-0,0 LVD SCSI 4mm Tape Drive
attribute     value description                          user_settable

block_size    512   BLOCK size (0=variable length)       True
compress      yes   Use data COMPRESSION                 True
density_set_1 71    DENSITY setting #1                   True
density_set_2 38    DENSITY setting #2                   True
extfm         yes   Use EXTENDED file marks              True
mode          yes   Use DEVICE BUFFERS during writes     True
ret           no    RETENSION on tape change or reset    True
ret_error     no    RETURN error on tape change or reset True
size_in_mb    36000 Size in Megabytes                    False
Change to the /tmp directory (or a directory where you would like to store the /image.data file from the mksysb image) and restore the /image.data file from the tape:
# cd /tmp
# restore -s2 -xqvf /dev/rmt0.1 ./image.data

Number of results found: 469.
Displaying results: 211 - 220.