Let's say you want to know what process is tying up port 25000:

# netstat -aAn | grep 25000
f100060020cf1398  tcp4  0  0  *.25000  *.*  LISTEN
f10006000d490c08  stream  0  0  f1df487f8  0  0  0  /tmp/.sapicm25000

# rmsock f100060020cf1398 tcpcb
The socket 0x20cf1008 is being held by proccess 1806748 (icman).

# lsof -i :[PORT]

# lsof -i :5710
COMMAND     PID   USER   FD   TYPE     DEVICE  SIZE/OFF NODE NAME
oracle  2638066 oracle   18u  IPv4 0xf1b3f398 0t1716253  TCP host:5710

If your AIX server level is below 5.3 TL06, the easiest way is just to upgrade your current OS to TL 06 at minimum (take note it will depend of configurations for Power6 processors) then clone your server and install it on the new p6.

But if you want to avoid an outage on your account, you can do the next using a NIM server (this is not official procedure for IBM, so they do not support this):

• Create your mksysb resource and do not create a spot from mksysb.
• Create an lppsource and spot with minimum TL required (I used TL08).
• Once you do nim_bosinst, choose the mksysb, and the created spot. It will send a warning message about spot is not at same level as mksysb, just ignore it.
• Do all necessary to boot from NIM.
• Once restoring the mksysb, there's some point where it is not able to create the bootlist because it detects the OS level is not supported on p6. So It will ask to continue and fix it later via SMS or fix it right now.
• Choose to fix it right now (it will open a shell). You will notice oslevel is as the same as mksysb.
• Create a NFS from NIM server or another server where you have the necessary TL and mount it on the p6.
• Proceed to do an upgrade, change the bootlist, exit the shell. Server will boot with new TL over the p6.

You can run invscout to do a microcode discovery on your system, that will generate a hostname.mup file. Then you go upload this hostname.mup file at this page on the IBM website and you get a nice overview of the status of all firmware on your system.

So far, so good. What if you have plenty of systems and you want to automate this? Here's a script to do this. This script first does a webget to collect the latest catalog.mic file from the IBM website. Then it distributes this catalog file to all the hosts you want to check. Then, it runs invscout on all these hosts, and collects the hostname.mup files. It will concatenate all these files into 1 large file and do an HTTP POST through curl to upload the file to the IBM website and have a report generated from it.

So, what do you need?

• You should have an AIX jump server that allows you to access the other hosts as user root through SSH. So you should have setup your SSH keys for user root.
• This jump server must have access to the Internet.
• You need to have wget and curl installed. Get it from the Linux Toolbox.
• Your servers should be AIX 5 or higher. It doesn't really work with AIX 4.
• Optional: a web server, like Apache 2, would be nice, so you can drop the resulting HTML file on your website every day.
• An entry in the root crontab to run this script every day.
• A list of servers you want to check.

#!/bin/ksh

# script:  generate_survey.ksh
# purpose: To generate a microcode survey html file

# where is my list of servers located?
SERVERS=/usr/local/etc/servers

# what temporary folder will I use?
TEMP=/tmp/mup

# what is the invscout folder
INV=/var/adm/invscout

# what is the catalog.mic file location for invscout?
MIC=${INV}/microcode/catalog.mic

# if you have a webserver,
# where shall I put a copy of survey.html?
APA=/usr/local/apache2/htdocs

# who's the sender of the email?
FROM=microcode_survey@ibm.com

# who's the receiver of the email?
TO="your.email@address.com"

# what's the title of the email?
SUBJ="Microcode Survey"

# user check
USER=`whoami`
if [ "$USER" != "root" ];
then
    echo "Only root can run this script."
    exit 1;
fi

# create a temporary directory
rm -rf $TEMP 2>/dev/null
mkdir $TEMP 2>/dev/null
cd $TEMP

# get the latest catalog.mic file from IBM
# you need to have wget installed 
# and accessible in $PATH
# you can download this on:
# www-03.ibm.com
# /systems/power/software/aix/linux/toolbox/download.html
wget techsupport.services.ibm.com/server/mdownload/catalog.mic
# You could also use curl here, e.g.:
#curl techsupport.services.ibm.com/server/mdownload/catalog.mic -LO

# move the catalog.mic file to this servers invscout directory
mv $TEMP/catalog.mic $MIC

# remove any old mup files
echo Remove any old mup files from hosts.
for server in `cat $SERVERS` ; do
   echo "${server}"
   ssh $server "rm -f $INV/*.mup"
done

# distribute this file to all other hosts
for server in `cat $SERVERS` ; do
   echo "${server}"
   scp -p $MIC $server:$MIC
done

# run invscout on all these hosts
# this will create a hostname.mup file
for server in `cat $SERVERS` ; do
   echo "${server}"
   ssh $server invscout
done

# collect the hostname.mup files
for server in `cat $SERVERS` ; do
   echo "${server}"
   scp -p $server:$INV/*.mup $TEMP
done

# concatenate all hostname.mup files to one file
cat ${TEMP}/*mup > ${TEMP}/muppet.$$

# delete all the hostname.mup files
rm $TEMP/*mup

# upload the remaining file to IBM.
# you need to have curl installed for this
# you can download this on:
# www-03.ibm.com
# /systems/power/software/aix/linux/toolbox/download.html
# you can install it like this:
# rpm -ihv 
#    curl-7.9.3-2.aix4.3.ppc.rpm curl-devel-7.9.3-2.aix4.3.ppc.rpm
# more info on using curl can be found on: 
# http://curl.haxx.se/docs/httpscripting.html
# more info on uploading survey files can be found on:
# www14.software.ibm.com/webapp/set2/mds/fetch?pop=progUpload.html

# Sometimes, the IBM website will respond with an
# "Expectation Failed" error message. Loop the curl command until
# we get valid output.

stop="false"

while [ $stop = "false" ] ; do

curl -H Expect: -F mdsData=@${TEMP}/muppet.$$ -F sendfile="Upload file" \ 
   http://www14.software.ibm.com/webapp/set2/mds/mds \
   > ${TEMP}/survey.html

#
# Test if we see Expectation Failed in the output
#

unset mytest
mytest=`grep "Expectation Failed" ${TEMP}/survey.html`

if [ -z "${mytest}" ] ; then
        stop="true"
fi

sleep 10

done

# now it is very useful to have an apache2 webserver running
# so you can access the survey file
mv $TEMP/survey.html $APA

# tip: put in the crontab daily like this:
# 45 9 * * * /usr/local/sbin/generate_survey.ksh 1>/dev/null 2>&1

# mail the output
# need to make sure this is sent in html format
cat - ${APA}/survey.html <<HERE | sendmail -oi -t
From: ${FROM}
To: ${TO}
Subject: ${SUBJ}
Mime-Version: 1.0
Content-type: text/html
Content-transfer-encoding: 8bit

HERE

# clean up the mess
cd /tmp
rm -rf $TEMP

A very usefull command to compary 2 files is sdiff. Let's say you want to compare the lslpp from 2 different hosts, then sdiff -s shows the differences between two files next to each other:

# sdiff -s /tmp/a /tmp/b
                                  >  bos.loc.com.utf          5.3.9.0
                                  >  bos.loc.utf.EN_US        5.3.0.0
                                  >                                    
gskta.rte               7.0.3.27  |  gskta.rte               7.0.3.17
lum.base.cli             5.1.2.0  |  lum.base.cli             5.1.0.0
lum.base.gui             5.1.2.0  |  lum.base.gui             5.1.0.0
lum.msg.en_US.base.cli   5.1.2.0  |  lum.msg.en_US.base.cli   5.1.0.0
lum.msg.en_US.base.gui   5.1.2.0  |  lum.msg.en_US.base.gui   5.1.0.0
rsct.basic.sp           2.4.10.0  <
                                  <
rsct.compat.basic.sp    2.4.10.0  <
                                  <
rsct.compat.clients.sp  2.4.10.0  <
                                  <
rsct.opt.fence.blade    2.4.10.0  <
rsct.opt.fence.hmc      2.4.10.0  <
bos.clvm.enh             5.3.8.3  |  bos.clvm.enh            5.3.0.50
lum.base.cli             5.1.2.0  |  lum.base.cli             5.1.0.0

To mount:

1. Build a logical volume (the size of an ISO image, better if a little bigger).
2. Create an entry in /etc/filesystem using that logical volume (LV), but setting its Virtual File System (V'S) to be cdrfs.
3. Create the mount point for this LV/ISO.
4. Copy the ISO image to the LV using dd.
5. Mount and work on it like a mounted CD-ROM.

/IsoCD:

dev = /dev/lv09
vfs = cdrfs
mount = false
options = ro
account = false

1. Unmount the file system.
2. Destroy the logical volume.

JFS2 filesystems allow you to create file system snapshots. Creating a snapshot is actually creating a new file system, with a copy of the metadata of the original file system (the snapped FS). The snapshot (like a photograph) remains unchanged, so it's possible to backup the snapshot, while the original data can be used (and changed!) by applications. When data on the original file system changes, while a snapshot exists, the original data is copied to the snapshot to keep the snapshot in a consistant state. For these changes, you'll need temporary space, thus you need to create a snapshot of a specific size to allow updates while the snapshot exists. Usually 10% is enough. Database file systems are usually not a very good subject for creating snapshots, because all database files change constantly when the database is active, causing a lot of copying of data from the original to the snapshot file system.

In order to have a snapshot you have to:

• Create and mount a JFS2 file system (source FS). You can find it in SMIT as "enhanced" file system.
• Create a snapshot of a size big enough to hold the changes of the source FS by issuing smitty crsnap. Once you have created this snapshot as a logical device or logical volume, there's a read-only copy of the data in source FS. You have to mount this device in order to work with this data.
• Mount your snapshot device by issuing smitty mntsnap. You have to provide a directory name over which AIX will mount the snapshot. Once mounted, this device will be read-only.

# snapshot -o snapfrom=$FILESYSTEM -o size=${SNAPSIZE}M

# snapshot -q $FILESYSTEM

# mkdir -p /snapshot$FILESYSTEM

# SNAPDEVICE=`snapshot -q $FILESYSTEM | grep -v ^Snapshots | grep -v ^Current | awk '{print $2}'`

# mount -v jfs2 -o snapshot $SNAPDEVICE /snapshot$FILESYSTEM

# unmount /snapshot$FILESYSTEM

# snapshot -d $SNAPDEVICE

# rm -rf /snapshot$FILESYSTEM

A very nice command to use when you either want to show someone remotely how to do something on AIX, or to allow a non-root user to have root access, is portmir.

First of all, you need 2 users logged into the system, you and someone else. Ask the other person to run the tty command in his/her telnet session and to tell you the result. For example:

user$ tty
/dev/pts/1

root# portmir -t /dev/pts/1

# portmir -o

To create a printer queue that dumps it contents to /dev/null:

# /usr/lib/lpd/pio/etc/piomkpq -A 'file' -p 'generic' -d '/dev/null' -D asc -q 'qnull'

# lpr -Pqfile /etc/motd

A very usefull command to list microcodes is lsmcode:

# lsmcode -c

CTRL-ALT-F1: Switch to Linux command line; no login possible. If you then click on CTRL-ALT-DEL the system will reboot.
CTRL-ALT-F2: Takes you back to the Xserver window.
CTRL-ALT-BACKSPACE: Kills of the Xserver and will start a new -fresh- one, so you can login again.