You can run into an issue with EMC storage on AIX systems using MPIO (No Powerpath) for your boot disks:

After installing the ODM_DEFINITONS of EMC Symmetrix on your client system, the system won't boot any more and will hang with LED 554 (unable to find boot disk).

The boot hang (LED 554) is not caused by the EMC ODM package itself, but by the boot process not detecting a path to the boot disk if the first MPIO path does not corresponding to the fscsiX driver instance where all hdisks are configured. Let me explain that more in detail:

Let's say we have an AIX system with four HBAs configured in the following order:

# lscfg -v | grep fcs
fcs2 (wwn 71ca) -> no devices configured behind this fscsi2 driver instance (path only configured in CuPath ODM table)
fcs3 (wwn 71cb) -> no devices configured behind this fscsi3 driver instance (path only configured in CuPath ODM table)
fcs0 (wwn 71e4) -> no devices configured behind this fscsi0 driver instance (path only configured in CuPath ODM table)
fcs1 (wwn 71e5) -> ALL devices configured behind this fscsi1 driver instance

# lspath -l hdisk2 -H -F"name parent path_id connection status"
name   parent path_id connection                      status
hdisk2 fscsi0 0       5006048452a83987,33000000000000 Enabled
hdisk2 fscsi1 1       5006048c52a83998,33000000000000 Enabled
hdisk2 fscsi2 2       5006048452a83986,33000000000000 Enabled
hdisk2 fscsi3 3       5006048c52a83999,33000000000000 Enabled

# rmpath -l hdisk2 -p fscsi0 -d
# rmpath -l hdisk2 -p fscsi2 -d
# rmpath -l hdisk2 -p fscsi3 -d
# lspath -l hdisk2 -H -F"name parent path_id connection status"

# lspv
hdisk10 0003027f7f7ca7e2 rootvg active
# lsdev -Cc disk
hdisk2 Defined   00-09-01 MPIO Other FC SCSI Disk Drive
hdisk10 Available 00-08-01 EMC Symmetrix FCP MPIO Raid6

If you run:

# powermt display dev=all

# powermt restore

# powermt reset
# powermt config

# powermt check

It will sometimes occur that a file system reports storage to be in use, while you're unable to find which file exactly is using that storage. This may occur when a process has used disk storage, and is still holding on to it, without the file actually being there anymore for whatever reason.

A good way to resolve such an issue, is to reboot the server. This way, you'll be sure the process is killed, and the disk storage space is released. However, if you don't want to use such drastic measures, here's a little script that may help you trying to find the process that may be responsible for an inode without a filename. Make sure you have lsof installed on your server.

#!/usr/bin/ksh

# Make sure to enter a file system to scan
# as the first attribute to this script.
FILESYSTEM=$1
LSOF=/usr/sbin/lsof

# A for loop to get a list of all open inodes
# in the filesystem using lsof.
for i in `$LSOF -Fi $FILESYSTEM | grep ^i | sed s/i//g` ; do
# Use find to list associated inode filenames.
if [ `find $FILESYSTEM -inum $i` ] ; then
echo > /dev/null
else
# If filename cannot be found,
# then it is a suspect and check lsof output for this inode.
echo Inode $i does not have an associated filename:
$LSOF $FILESYSTEM | grep -e $i -e COMMAND
fi
done

An easy way to see the status of your SAN devices is by using the following command:

# powermt display
Symmetrix logical device count=6
CLARiiON logical device count=0
Hitachi logical device count=0
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0
==============================================================
- Host Bus Adapters -  --- I/O Paths ----  ------ Stats ------
### HW Path            Summary Total Dead  IO/Sec Q-IOs Errors
==============================================================
  0 fscsi0             optimal     6    0       -     0      0
  1 fscsi1             optimal     6    0       -     0      0

# powermt display dev=all

From powerlink.emc.com:

1. Before making any changes, collect host logs to document the current configuration. At a minimum, save the following: inq, lsdev -Cc disk, lsdev -Cc adapter, lspv, and lsvg
2. Shutdown the application(s), unmount the file system(s), and varyoff all volume groups except for rootvg. Do not export the volume groups.

   # varyoffvg <vg_name>

   Check with lsvg -o (confirm that only rootvg is varied on)
   If no PowerPath, skip all steps with power names.
3. For CLARiiON configuration, if Navisphere Agent is running, stop it:

   # /etc/rc.agent stop

4. Remove paths from Powerpath configuration:

   # powermt remove hba=all

5. Delete all hdiskpower devices:

   # lsdev -Cc disk -Fname | grep power | xargs -n1 rmdev -dl

6. Remove the PowerPath driver instance:

   # rmdev -dl powerpath0

7. Delete all hdisk devices:
   For Symmetrix devices, use this command:

   # lsdev -CtSYMM* -Fname | xargs -n1 rmdev -dl

   For CLARiiON devices, use this command:

   # lsdev -CtCLAR* -Fname | xargs -n1 rmdev -dl

8. Confirm with lsdev -Cc disk that there are no EMC hdisks or hdiskpowers.
9. Remove all Fiber driver instances:

   # rmdev -Rdl fscsiX

   (X being driver instance number, i.e. 0,1,2, etc.)
10. Verify through lsdev -Cc driver that there are no more fiber driver instances (fscsi).
11. Change the adapter instances in Defined state

    # rmdev -l fcsX

    (X being adapter instance number, i.e. 0,1,2, etc.)
12. Create the hdisk entries for all EMC devices:

    # emc_cfgmgr

    or

    # cfgmgr -vl fcsx

    (x being each adapter instance which was rebuilt). Skip this part if no PowerPath.
13. Configure all EMC devices into PowerPath:

    # powermt config

14. Check the system to see if it now displays correctly:

    # powermt display
    # powermt display dev=all
    # lsdev -Cc disk
    # /etc/rc.agent start

SAN storage places the physical disk outside a computer system. It is now connected to a Storage Area Network (SAN). In a Storage Area Network, storage is offered to many systems, including AIX systems. This is done via logical blocks of disk space (LUNs). In the case of an AIX system, every SAN disk is seen as a seperate hdisk, with the advantage of easily expanding the AIX system with new SAN disks, avoiding buying and installing new physical hard disks.

• Disk storage is no longer limited to the space in the computer system itself or the amount of available disk slots.
• After the initial investment in the SAN network and storage, the costs of storage per gigabyte are less than disk space within the computer systems.
• Using two different SAN networks (fabrics), you can avoid having disruptions in your storage, the same as mirroring your data on separate disks. The two SAN fabrics should not be connected to each other.
• Using two seperate, geographically dispersed storage systems (e.g. ESS), a disruption in a computer center will not cause your computer systems to go down.
• When you place to SAN network adapters (called Host Bay adapters on Fibre Channel or HBA) in every computer system, you can connect your AIX system to two different fabrics, thus increasing the availability of the storage. Also, you'll be able to load balance the disk storage over these two host bay adapters. You'll need Multipath I/O software (e.g. SDD or PowerPath) for this to work.
• By using 2 HBAs, a defect in a single HBA will not cause downtime.
• AIX systems are able to boot from SAN disks.

If you have Emulex HBA''s and the hbanyware software installed, for example on Linux, then you can use the following commands to retrieve information about the HBA''s:

To run a GUI version:

# /usr/sbin/hbanyware/hbanyware

# /usr/sbin/hbanyware/hbacmd listhbas

# /usr/sbin/hbanyware/hbacmd listhbas 10:00:00:00:c9:6c:9f:d0

If you wish to get rid of the SCSI disk reservation bit on SCSI, SSA and VPATH devices, there are two ways of achieving this:

Firstly, HACMP comes along with some binaries that do this job:

# /usr/es/sbin/cluster/utilities/cl_SCSIdiskreset /dev/vpathx

# lquerypr -vh /dev/vpathx

# lquerypr -ch /dev/vpathx

# lquerypr -ch /dev/vpathx
# lquerypr -rh /dev/vpathx
# lquerypr -ph /dev/vpathx

# lquerypr -V -v -c /dev/vpathXX

# pcmquerypr -V -v -c /dev/hdiskXX

Check the relation between vpaths and hdisks:

# lsvpcfg

# datapath query adapter

# lsvg -o | lsvg -i | grep -i stale

If you have any VMWare images, where you made the disk size a little too small, then fortunately in VMWare Workstation you can change the size of a disk with a simple command line program. Sadly the command only makes your drive bigger not the actual partition. And especially Windows won't allow you to resize the partition where the Windows binaries are installed. So how can you get around that?

First, create a copy of your vmdk file to somewhere else, should the next action fail for some reason.

Then resize the disk to the required size:

# vmware-vdiskmanager -x 8GB myDisk.vmdk