Tech Blog

These are blog entries written by the UNIX Health Check development team. Our team has extensive technical experience on both AIX and Red Hat systems, and we like to share our knowledge with our visitors.

Topics: AIX, Storage, System Admin

Using a DVD-RAM as a regular file system

To use a DVD-RAM for writing and reading like a file system, use the following commands. This will work only at AIX 5.2 and above:

  • Put a DVD-RAM into the drive
    # udfcreate -d /dev/cd0
  • Mount the DVD-RAM:
    # mount -V udfs /dev/cd0 /mnt
    If you get an error, ensure /etc/vfs contains this line (and retry the mount command after validating):
    udfs 34 /sbin/helpers/udfmnthelp
Then use this as a regular filesystem.

Topics: AIX, SSA, Storage

SSA batteries

To find the status of the batteries of an SSA adapter, enter as root:

# ssa_fw_status -a ssaX
X is the number of your adapter, for example:
# ssa_fw_status -a ssa0
Fast write cache size: 32
Expected battery life: 22000
Powered on hours: 20706
Battery state: Active
After installing a new battery, enter the following command:
# ssa_format - l ssaX - b
This will reset the lifetime counter.

Topics: AIX, SSA, Storage, System Admin

Renaming pdisks

If, for some reason, the pdisk and hdisk numbering of SSA disks is not sequential anymore, then there's a way to bring order in to chaos. Usually, the pdisk and hdisk numbering order are screwed up when you replace multiple disks together. Especially on HACMP clusters, a correct numbering of pdisks and hdisks on all nodes of the cluster, comes in handy.

Unmount all file systems on the specific disks, then varyoff the volume group:

# /usr/lib/methods/cfgssar -l ssar
If this doesn't help (it sometimes will), then renumber the disks manually:

Write down the pdisk names, hdisk names, location of the disks in the SSA drawer and the connection ID's of the disks. You can use lsdev -Cc pdisk to show you all the pdisks and the drawer and location codes. Use lsdev -Clpdiskx -Fconnwhere to show the connection ID of a pdisk. Then, figure out, how you want all disks numbered.

Remove the pdisks and hdisks with the rmdev -dl command.
Create the pdisks again:
# mkdev -p ssar -t scsd -c pdisk -s ssar -w [connection-ID] -l pdisk1
Create the hdisks again:
# mkdev -p ssar -t hdisk -c disk -s ssar -w [connection-ID] -l hdisk3
Test with:
# ssaxlate -l pdisk1
if it shows hdisk3 (Usually the hdisk number is 2 higher than the pdisk number if you use 2 SCSI disks in the rootvg).

If you've done all disks this way, check with lsdev -Cc pdisk. If you're happy, then varyon the volume group again and mount all filesystems.

Topics: AIX, Performance, Storage, System Admin

Creating a large file

When you wish to create a very large file for test purposes, try this command:

# dd if=/dev/zero bs=1024 count=2097152 of=./test.large.file
This wil create a file consisting of 2097152 blocks of 1024 bytes, which is 2GB. You can change the count value to anything you like.

Be aware of the fact, that if you wish to create files larger than 2GB, that your file system needs to be created as a "large file enabled file system", otherwise the upper file size limit is 2GB (under JFS; under JFS2 the upper limit is 64GB). Also check the ulimit values of the user-id you use to create the large file: set the file limit to -1, which is unlimited. Usually, the file limit is default set to 2097151 in /etc/security/limits, which stands for 2097151 blocks of 512 bytes = 1GB.

Another way to create a large file is:
# /usr/sbin/lmktemp ./test.large.file 2147483648
This will create a file of 2147483648 bytes (which is 1024 * 2097152 = 2GB).

You can use this large file for adapter throughput testing purposes:

Write large sequential I/O test:
# cd /BIG
# time /usr/sbin/lmktemp 2GBtestfile 2147483648
Divide 2048/#seconds for MB/sec write speed.

Read large sequential I/O test:
# umount /BIG
(This will flush file from memory)
# mount /BIG
# time cp 2GBtestfile /dev/null
Divide 2048/#seconds for MB/sec read speed.

Tip: Run nmon (select a for adapter) in another window. You will see the throughput for each adapter.

More information on JFS and JFS2 can be found here.

Topics: AIX, Storage, System Admin

Finding and removing core dumps

Use the following command to interactively find and remove core dumps on your system:

# find / -name core -exec file {} \; -exec rm -i {} \;

Topics: EMC, SAN, Storage

EMC PowerPath key installation

This describes how to configure the EMC PowerPath registration keys.

First, check the current configuration of PowerPath:

# powermt config
Warning: all licenses for storage systems support are missing or expired.
The install the keys:
# emcpreg -install

=========== EMC PowerPath Registration ===========
Do you have a new registration key or keys to enter?[n] y
Enter the registration keys(s) for your product(s),
one per line, pressing Enter after each key.
After typing all keys, press Enter again.

Key (Enter if done): P6BV-4KDB-QET6-RF9A-QV9D-MN3V
1 key(s) successfully added.
Key successfully installed.

Key (Enter if done):
1 key(s) successfully registered.
(Note: the license key used in this example is not valid).

Topics: EMC, SAN, Storage

BCV issue with Solution Enabler

There is a known bug on AIX with Solution Enabler, the software responsible for BCV backups. Hdiskpower devices dissapear and you need to run the following command to make them come back. This will happen when a server is rebooted. BCV devices are only visible on the target servers.

# /usr/lpp/EMC/Symmetrix/bin/mkbcv -a ALL
hdisk2 Available
hdisk3 Available
hdisk4 Available
hdisk5 Available
hdisk6 Available
hdisk7 Available
hdisk8 Available
hdiskpower1 Available
hdiskpower2 Available
hdiskpower3 Available
hdiskpower4 Available

Topics: EMC, SAN, Storage

Reset reservation bit

If you run into not being able to access an hdiskpowerX disk, you may need to reset the reservation bit on it:

# /usr/lpp/EMC/Symmetrix/bin/emcpowerreset fscsiX hdiskpowerX

Topics: EMC, SAN, Storage

EMC Grab

EMC Grab is a utility that is run locally on each host and gathers storage-specific information (driver version, storage-technical details, etc). The EMC Grab report creates a zip file. This zip file can be used by EMC support.

You can download the "Grab Utility" from the following locations:

When you've downloaded EMCgrab, and stored in a temporary location on the server like /tmp/emc, untar it using:
tar -xvf *tar
Then run:
/tmp/emc/emcgrab/emcgrab.sh
The script is interactive and finishes after a couple of minutes.

Topics: Hardware, SAN, SDD, Storage

How-to replace a failing HBA using SDD storage

This is a procedure how to replace a failing HBA or fibre channel adapter, when used in combination with SDD storage:

  • Determine which adapter is failing (0, 1, 2, etcetera):
    # datapath query adapter
  • Check if there are dead paths for any vpaths:
    # datapath query device
  • Try to set a "degraded" adapter back to online using:
    # datapath set adapter 1 offline
    # datapath set adapter 1 online
    (that is, if adapter "1" is failing, replace it with the correct adapter number).
  • If the adapter is still in a "degraded" status, open a call with IBM. They most likely require you to take a snap from the system, and send the snap file to IBM for them to analyze and they will conclude if the adapter needs to be replaced or not.
  • Involve the SAN storage team if the adapter needs to be replaced. They will have to update the WWN of the failing adapter when the adapter is replaced for a new one with a new WWN.
  • If the adapter needs to be replaced, wait for the IBM CE to be onsite with the new HBA adapter. Note the new WWN and supply that to the SAN storage team.
  • Remove the adapter:
    # datapath remove adapter 1
    (replace the "1" with the correct adapter that is failing).
  • Check if the vpaths now all have one less path:
    # datapath query device | more
  • De-configure the adapter (this will also de-configure all the child devices, so you won't have to do this manually), by running: diag, choose Task Selection, Hot Plug Task, PCI Hot Plug manager, Unconfigure a Device. Select the correct adapter, e.g. fcs1, set "Unconfigure any Child Devices" to "yes", and "KEEP definition in database" to "no". Hit ENTER.
  • Replace the adapter: Run diag and choose Task Selection, Hot Plug Task, PCI Hot Plug manager, Replace/Remove a PCI Hot Plug Adapter. Choose the correct device (be careful, you won't see the adapter name here, but only "Unknown", because the device was unconfigured).
  • Have the IBM CE replace the adapter.
  • Close any events on the failing adapter on the HMC.
  • Validate that the notification LED is now off on the system, if not, go back into diag, choose Task Selection, Hot Plug Task, PCI Hot Plug Manager, and Disable the attention LED.
  • Check the adapter firmware level using:
    # lscfg -vl fcs1
    (replace this with the actual adapter name).

    And if required, update the adapter firmware microcode. Validate if the adapter is still functioning correctly by running:
    # errpt
    # lsdev -Cc adapter
  • Have the SAN admin update the WWN.
  • Run:
    # cfgmgr -S
  • Check the adapter and the child devices:
    # lsdev -Cc adapter
    # lsdev -p fcs1
    # lsdev -p fscsi1
    (replace this with the correct adapter name).
  • Add the paths to the device:
    # addpaths
  • Check if the vpaths have all paths again:
    # datapath query device | more

Number of results found for topic Storage: 51.
Displaying results: 31 - 40.