Topics: AIX, System Admin

Storing core files in one location

Core files have the habit to be scattered all over the server, depending on what processes are running, what the working directories are of processes, and which of the processes dump a core file. That is often very annoying, and you may have to use the find command to find all the core files to clean them up.

There is a way to create a centralized repository for your core files, and you can use some not so very well known user settings to do just that.

First, create a location where you can store core files, for example, create a file system /corefiles with plenty of space:

# df -g /corefiles
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/corefilelv     19.92      8.56   58%       94     1% /corefiles
Now, change the default core settings to point to this location:
# chsec -f /etc/security/user -s default -a core_path=on -a core_pathname=/corefiles -a core_compress=on -a core_naming=on
The command above changes the default settings for the core files - for all users. You can obviously do the same for individual users instead, just change "default" to whatever user you want to set this for.

The four options in the chsec command above are:

core_compress

Enables or disables core file compression. Valid values for this attribute are On and Off. If this attribute has a value of On, compression is enabled; otherwise, compression is disabled. The default value of this attribute is Off. This will help you save disk space.

core_path

Enables or disables core file path specification. Valid values for this attribute are On and Off. If this attribute has a value of On, core files will be placed in the directory specified by core_pathname (the feature is enabled); otherwise, core files are placed in the user's current working directory. The default value of this attribute is Off. You'll need to set this if you wish to specify a specific directory to store core files.

core_pathname

Specifies a location to be used to place core files, if the core_path attribute is set to On. If this is not set and core_path is set to On, core files will be placed in the user's current working directory. This attribute is limited to 256 characters. This is where you specifiy the directory to store core files.

core_naming

Selects a choice of core file naming strategies. Valid values for this attribute are On and Off. A value of On enables core file naming in the form core.pid.time, which is the same as what the CORE_NAMING environment variable does. A value of Off uses the default name of core. This will create core files with the name in a form of core.pid.time, where time is ddhhmmss (pid = the process id; dd = day of the month, hh = hours, mm = mintues, ss = seconds). You can leave out this option and instead set environment variable CORE_NAMING to true.

Doing so - and after restarting any applications (or the whole server), your core files should now be all stored in /corefiles - that is, if you have any processes that generate core files of course.

Note: The same can be achieved with the chcore command:
# chcore -c on -p on -l /corefiles -n on -d
Validate the settings as follows:
# grep -p default /etc/security/user | grep core
        core_compress = on
        core_path = on
        core_naming = on
        core_pathname = /corefiles

Topics: AIX, System Admin

Use machstat to identify power or cooling issues

Not very much known is the machstat command in AIX that can be used to display the status of the Power Status Register, and thus can be helpful to identify any issues with either Power or Cooling.

# machstat -f
0 0 0
If it returns all zeroes, everything is fine. Anything else is not good. The first digit (the so-called EPOW Event) indicates the type of problem:

EPOW EventDescription
0normal operation
1non-critical cooling problem
2non-critical power problem
3severe power problem - halt system
4severe problems - halt immediately
5unhandled issue
7unhandled issue

Another way to determine if the system may have a power or cooling issue, is by looking at a crontab entry in the root user's crontab:
# crontab -l root | grep -i powerfail
0 00,12 * * * wall%rc.powerfail:2::WARNING!!! The system is now operating with a power problem. This message will be walled every 12 hours. Remove this crontab entry after the problem is resolved.
If a powerfail message is present in the crontab of user root, this may indicate that there is an issue to be looked into. Contact your IBM representative to check the system out. Afterwards, make sure to remove the powerfail entry from the root user's crontab.

Topics: AIX, LVM, System Admin

LVM command history

Want to know which LVM commands were run on a system? Simply run the following command, and get a list of the LVM command history:

# alog -o -t lvmcfg
To filter out only the actual commands:
# alog -o -t lvmcfg | grep -v -E "workdir|exited|tellclvmd"
[S 06/11/13-16:52:02:236 lvmstat.c 468] lvmstat -v testvg
[S 06/11/13-16:52:02:637 lvmstat.c 468] lvmstat -v rootvg
[S 07/20/13-15:02:15:076 extendlv.sh 789] extendlv testlv 400
[S 07/20/13-15:02:33:199 chlv.sh 527] chlv -x 4096 testlv
[S 08/22/13-12:29:16:807 chlv.sh 527] chlv -e x testlv
[S 08/22/13-12:29:26:150 chlv.sh 527] chlv -e x fslv00
[S 08/22/13-12:29:46:009 chlv.sh 527] chlv -e x loglv00
[S 08/22/13-12:30:55:843 reorgvg.sh 590] reorgvg
The information for this LVM command history is stored in /var/adm/ras/lvmcfg.log. You can check the location for a circular log, by running:
# alog -t lvmcfg -L
#file:size:verbosity
/var/adm/ras/lvmcfg.log:51200:3
More detail can also be found in the lvmt log, by running:
# alog -t lvmt -o

Topics: AIX, System Admin

Suspending and resuming a process

You may be familiar with suspending a process that is running in the foreground by pressing CTRL-Z. It will suspend the process, until you type "fg", and the process will resume again.

# sleep 400
After pressing CTRL-Z, you'll see:
[1] + Stopped (SIGTSTP)        sleep 400
Then type "fg" to resume the process:
# fg
sleep 400
But what if you wish to suspend a process that is not attached to a terminal, and is running in the background? This is where the kill command is useful. Using signal 17, you can suspend a process, and using signal 19 you can resume a process.

This is how it works: First look up the process ID you wish to suspend:
# sleep 400 &
[1]     8913102
# ps -ef | grep sleep
    root  8913102 10092788   0 07:10:30  pts/1  0:00 sleep 400
    root 14680240 10092788   0 07:10:34  pts/1  0:00 grep sleep
Then suspend the process with signal 17:
# kill -17 8913102
[1] + Stopped (SIGSTOP)        sleep 400 &
To resume it again, send signal 19:
# kill -19 8913102

Topics: AIX, System Admin

RANDOM Korn Shell built-in

The use of $RANDOM in Korn Shell can be very useful. Korn shell built-in $RANDOM can generate random numbers in the range 0:32767. At every call a new random value is generated:

# echo $RANDOM
19962
# echo $RANDOM
19360
The $RANDOM Korn shell built-in can also be used to generate numbers within a certain range, for example, if you want to run the sleep command for a random number of seconds.

To sleep between 1 and 600 seconds (up to 10 minutes):
# sleep $(print $((RANDOM%600+1)))

Topics: AIX, System Admin

Number of active virtual processors

To know quickly how many virtual processors are active, run:

# echo vpm | kdb
For example:
# echo vpm | kdb
...
VSD Thread State.
 CPU VP_STATE   SLEEP_STATE  PROD_TIME: SECS   NSECS     CEDE_LAT

   0  ACTIVE    AWAKE        0000000000000000  00000000  00
   1  ACTIVE    AWAKE        0000000000000000  00000000  00
   2  ACTIVE    AWAKE        0000000000000000  00000000  00
   3  ACTIVE    AWAKE        0000000000000000  00000000  00
   4  DISABLED  AWAKE        00000000503536C7  261137E1  00
   5  DISABLED  SLEEPING     0000000051609EAF  036D61DC  02
   6  DISABLED  SLEEPING     0000000051609E64  036D6299  02
   7  DISABLED  SLEEPING     0000000051609E73  036D6224  02

Topics: AIX, System Admin

How to read the /var/adm/ras/diag log file

There are 2 ways for reading the Diagnostics log file, located in /var/adm/ras/diag:

The first option uses the diag tool. Run:

# diag
Then hit ENTER and select "Task Selection", followed by "Display Previous Diagnostic Results" and "Display Previous Results".

The second option is to use diagrpt. Run:
# /usr/lpp/diagnostics/bin/diagrpt -s 010101
To display only the last entry, run:
# /usr/lpp/diagnostics/bin/diagrpt -o

Topics: AIX, Backup & restore, System Admin, Virtual I/O Server, Virtualization

How to make a system backup of a VIOS

To create a system backup of a Virtual I/O Server (VIOS), run the following commands (as user root):

# /usr/ios/cli/ioscli viosbr -backup -file vios_config_bkup
-frequency daily -numfiles 10
# /usr/ios/cli/ioscli backupios -nomedialib -file /mksysb/$(hostname).mksysb -mksysb
The first command (viosbr) will create a backup of the configuration information to /home/padmin/cfgbackups. It will also schedule the command to run every day, and keep up to 10 files in /home/padmin/cfgbackups.

The second command is the mksysb equivalent for a Virtual I/O Server: backupios. This command will create the mksysb image in the /mksysb folder, and exclude any ISO repositiory in rootvg, and anything else excluded in /etc/exclude.rootvg.

Topics: AIX, Backup & restore, Storage, System Admin

Using mkvgdata and restvg in DR situations

It is useful to run the following commands before you create your (at least) weekly mksysb image:

# lsvg -o | xargs -i mkvgdata {}
# tar -cvf /sysadm/vgdata.tar /tmp/vgdata
Add these commands to your mksysb script, just before running the mksysb command. What this does is to run the mkvgdata command for each online volume group. This will generate output for a volume group in /tmp/vgdata. The resulting output is then tar'd and stored in the /sysadm folder or file system. This allows information regarding your volume groups, logical volumes, and file systems to be included in your mksysb image.

To recreate the volume groups, logical volumes and file systems:
  • Run:
    # tar -xvf /sysadm/vgdata.tar
  • Now edit /tmp/vgdata/{volume group name}/{volume group name}.data file and look for the line with "VG_SOURCE_DISK_LIST=". Change the line to have the hdisks, vpaths or hdiskpowers as needed.
  • Run:
    # restvg -r -d /tmp/vgdata/{volume group name}/{volume group name}.data
Make sure to remove file systems with the rmfs command before running restvg, or it will not run correctly. Or, you can just run it once, run the exportvg command for the same volume group, and run the restvg command again. There is also a "-s" flag for restvg that lets you shrink the file system to its minimum size needed, but depending on when the vgdata was created, you could run out of space, when restoring the contents of the file system. Just something to keep in mind.

Topics: AIX, System Admin

A quick way to remove all printer queues

Here's a quick way to remove all the printer queues from an AIX system:

/usr/lib/lpd/pio/etc/piolsvp -p | grep -v PRINTER | \
   while read queue device rest ; do
   echo $queue $device
   rmquedev -q$queue -d$device
   rmque -q$queue
done

Number of results found for topic AIX: 231.
Displaying results: 31 - 40.