UNIX Health Check - Performance

How to view threads of a process on Linux

Threads are a popular programming abstraction for parallel execution on modern operating systems. When threads are forked inside a program for multiple flows of execution, these threads share certain resources (e.g., memory address space, open files) among themselves to minimize forking overhead and avoid expensive IPC (inter-process communication) channel. These properties make threads an efficient mechanism for concurrent execution.

In Linux, threads (also called Lightweight Processes (LWP)) created within a program will have the same "thread group ID" as the program's PID. Each thread will then have its own thread ID (TID). To the Linux kernel's scheduler, threads are nothing more than standard processes which happen to share certain resources. Classic command-line tools such as ps or top, which display process-level information by default, can be instructed to display thread-level information.

Here are several ways to show threads for a process on Linux:

Using the ps command

The "-T" option for the ps command enables thread views. The following command list all threads created by a process with :

# ps -T -p <pid>

For example to list the threads for the following java process:

# ps -ef | grep 97947
deploy   97947 97942  1 00:51 ?       00:13:51 java

Run the following command:

# ps -T -p 97947
  PID  SPID TTY      TIME CMD
97947 97947 ?    00:00:00 java
97947 97948 ?    00:00:00 java
97947 97949 ?    00:00:00 java

The "SID" column represents thread IDs, and "CMD" column shows thread names.

Using the top command

The top command can show a real-time view of individual threads. To enable thread views in the top output, invoke top with "-H" option. This will list all Linux threads. You can also toggle on or off thread view mode while top is running, by pressing 'H' key.

top - 14:43:25 up 6 days,  5:40,  2 users,  load average: 0.87, 0.33, 0.22
Threads: 684 total,   1 running, 683 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.3 us,  4.0 sy,  0.0 ni, 89.6 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7910136 total,   384812 free,  1603096 used,  5922228 buff/cache
KiB Swap:  8388604 total,  8239100 free,   149504 used.  5514264 avail Mem

Note how in the example above the number of threads on the system is listed.

To restrict the top output to a particular process and check all threads running inside the process:

# top -H -p <pid>

Using htop

A more user-friendly way to view threads per process is via htop, an ncurses-based interactive process viewer. This program allows you to monitor individual threads in tree views.

To enable thread views in htop, launch htop, and press F2 to enter htop setup menu. Choose "Display option" under "Setup" column, and toggle on "Tree view" and "Show custom thread names" options. Presss F10 to exit the setup.

Topics: AIX, Backup & restore, LVM, Performance, Storage, System Admin ↑

Using lvmstat

One of the best tools to look at LVM usage is with lvmstat. It can report the bytes read and written to logical volumes. Using that information, you can determine which logical volumes are used the most.

Gathering LVM statistics is not enabled by default:

# lvmstat -v data2vg
0516-1309 lvmstat: Statistics collection is not enabled for
        this logical device. Use -e option to enable.

As you can see by the output here, it is not enabled, so you need to actually enable it for each volume group prior to running the tool using:

# lvmstat -v data2vg -e

The following command takes a snapshot of LVM information every second for 10 intervals:

# lvmstat -v data2vg 1 10

This view shows the most utilized logical volumes on your system since you started the data collection. This is very helpful when drilling down to the logical volume layer when tuning your systems.

# lvmstat -v data2vg

Logical Volume    iocnt   Kb_read  Kb_wrtn   Kbps
  appdatalv      306653  47493022   383822  103.2
  loglv00            34         0     3340    2.8
  data2lv           453    234543   234343   89.3

What are you looking at here?

iocnt: Reports back the number of read and write requests.
Kb_read: Reports back the total data (kilobytes) from your measured interval that is read.
Kb_wrtn: Reports back the amount of data (kilobytes) from your measured interval that is written.
Kbps: Reports back the amount of data transferred in kilobytes per second.

You can use the -d option for lvmstat to disable the collection of LVM statistics.

Topics: AIX, Backup & restore, LVM, Performance, Storage, System Admin ↑

Spreading logical volumes over multiple disks

A common issue on AIX servers is, that logical volumes are configured on only one single disk, sometimes causing high disk utilization on a small number of disks in the system, and impacting the performance of the application running on the server.

If you suspect that this might be the case, first try to determine which disks are saturated on the server. Any disk that is in use more than 60% all the time, should be considered. You can use commands such as iostat, sar -d, nmon and topas to determine which disks show high utilization. If the do, check which logical volumes are defined on that disk, for example on an IBM SAN disk:

# lspv -l vpath23

A good idea always is to spread the logical volumes on a disk over multiple disk. That way, the logical volume manager will spread the disk I/O over all the disks that are part of the logical volume, utilizing the queue_depth of all disks, greatly improving performance where disk I/O is concerned.

Let's say you have a logical volume called prodlv of 128 LPs, which is sitting on one disk, vpath408. To see the allocation of the LPs of logical volume prodlv, run:

# lslv -m prodlv

Let's also assume that you have a large number of disks in the volume group, in which prodlv is configured. Disk I/O usually works best if you have a large number of disks in a volume group. For example, if you need to have 500 GB in a volume group, it is usually a far better idea to assign 10 disks of 50 GB to the volume group, instead of only one disk of 512 GB. That gives you the possibility of spreading the I/O over 10 disks instead of only one.

To spread the disk I/O prodlv over 8 disks instead of just one disk, you can create an extra logical volume copy on these 8 disks, and then later on, when the logical volume is synchronized, remove the original logical volume copy (the one on a single disk vpath408). So, divide 128 LPs by 8, which gives you 16LPs. You can assign 16 LPs for logical volume prodlv on 8 disks, giving it a total of 128 LPs.

First, check if the upper bound of the logical volume is set ot at least 9. Check this by running:

# lslv prodlv

The upper bound limit determines on how much disks a logical volume can be created. You'll need the 1 disk, vpath408, on which the logical volume already is located, plus the 8 other disks, that you're creating a new copy on. Never ever create a copy on the same disk. If that single disk fails, both copies of your logical volume will fail as well. It is usually a good idea to set the upper bound of the logical volume a lot higher, for example to 32:

# chlv -u 32 prodlv

The next thing you need to determine is, that you actually have 8 disks with at least 16 free LPs in the volume group. You can do this by running:

# lsvg -p prodvg | sort -nk4 | grep -v vpath408 | tail -8
vpath188  active  959   40  00..00..00..00..40
vpath163  active  959   42  00..00..00..00..42
vpath208  active  959   96  00..00..96..00..00
vpath205  active  959  192  102..00..00..90..00
vpath194  active  959  240  00..00..00..48..192
vpath24   active  959  243  00..00..00..51..192
vpath304  active  959  340  00..89..152..99..00
vpath161  active  959  413  14..00..82..125..192

Note how in the command above the original disk, vpath408, was excluded from the list.

Any of the disks listed, using the command above, should have at least 1/8th of the size of the logical volume free, before you can make a logical volume copy on it for prodlv.

Now create the logical volume copy. The magical option you need to use is "-e x" for the logical volume commands. That will spread the logical volume over all available disks. If you want to make sure that the logical volume is spread over only 8 available disks, and not all the available disks in a volume group, make sure you specify the 8 available disks:

# mklvcopy -e x prodlv 2 vpath188 vpath163 vpath208 \
vpath205 vpath194 vpath24 vpath304 vpath161

Now check again with "mklv -m prodlv" if the new copy is correctly created:

# lslv -m prodlv | awk '{print $5}' | grep vpath | sort -dfu | \
while read pv ; do
result=`lspv -l $pv | grep prodlv`
echo "$pv $result"
done

The output should similar like this:

vpath161 prodlv  16  16  00..00..16..00..00  N/A
vpath163 prodlv  16  16  00..00..00..00..16  N/A
vpath188 prodlv  16  16  00..00..00..00..16  N/A
vpath194 prodlv  16  16  00..00..00..16..00  N/A
vpath205 prodlv  16  16  16..00..00..00..00  N/A
vpath208 prodlv  16  16  00..00..16..00..00  N/A
vpath24  prodlv  16  16  00..00..00..16..00  N/A
vpath304 prodlv  16  16  00..16..00..00..00  N/A

Now synchronize the logical volume:

# syncvg -l prodlv

And remove the original logical volume copy:

# rmlvcopy prodlv 1 vpath408

Then check again:

# lslv -m prodlv

Now, what if you have to extend the logical volume prodlv later on with another 128 LPs, and you still want to maintain the spreading of the LPs over the 8 disks? Again, you can use the "-e x" option when running the logical volume commands:

# extendlv -e x prodlv 128 vpath188 vpath163 vpath208 \
vpath205 vpath194 vpath24 vpath304 vpath161

You can also use the "-e x" option with the mklv command to create a new logical volume from the start with the correct spreading over disks.

Topics: AIX, Performance, System Admin ↑

Creating a CSV file from NMON data

Shown below a script that can be used to create a simple comma separated values file (CSV) from NMON data.

If you wish to create a CSV file of the CPU usage on your system, you can grep for "CPU_ALL," in the nmon file. If you want to create a CSV file of the memory usage, grep for "MEM," in the nmon file. The script below creates a CSV file for the CPU usage.

#!/bin/ksh

node=`hostname`
rm -f /tmp/cpu_all.tmp /tmp/zzzz.tmp /tmp/${node}_nmon_cpu.csv
for nmon_file in `ls /var/msgs/nmon/*nmon`
do
  datestamp=`echo ${nmon_file} | cut -f2 -d"_"`
  grep CPU_ALL, $nmon_file > /tmp/cpu_all.tmp
  grep ZZZZ $nmon_file > /tmp/zzzz.tmp
  grep -v "CPU Total " /tmp/cpu_all.tmp | sed "s/,/ /g" | \
  while read NAME TS USER SYS WAIT IDLE rest
  do
    timestamp=`grep ${TS} /tmp/zzzz.tmp | awk 'FS=","{print $4" "$3}'`
    TOTAL=`echo "scale=1;${USER}+${SYS}" | bc`
    echo $timestamp,$USER,$SYS,$WAIT,$IDLE,$TOTAL >> \
    /tmp/${node}_nmon_cpu.csv
  done
  rm -f /tmp/cpu_all.tmp /tmp/zzzz.tmp
done

Note: the script assumes that you've stored the NMON output files in /var/msgs/nmon. Update the script to the folder you're using to store NMON files.

Topics: Performance, Red Hat / Linux, Storage, System Admin ↑

Creating a RAM disk on Linux

On Linux, you can use the tmpfs to create a RAM disk:

# mkdir -p /mnt/tmp
# mount -t tmpfs -o size=20m tmpfs /mnt/tmp

This will create a 20 Megabyte sized RAM file system, mounted on /mtn/tmp. If you leave out the "-o size" option, by default half of the memory will be allocated. However, the memory will not be used, as long as no data is written to the RAM file system.

Topics: AIX, Performance, Storage, System Admin ↑

Creating a RAM disk on AIX

The AIX mkramdisk command allows system administrators to create memory-resident file systems. The performance benefits of using RAM disk can be astonishing. The unload of a large TSM database was reduced from 40 hours on SAN disks down to 10 minutes using RAM disk.

The configuration of a RAM disk file system is very simple and takes just a few minutes. Once the file system is mounted, it can be used like any other file system. There are three steps involved: creating the RAM disk, making the file system and then mounting the file system.

First, we create the RAM disk, specifying the size we want. Let's create a RAM disk of 4 GB:

# mkramdisk 4G

The system will assign the next available RAM disk. Since this is our first one, it will be assigned the name ramdisk0:

# ls -l /dev/ram*
brw-------    1 root system  46, 0 Sep 22 08:01 /dev/ramdisk0

If there isn't sufficient available memory to create the RAM disk you have requested, the mkramdisk command will alert you. Free up some memory or create a smaller size RAM disk. You can use Dynamic LPAR on the HMC or IVM to assign more memory to your partition.

We could use the RAM disk /dev/ramdisk0 as a raw logical volume, but here we’re going to create and mount a JFS2 file system. Here's how to create the file system using the RAM disk as its logical volume:

# mkfs -V jfs2 /dev/ramdisk0

Now create the mount point:

# mkdir -p /ramdisk0

And mount the file system:

# mount -V jfs2 -o log=NULL /dev/ramdisk0 /ramdisk0

Note: mounting a JFS2 file system with logging disabled (log=NULL) only works in AIX 6.1. On AIX 5.3, here are the steps to create the ramdisk:

# mkramdisk 4G
# mkfs -V jfs /dev/ramdisk0
# mkdir /ramdisk0
# mount -V jfs -o nointegrity /dev/ramdisk0 /ramdisk0

You should now be able to see the new file system using df and you can write to it as you would any other file system. When you're finished, unmount the file system and then remove the ramdisk using the rmramdisk command.

# rmramdisk ramdisk0

Topics: AIX, Performance, System Admin ↑

Nmon analyser - A free tool to produce AIX performance reports

Searching for an easy way to create high-quality graphs that you can print, publish to the Web, or cut and paste into performance reports? Look no further. The nmon_analyser tool takes files produced by the NMON performance tool, turns them into Microsoft Excel spreadsheets, and automatically produces these graphs.

You can download the tool here:
http://www.ibm.com/developerworks/aix/library/au-nmon_analyser/

Topics: AIX, Performance, System Admin ↑

Paging space best practices

Here are a couple of rules that your paging spaces should adhere to, for best performance:

The size of paging space should match the size of the memory.
Use more than one paging space, on different disks to each other.
All paging spaces should have the same size.
All paging spaces should be mirrored.
Paging spaces should not be put on "hot" disks.

Topics: AIX, Performance, System Admin ↑

AIX memory consumption

The command svmon -G can be used to determine the actual memory consumption of a server. To determine if the memory is over-committed, you need to divide the memory-virtual value by the memory-size value, e.g.:

# svmon -G
               size       inuse        free         pin     virtual
memory      5079040     5076409        2631      706856     2983249
pg space    7864320       12885

               work        pers        clnt       other
pin          540803           0        2758      163295
in use      2983249           0     2093160

PageSize   PoolSize      inuse       pgsp        pin    virtual
s   4 KB          -    4918761      12885     621096    2825601
m  64 KB          -       9853          0       5360       9853

In this example, the memory-virtual value is 2983249, and the memory-size value is 5079040. Note that the actual memory-inuse (5076409) is nearly the same as the memory-size (5079040) value. This is simply AIX caching as much as possible in its memory. Hence, the memory-free value is typically very low, 2631 in the example above. As such, determining the memory size based on the memory-free value does not provide a good interpretation of the actual memory consumption, as memory typically includes a lot of cached data.

Now, to determine the actual memory consumption, divide memory-virtual by memory-size:

# bc
scale=2
2983249/5079040
.58

Thus, the actual memory consumption is 58% of the memory. The size of the memory is 5079040 blocks of 4 KB = 19840 MB. The free memory is thus: (100% - 58%) * 19840 MB = 8332 MB.

Try to keep the value of memory consumption less than 90%. Above that, you will generally start seeing paging activity using the vmstat command. By that time, it is a good idea to lower the load on the system or to get more memory in your system.

Topics: AIX, Performance, System Admin ↑

Keeping a processor busy

There are times that you would like to create some "load" on the system. A very, very easy way of keeping a processor very busy is:

# yes > /dev/null

The yes command will continiously echo "yes" to /dev/null. This is a single-threaded process, so it will put load on a single processor. If you wish to put load on multiple processors, why not run yes a couple of times?

Number of results found for topic Performance: 13.
Displaying results: 1 - 10.

Order

No time to lose? Need to know what's wrong with
your UNIX system now? Then get started TODAY!