When you're using HACMP, you usually have multiple network adapters installed and thus multiple network interface to handle with. If AIX configured the default gateway on a wrong interface (like on your management interface instead of the boot interface), you might want to change this, so network traffic isn't sent over the management interface. Here's how you can do this:

First, stop HACMP or do a take-over of the resource groups to another node; this will avoid any problems with applications when you start fiddling with the network configuration.

Then open up a virtual terminal window to the host on your HMC. Otherwise you would loose the connection, as soon as you drop the current default gateway.

Now you need to determine where your current default gateway is configured. You can do this by typing:

# lsattr -El inet0
# netstat -nr

# odmget -q"attribute=route" CuAt

# lsattr -El inet0 | awk '$2 ~ /hopcount/ { print $2 }' | read GW
# chdev -l inet0 -a delroute=${GW}

# route add 0 [ip address of default gateway: xxx.xxx.xxx.254] -if enX

# chdev -l inet0 -a addroute=net,-hopcount,0,,0,[ip address of default gateway]

# chdev -l inet0 -a addroute=net,-hopcount,0,if,enX,,0,[ip address of default gateway]

# lsattr -El inet0
# odmget -q"attribute=route" CuAt

• clstat - show cluster state and substate; needs clinfo.
• cldump - SNMP-based tool to show cluster state.
• cldisp - similar to cldump, perl script to show cluster state.
• cltopinfo - list the local view of the cluster topology.
• clshowsrv -a - list the local view of the cluster subsystems.
• clfindres (-s) - locate the resource groups and display status.
• clRGinfo -v - locate the resource groups and display status.
• clcycle - rotate some of the log files.
• cl_ping - a cluster ping program with more arguments.
• clrsh - cluster rsh program that take cluster node names as argument.
• clgetactivenodes - which nodes are active?
• get_local_nodename - what is the name of the local node?
• clconfig - check the HACMP ODM.
• clRGmove - online/offline or move resource groups.
• cldare - sync/fix the cluster.
• cllsgrp - list the resource groups.
• clsnapshotinfo - create a large snapshot of the HACMP configuration.
• cllscf - list the network configuration of an HACMP cluster.
• clshowres - show the resource group configuration.
• cllsif - show network interface information.
• cllsres - show short resource group information.
• lssrc -ls clstrmgrES - list the cluster manager state.
• lssrc -ls topsvcs - show heartbeat information.
• cllsnode - list a node centric overview of the hacmp configuration.

If you wish to enable MAC Address take-over on an HACMP cluster, you need a virtual MAC address. You can do a couple of things to make sure you have a unique MAC Address on your network:

• Use the MAC address of an old system, that you know has been destroyed.
• Buy a new network card, use the MAC address, then destroy this card.
• Use a DEADBEEF address: (0xdeadbeef1234). This is a non-existent hardware vendor. You might run into problems with someone else making up a deadbeef address, so use this option with caution.

HACMP provides events, which can be used to most accurately monitor the cluster status, for example via the Tivoli Enterprise Console. Each change in the cluster status is the result of an HACMP event. Each HACMP event has an accompanying notify method that can be used to handle the kind of notification we want.

Interesting Cluster Events to monitor are:

• node_up
• node_down
• network_up
• network_down
• join_standby
• fail_standby
• swap_adapter
• config_too_long
• event_error

# smitty hacmp
Cluster Configuration
Cluster Resources
Cluster Events
Change/Show Cluster Events

# odmget HACMPevent

This describes how to configure the EMC PowerPath registration keys.

First, check the current configuration of PowerPath:

# powermt config
Warning: all licenses for storage systems support are missing or expired.

# emcpreg -install

=========== EMC PowerPath Registration ===========
Do you have a new registration key or keys to enter?[n] y
Enter the registration keys(s) for your product(s),
one per line, pressing Enter after each key.
After typing all keys, press Enter again.

Key (Enter if done): P6BV-4KDB-QET6-RF9A-QV9D-MN3V
1 key(s) successfully added.
Key successfully installed.

Key (Enter if done):
1 key(s) successfully registered.

• Make sure the relay host, e.g. the Exchange server, allows incoming email from your Linux server.
• Make sure no firewall is blocking SMTP traffic from the Linux host. You can use nmap for this purpose:

  # nmap -sS smtp.server.com

  (Replace "smtp.server.com" for the actual SMTP server hostname of your environment).
• Check it the DNS configuration is correct in /etc/resolv.conf and make sure you can resolve the hostname and its IP address reversely:

  # nslookup hostname
  # nslookup ipaddress
  

  (use the IP address returned by the first DNS lookup on the hostname to reversely lookup the hostname by the IP address).
• Make a copy of sendmail.mc and sendmail.cf in /etc/mail.
• Edit sendmail.mc (add in the name of your SMTP server):

  define(`confTRUSTED_USER', `root')dnl
  define(`SMART_HOST', `esmtp:smtp.server.com')dnl
  MASQUERADE_AS(`hostname.com')dnl
  FEATURE(masquerade_envelope)dnl
  FEATURE(masquerade_entire_domain)dnl
  

• Then run:

  # make -C /etc/mail

• Edit sendmail.cf by modifying the "C{E}" line in sendmail.cf. Take any user listed on that line including root off that line, so mail sent from root gets masqueraded as well. Towards the bottom of sendmail.cf file, there is a section for Ruleset 94. Make sure that after "R$+" there is ONE tab (no space, or multiple spaces/tabs):

  SMasqEnv=94
  R$+ $@ $>MasqHdr $1
  

• Clean out /var/spool/clientmqueue and /var/spool/mqueue (there may be lots of OLD emails there, we may not want to send these anymore).
• Then restart sendmail:

  # service sendmail restart

  (or "service sendmail start" if it isn't running yet; check the status with: "service sendmail status").
• Make sure that sendmail is started at system restart:

  # chkconfig sendmail on # chkconfig --list sendmail

• Open a "tail -f /var/log/maillog" so you can watch any syslog activity for mail (of course there should be a "mail.*" entry in /etc/syslog.conf directing output to /var/log/maillog for this to work).
• Send a test email message:

  # echo "test" | sendmail -v address@email.com

  (and check that the email message is actually accepted for delivery in the verbose output).
• Wait for the mail to arrive in your mailbox.

This is a procedure to enable time synchronization (ntpd) on Linux (in this example, replace the IP address of the time server with the IP address of your time server):

• Stop all applications on the server.
• Check if you can access the time servers, e.g.:

  # ntpdate -q 10.250.9.11

• Check if the current timezone setting is correct by simply running the date command.
• Set the time and date correct:

  # ntpdate 10.250.9.11

• Start the NTP server:

  # service ntpd start

• Check the status:

  # service ntpd status

• Check the time synchronization (it may take some time for the client to synchronize with its time server):

  # ntpq -p

• Check that ntpd is started at system restart:

  # chkconfig ntpd on
  # chkconfig --list | grep ntpd

• Check the process:

  # ps -ef | grep ntpd

• Reboot the server:

  # reboot

This describes how to get SSH access to a Linux system on a HP blade system, which requires you to work through the ILO:

First of all, you need to know the ILO IP address. Simply open up an SSH session to this IP address:

# ssh -l ilo-admin 10.250.21.37
ilo-admin@10.250.21.37's password:
User:ilo-admin logged-in to 10.250.21.37
iLO 2 Advanced 1.60 at 16:05:58 Jul 11 2008
Server Name:
Server Power: On

</>hpiLO->

hpiLO-> VSP
Starting virtual serial port
Press 'ESC (' to return to the CLI Session
</>hpiLO-> Virtual Serial Port active: IO=0x02F8 INT=3
[ENTER]
</>hpiLO-> Virtual Serial Port active: IO=0x02F8 INT=3
Red Hat Enterprise Linux ES release 4 (Nahant Update 8)
Kernel 2.6.9-89.ELsmp on an i686
hostname login:

mo1::off:/sbin/mgetty -x 0 -D -s38400 -a /dev/ttyS1

ilo:2345:respawn:/sbin/agetty ttyS1 115200 vt100

# echo "ttyS1" >> /etc/securetty

# kill -HUP 1

If you're receiving messages like these on your console:

Mar 9 11:47:29 daemon:notice bootpd[192990]: received short packet
Mar 9 11:47:31 daemon:notice bootpd[192990]: received short packet
Mar 9 11:47:38 daemon:notice bootpd[192990]: hardware address not found: E41F132E3D6C

# refresh -s inetd

#daemon.notice /dev/console
daemon.notice /nsr/logs/messages

# refresh -s syslogd

To manage Linux server, running on HP blades, you need to be able to access the ILO. Without it, you won't be able to remotely access the console or to reboot the server. If your ILO is on a separate network and your only access is through SSH, it can be difficult to access the ILO, since the web-based GUI is only supported in Microsoft Internet Explorer. By means of SSH tunneling however, it is possible to access the ILO.

First, you need to have a UNIX box which is capable of accessing the ILO directly through SSH (in our example called "jumpserver"). Also, you'll need to have the ILO's IP address.

On your Linux client in a system console (let's assume, this client is called "desktop"), create the SSH tunnel and forward the correct ports needed for ILO access, by running as user root:

# ssh -g -L 23:10.250.21.38:23 -L 443:10.250.21.38:443 -L 17988:10.250.21.38:17988 -L 3389:10.250.21.38:3389 jumpserver -N