Topics: AIX, Networking

Etherchannel failover testing

When an Etherchannel has been configured on AIX, using a primary and a backup adapter for failover purposes, it is possible to force a failover between these adapters.

To do this, first check what the currently active adapter is within the Etherchannel. For example, if the Etherchannel is called ent4, run:

# entstat -d ent4 | grep "Active channel"
Active channel: backup adapter
As you can see, the Etherchannel is currently active on the backup adapter. To see the defined adapters for both primary and backup, run:
# lsattr -El ent4
Now, force it to fail over to the primary adapter:
# /usr/lib/methods/ethchan_config -f ent4
Please note that it is best to run this command from the console, as you may temporarily lose network connectivity when the Etherhannel is failing over. You may also notice messages being logged in the error report by running the errpt command in the form of "ETHERCHANNEL FAILOVER".

Now run the entstat command again to determine the active channel:
# entstat -d ent4 | grep "Active channel"
Active channel: primary channel

Topics: Networking

Using tcpdump to monitor DHCP network traffic

The tcpdump command can be used to monitor DHCP related network traffic. This is very useful in cases where DHCP issues may have to be investigated. Basically, the tcpdump command can be used to do some packet sniffing on the network.

The method to capture DHCP traffic is to define a filter so that tcpdump dumps only DHCP related traffic. In DHCP, UDP port number 67 is used by a DHCP server, and UDP port number 68 is used by DHCP clients. Thus, you want to capture traffic with port number 67 or 68 as follows, assuming that eth0 is the network interface that will be used to monitor:

# tcpdump -i eth0 port 67 or port 68 -e -n -vv
By using the -vv option (for very verbose) you may see a lengthy output from the tcpdump command displaying a lot of information. In it you can see which DHCP server is responding, and what IP address is assigned. For example:
Client-ID Option 61, length 7: ether ec:9b:f3:6b:97:4b
Requested-IP Option 50, length 4: 192.168.0.3
Server-ID Option 54, length 4: 192.168.0.1
MSZ Option 57, length 2: 1500
Vendor-Class Option 60, length 16: "android-dhcp-7.0"
Hostname Option 12, length 16: "SAMSUNG-SM-G890A"
In the example above, you can see that a Samsung SM-G890A phone, running Android, gets IP address 192.168.0.3 assigned from DHCP server 192.168.0.1. You can also see the MAC (or "hardware") address of the phone: ec:9b:f3:6b:97:4a.

One you're finished sniffing the network for DHCP related traffic, you can simply CTRL-C out of the tcpdump command.

Topics: Networking, Red Hat

Using tcpdump to discover network information

For most switches it is impossible to see which switch and switch port you are when you are connected to an 'access' port.

Using the Cisco Discovery Protocol or CDP (Cisco) and Link Layer Discovery Protocol or LLDP (Juniper or Dell) you can find out quite a bit of information about the switch that a host is connected to.

Enabling CDP/LLDP on an access port is arguably a security risk (information exposure), so it might not be enabled on your network. You can use the tcpdump command to disassemble CDP/LLDP packets which will usually show information like the name of the switch, its IP address, the switch port connected to, and sometimes the VLAN in use.

For Cisco CDP, assuming the network interface you wish to check is called "eth0":

# tcpdump -nn -v -i eth0 -s 1500 -c 1 'ether[20:2] == 0x2000' 
For Juniper LLDP:
# tcpdump -nn -v -i eth0 -s 1500 -c 1 '(ether[12:2]=0x88cc or ether[20:2]=0x2000)'

Topics: Networking, Red Hat

How to add a new static route on RHEL 7

Any network we are trying to reach is accessed via the default gateway only, if it is not implicitly overwritten by another static route definition. Let's have a look at a routing table on a Red Hat 7 system:

# ip route show
default via 192.168.0.1 dev em1 proto static metric 100
192.168.0.0/24 dev em1 proto kernel scope link src 192.168.0.204 metric 100
192.168.1.0/24 dev em2 proto kernel scope link src 192.168.1.32
From the above we can see that any packets to reach a destination network IP 192.168.0.0/24 (meaning anything on the network 192.168.0.X with a subnet mask of 255.255.255.0) should travel via interface em1 with IP address 192.168.0.204, and any other destination network not implicitly defined should use default gateway 192.168.0.1.

Sometimes you'll require a static route. Static routes are for traffic that must not, or should not, go through the default gateway. Routing is often handled by devices on the network dedicated to routing (although any device can be configured to perform routing). Therefore, it is often not necessary to configure static routes on RHEL servers or clients. Exceptions include traffic that must pass through an encrypted VPN tunnel, or traffic that should take a specific route for reasons of cost or security or bandwidth. The default gateway is for any and all traffic which is not destined for the local network and for which no preferred route is specified in the routing table. The default gateway is traditionally a dedicated network router.

To add a new static route means to define yet another destination network as well as specify via which IP address and interface the packet should travel through in order to reach its destination. Usually this comes in handy when you have a second interface on the system that can be used to reach other networks (other than the networks that can be reached through the default gateway). In the example above that's interface em2.

For example, let's add a static route to destination network 192.168.2.0/24 via the 192.168.1.32 IP address and em2 network interface.

There are 2 ways of accomplishing this: By either using the "ip route add" command, which will define the route, but it will be lost upon reboot. Or by creating a route configuration file in the /etc/sysconfig/network-scripts/ directory.

First, the "ip route add" command:

To add a static route to a network, in other words, representing a range of IP addresses, issue this command as root:
# ip route add 192.168.2.0/24 via 192.168.1.32 dev em2
Where 192.168.2.0 is the IP address of the destination network in dotted decimal notation and /24 is the network prefix, which is equal to a subnet mask of 255.255.255.0. The network prefix is the number of enabled bits in the subnet mask. If you now rerun the "ip route show" command, you'll see that the route has been added.
192.168.2.0/24 via 192.168.1.32 dev em2 proto static metric 1
If you ever need to delete the route, you can use the same command, but just replace "add" with "delete".

Static route configuration can be stored per-interface in a /etc/syconfig/network-scripts/route-interface file. For example, fstatic routes for the em2 interface would be stored in the /etc/sysconfig/network-scripts/route-em2 file, for example:
# cat /etc/sysconfig/network-scripts/route-em2
192.168.2.0/24 via 192.168.1.32 dev em2
Once done, restart your network service (or restart the server):
# systemctl restart network

Topics: Networking, Red Hat, Storage

How to install and configure Samba on CentOS 7 for file sharing on Windows

Here's how to set up a secure Samba share from a CentOS 7 (or RHEL 7) system, and share it with a Windows client.

First, install Samba:

# yum install samba samba-client samba-common
Add an exception to the firewall, if the firewall is active:
# firewall-cmd --permanent --zone=public --add-service=samba
# firewall-cmd --reload
Next, you'll need to know the workgroup the Windows system is configured in. By far, the easiest way to do this, is to open a command prompt on the Windows system, and run:
net config workstation
For the sake of this tutorial, we'll assume the workgroup is called WORKGROUP.

Make a copy of the Samba config file:
# cp /etc/samba/smb.conf /etc/samba/smb.conf.orig
Set up a secure file share. In the example below, the share will be located in /media/windows/share on the CentOS 7 system. Be sure to set the permissions in such a way that the user account used for the share (see below) indeed has access to this folder.
# mkdir -p /media/windows/share
# chmod -R 0755 /media/windows/share
# chown -R user:group /media/windows/share
Edit file /etc/samba/smb.conf and add:
[global]
        workgroup = WORKGROUP
        netbios name = centos

[Share]
        comment = Shared Folder
        path = /media/windows/share
        valid users = user
        browsable = yes
        writable = yes
        guest ok = no
        read only = no
Set the SMB passwd for the user (this will be the username and password used to access the share from Windows):
# smbpasswd -a user
New SMB password:
Retype new SMB password:
Make sure everything is okay:
# testparm
Now enable and start Samba:
# systemctl enable smb.service
# systemctl enable nmb.service
# systemctl start smb.service
# systemctl start nmb.service
On the Windows host, io File explore type the IP address of the CentOS system, for example:
\\192.168.0.206
You will be asked for the username and password used when you ran the smbpasswd command.

And that should do it; You should now have a secured Samba share available on a Windows system.

Windows may cache any credentials that are used for the Samba share(s). When configuring the Samba share(s), it may be needed to have Windows "forget" these credentials. This can be easily achieved by running from a Command Prompt:
net use * /del

Topics: Monitoring, Networking, Red Hat

Securely enabling SNMP on Red Hat

Monitoring tools often use SNMP to query another system's information and status. For that to work on a Red Hat Enterprise Linux system, that system will have to have SNMP configured. And to allow a remote (monitoring) system to query SNMP information of a Red Hat Enterprise Linux system, one has to complete the following 3 items:

  • Set up SNMP.
  • Configure SNMP to use a non-public community name.
  • Allow access through the firewall, if configured.
For the configuration of SNMP, you'll need to install the following 2 packages:
# yum -y install net-snmp net-snmp-utils
Next, start and enable (at boot time) the SNMP daemon to run on the system:
# systemctl enable snmpd
# systemctl start snmpd
Now you can test if you can query SNMP infomation -locally- on the system, by using the snmpwalk command:
# snmpwalk -v2c -c public localhost | head -5
The community string used above ("public") is a well-known SNMP community string, and this can be (and probably "is") utilized by hackers or other unfriendly people to obtain information about the system remotely, and as such, it's best practice to change the public community name into something a littlebit different, preferably something that can't be guessed very easily. For the sake of this tutorial, we'll change it to "kermit".

Basically, you'll have to update this line in /etc/snmp/snmpd.conf from "public" to "kermit":

Before:
com2sec notConfigUser  default       public
After:
com2sec notConfigUser  default       kermit
Then, restart the SNMP daemon, so it picks up the changes to configuration file /etc/snmp/snmpd.conf:
# systemctl restart snmpd
Now test again with the snmpwalk command but this time by using the "kermit" community name:
# snmpwalk -v2c -c kermit localhost
That should give you quite a bit of output. If it doesn't, you've made a mistake, and you'll have to re-trace your steps.

The final step is to allow remote access. That will be needed if a remote system is being used to monitor the server, for example by a tool like Solarwinds. By default, remote access will be blocked by the firewall daemon on the system. To allow remote access, open up UDP port 161 on the client:
# firewall-cmd --zone=public --add-port=161/udp --permanent
# firewall-cmd --reload
Now log in to a remote system and run a similar snmpwalk command, but this time, specify the hostname of the server that you're querying (instead of "localhost"). For example, if the name of the host is "myserver", run:
# snmpwalk -v2c -c kermit myserver
And that's it. You can now remotely monitor a Linux server using SNMP, and you've secured it by changing the community name.

Topics: Networking, Red Hat

Running tcpdump

From time to time, there may be a need to run a tcpdump, to analyze the TCP traffic on a Red Hat system.

Now, there's a perfectly good description on how to that on the Red Hat website at https://access.redhat.com/solutions/8787, so we won't be repeating that on this blog.

Just a few simple commands to get the tcpdump command going:

To start a tcpdump, for example on network interface em1, and dump the output to a file called /tmp/tcpdump.out, run:

# tcpdump -s 0 -i em1 -w /tmp/tcpdump.out -v
The "-v" option used in the example above, shows the number of packets that it captured, while the tcpdump command is running, and thus is very useful. Once you think you have gathered enough information, hit CTRL-C to stop the tcpudmp. Be careful, running tcpdump can create quite a bit of output, especially if there's a lot of network traffic going on. This may fill up the the file system where the tcpdump output file is located in, pretty quickly, so don't leave the tcpdump running for prolonged periods of time.

To review the contents of the tcpdump output, use the "-r" option:
# tcpdump -r /tmp/tcpdump.out
The "tcpdump -r" command will show you detailed information about the captured network packets.

Topics: Networking, Red Hat

Setting up a bonded network interface on RHEL 7

The following procedure describes how to set up a bonded network interface on Red Hat Enterprise Linux. It assumes that you already have a working single network interface, and wish to now move the system to a bonded network interface set-up, to allow for network redundancy, for example by connecting two separate network interfaces, preferably on two different network cards in the server, to two different network switches. This provides both redundancy, should a network card in the server fail, and also if a network switch would fail.
First, log in as user root on the console of the server, as we are going to change the current network configuration to a bonded network configuration, and while doing so, the system will lose network connectivity temporarilty, so it is best to work from the console.

In this procedure, We'll be using network interfaces em1 and p3p1, on two different cards, to get card redundancy (just in case one of the network cards will fail).

Let's assume that IP address 172.29.126.213 is currently configured on network interface em1. You can verify that, by running:

# ip a s
Also, we'll need to verify, using the ethtool command, that there is indeed a good link status on both the em1 and p3p1 network interfaces:
# ethtool em1
# ethtool p3p1
Run, to list the bonding module info (should be enabled by default already, so this is just to verify):
# modinfo bonding
Create copies of the current network files, just for safe-keeping:
# cd /etc/sysconfig/network-scripts
# cp ifcfg-em1 /tmp
# cp ifcfg-p3p1 /tmp
Now, create a new file ifcfg-bond0 in /etc/sysconfig/network-scripts. We'll configure the IP address of the system (the one that was configured previously on network interface em1) on a new bonded network interface, called bond0. Make sure to update the file with the correct IP address, gateway and network mask for your environment:
# cat ifcfg-bond0
DEVICE=bond0
TYPE=Bond
NAME=bond0
BONDING_MASTER=yes
BOOTPROTO=none
ONBOOT=yes
IPADDR=172.29.126.213
NETMASK=255.255.255.0
GATEWAY=172.29.126.1
BONDING_OPTS="mode=5 miimon=100"
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
The next thing to do is to create two more files, one for each network interfaces that will be the slaves of the bonded network interface. In our example, that will be em1 and p3p1.

Create file /etc/sysconfig/network-scripts/ifcfg-em1 (be sure to update the file to your environment, for example, use the correct UUID. You may find that in the copies you've made of the previous network interface files. In this file, you'll also specify that the bond0 interface is now the master.
# cat ifcfg-em1
TYPE=Ethernet
BOOTPROTO=none
NAME=em1
UUID=cab24cdf-793e-4aa7-a093-50bf013910db
DEVICE=em1
ONBOOT=yes
MASTER=bond0
SLAVE=yes
Create file ifcfg-p3p1:
# cat ifcfg-p3p1
TYPE=Ethernet
BOOTPROTO=none
NAME=p3p1
UUID=5017c829-2a57-4626-8c0b-65e807326dc0
DEVICE=p3p1
ONBOOT=yes
MASTER=bond0
SLAVE=yes
Now, we're ready to start using the new bonded network interface. Restart the network service:
# systemctl restart network.service
Run the ip command to check the current network config:
# ip a s
The IP address should now be configured on the bond0 interface.

Ping the default gateway, to test if your bonded network interface can reach the switch. In our example, the default gateway is set to 172.29.126.1:
# ping 172.29.126.1
This should work. If not, re-trace the steps you've done so far, or work with your network team to identify the issue.

Check that both interfaces of the bonded interface are up, and what the current active network interface is. You can do this by looking at file /proc/net/bonding/bond0. In this file you can see what the currently active slave is, and if all slaves of the bonded network interface are up. For example:
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: transmit load balancing
Primary Slave: None
Currently Active Slave: p3p1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: p3p1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0a:f7:ce:26:30
Slave queue ID: 0

Slave Interface: em1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0a:f7:bd:b7:9e
Slave queue ID: 0
In the example above, the active network interface is p3p1. Let's bring it down, to see if it fails over to network interface em1. You can bring down a network interface using the ifdown command:
# ifdown p3p1
Device 'p3p1' successfully disconnected.
Again, look at the /proc/net/bonding/bond0 file. You can now see that the active network interface has changed to em1, and that network interface p3p1 is no longer listed in the file (because it is down):
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: transmit load balancing
Primary Slave: None
Currently Active Slave: em1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: em1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0a:f7:bd:b7:9e
Slave queue ID: 0
Now ping the default gateway again, and make sure it still works (now that we're using network interface em1 instead of network interface p3p1).

Then bring the p3p1 interface back up, using the ifup command:
# ifup p3p1
And check the bonding status again:
# cat /proc/net/bonding/bond0
It should show that the active network interface is still on em1, it will not fail back to network interface p3p1 (After all, why would it?! Network interface em1 works just fine).

Now repeat the same test, by bringing down network interface em1, ping the default gateway again, and check the bonding status, and bring em1 back up:
# ifdown em1
# cat /proc/net/bonding/bond0
# ping 172.29.126.1
# ifup em1
# cat /proc/net/bonding/bond0
# ping 172.29.126.1
If this all works fine, then you're all set.

Topics: AIX, Networking

Adding and deleting a static network route using the command line

There are two commands that can be used to add a route on an AIX system.

The first one is route, and can be used to temporarily add a route to an AIX system. Meaning, if the system is rebooted, after the route has been added, the route will be lost again after the reboot.

The second command is chdev -l inet0 that can be used to permanently add a route on an AIX system. When this command is used, the route will persist during reboots, as this command writes to information of the route in to the ODM of AIX.

Let's say you have a need to add a route on a system to network 10.0.0.0. And that network uses a netmask of 255.255.255.0 (or "24" for the short mask notation). Finally, the gateway that can be used to access this network is 192.168.0.1. Obviously, please adjust this to your own situation.

To temporarily add a route on a system for this network, use the following route command:

# route add -net 10.0.0.0 -netmask 255.255.255.0 192.168.0.1
After running this command, you can use the netstat -nr command to confirm that the route indeed has been set up:
# netstat -nr | grep 192.168.0.1
172.30.224/24   192.168.0.1   UG   0   0   en1   -   -
To remove that route again, simply change the route command from "add" to "delete":
# route delete -net 10.0.0.0 -netmask 255.255.255.0 192.168.0.1
Again, confirm with the netstat -nr command that the route indeed has been removed.

Now, as mentioned earlier, the route command will only temporarily (until the next reboot) add a route on the AIX system. To make things permanent, use the chdev command. This command takes the following form:

chdev -l inet0 -a route=net,-netmask,[your-netmask-goes-here],-static,[your-network-address-goes-here],[your-gateway-goes-here]

For example:
# chdev -l inet0 -a route=net,-netmask,255.255.255.0,-static,10.0.0.0,
192.168.0.1
inet0 changed
This time, again, you can confirm with the netstat -nr command that the route has been set up. But now you can also confirm that the route has been added to the ODM, by using this command:
# lsattr -El inet0 -a route | grep 192.168.0.1
route net,-netmask,255.255.255.0,-static,10.0.0.0,192.168.0.1 Route True
At this point, you can reboot the system, and you'll notice that the route is still there, by repeating the netstat -nr and lsattr -El inet0 commands.

To remove this permanent route from the AIX system, simply change the chdev command above from "route" to "delroute":
# chdev -l inet0 -a delroute=net,-netmask,255.255.255.0,-static,10.0.0.0,
192.168.0.1
inet0 changed
Finally, again confirm using the netstat -nr and lsattr -El inet0 commands that the route indeed has been removed.

Topics: Networking, Red Hat, System Admin

RHEL: Delete multiple default gateways

A Red Hat Enterprise Linux system should have a single default gateway defined. However, sometimes, it does occur that a system has multiple default gateways. Here's information to detect multiple default gateways and how to get rid of them:

First, check the number of default gateways defined, by running the netstat command and looking for entries that start with 0.0.0.0:

# netstat -nr | grep ^0.0.0.0
0.0.0.0     192.168.0.1     0.0.0.0    UG        0 0        0 em1
0.0.0.0     192.168.1.1     0.0.0.0    UG        0 0        0 em2
In the example above, there are 2 default gateway entries, one to 192.168.0.1, and another one to 192.168.1.1.

Quite often, more than 1 default gateways will be defined on a RHEL system, if there are multiple network interfaces present, and a GATEWAY entry is defined in each of the network interface files in /etc/sysconfig/network-script/ifcfg-*:
# grep GATEWAY /etc/sysconfig/network-scripts/ifcfg-*
ifcfg-em1:GATEWAY=192.168.0.1
ifcfg-em2:GATEWAY=192.168.1.1
On a system with multiple network interfaces, it is best to define the default gateway in file /etc/sysconfig/network instead. This file is global network file. Put the following entries in this file, assuming your default gateway is 192.168.0.1 and the network interface to be used for the default gateway is em1:
GATEWAY=192.168.0.1
GATEWAYDEV=em1
Next, remove any GATEWAY entries in any of the ifcfg-* files in /etc/sysconfig/network-scripts.

Finally, restart the network service:
# service network restart
This should resolve multiple default gateways, and the output of the netstat command should now only show one single entry with 0.0.0.0.

Note: If the netstat command is not available on the system, you may also determine the number of default gateways, by running:
# ip route show | grep ^default

Number of results found for topic Networking: 30.
Displaying results: 1 - 10.