Topics: Red Hat / Linux, Security

Monitor SSH logins

To monitor SSH logins on a Linux server, run the following command:

# journalctl -S @0 -u sshd
If you wish to continuously monitor the traffic, add the -f option. This will "tail" the output:
# journalctl -S @0 -u sshd -f

Topics: Networking, Security

Testing for open ports

Something every system administrator will have to do from time to time, is to test if a certain port is open and reachable over the network.

There are a few ways to go about doing that.

The most common way is to use the nc or ncat or netcat utility. It can easily be used to test a port on a system. For example, to test if port 22 (for ssh) is open on a remote system, run:

# nc -zv systemname 22
Replace "systemname" with the hostname or IP address of the system to be tested. You may also test it locally on a system, by using "localhost".

This will show something like this (if the port is open):
# nc -zv localhost 22
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to ::1:22.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
The nc utility is part of the nmap-ncat RPM package, and can be installed as follows:
# yum -y install nmap-ncat
If installing this utility is not an option, you can also directly test the port, by running this:
# bash -c "</dev/tcp/cielo/22"
If that works, no output is show, and the return-code will be 0 (zero). If it doesn't work, because the port is closed, you'll see an error message, and the return-code will be 1 (one). For example:
# bash -c "</dev/tcp/localhost/23"
bash: connect: Connection refused
bash: /dev/tcp/localhost/23: Connection refused
# echo $?
1
Finally, one other method of testing for open ports, is by using telnet, if it is installed on the system. This can be done by specifying the hostname and the port to connect to:
$ telnet localhost 22
Trying ::1...
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_7.4

Topics: Security, System Admin

Entropy

If you run this command on Linux:

# cat /proc/sys/kernel/random/entropy_avail
it returns a number that indicates how much "entropy" is available to the kernel. But what exactly is "entropy", and what unit is this entropy measured in? What is it used for? You may have heard that a low number of entropy is not good. How low is "low" and what "bad" things will happen if it is? What's a good range for it to be at? How is it determined?

Entropy is similar to "randomness". A Linux system gathers "real" random numbers by keeping an eye on different events: network activity, hard drive rotation speeds, hardware random number generator (if available), key-clicks, and so on. If feeds those to the kernel entropy pool, which is used by /dev/random. Applications which use crypto functions, use /dev/random as their entropy source, or in other words, the randomness source.

If /dev/random runs out of available entropy, it's unable to serve out more randomness and the application waiting for the randomness may stall until more random bits are available. On Red Hat Enterprise Linux systems, you can see that RPM package rng-tools is installed, and that a rngd - random nubmer generator deamon - is active. This deamon feeds semi-random numbers from /dev/urandom to /dev/random in case /dev/random runs out of "real" entropy. On Ubuntu based systems, there is no rngd. If more entropy is needed you can install haveged, which can achieve the same. Note that haveged is not available for Red Hat Enterprise Linux based systems, because these systems already have rngd.

Some applications (such as applications using encryption) need random numbers. You can generate random numbers using an algorithm - but although these seem random in one sense they are totally predictable in another. For instance if I give you the digits 58209749445923078164062862089986280348253421170679, they look pretty random. But if you realize they are actually digits of PI, then you may know the next one is going to be 8.

For some applications this is okay, but for other applications (especially security related ones) people want genuine unpredictable randomness - which can't be generated by an algorithm (i.e. program), since that is by definition predictable. This is a problem in that your computer essentially is a program, so how can it possibly get genuine random numbers? The answer is by measuring genuinely random events from the outside world - for example gaps between your key strokes and using these to inject genuine randomness into the otherwise predictable random number generator. The "entropy pool" could be thought of as the store of this randomness which gets built up by the keystrokes (or whatever is being used) and drained by the generation of random numbers.

The value stored in /proc/sys/kernel/random/entropy_avail is the measure of bits currently available to be read from /dev/random. It takes time for the computer to read entropy from its environment. If you have 4096 bits of entropy available and you "cat /dev/random" you can expect to be able to read 512 bytes of entropy (4096 bits) before the file blocks while it waits for more entropy. For example, if you "cat /dev/random" your entropy will shrink to zero. At first you'll get 512 bytes of random garbage, but it will stop and little by little you'll see more random data trickle trough.

This is not how people should operate /dev/random though. Normally developers will read a small amount of data, like 128 bits, and use that to seed some kind of PRNG algorithm. It's polite to not read any more entropy from /dev/random than you need to, since it takes so long to build up entropy, and it is considered valuable. Thus if you drain it by carelessly catting the file like above, you'll cause other applications that need to read from /dev/random to block.

A good number of available entropy is usually between 2500 and 4096 bits. Entropy is considered to be low when it is below 1000.

Topics: System Admin

Sending emails with attachments

If you wish to send a file from a Linux or UNIX system, you can cat the file and pipe it into an email. For example, if you wish to send the contents of the /etc/hosts file in an email:

# cat /etc/hosts | mailx -s "subject goes here" your@email-address.com
This may not always be what you want, though. Sometimes, the file may be too large, or you may rather attach the file to an email.

To attach a file to an email, run it like this using -a option:
# echo "your message goes here" | \
mailx -s "your subject goes here" -a /etc/hosts your@email-address.com
Most email applications, like Microsoft Outlook, will put you text ("your message goes here") into the body of the email, and will attach /etc/hosts as a hosts.txt file for easy reading.

To attach larger files to an email (or for mail applications that handle zip files better as attachments), zip the file first before attaching it to an email message:
# zip /tmp/myzipfile.zip /etc/hosts
# echo "your message goes here" | \
mailx -s "your subject goes here" -a /tmp/myzipfile.zip your@email-address.com

Topics: Scripting

ShellCheck

ShellCheck is a tool that can be used to check the integrity of shell scripts, which can be very useful while developing shell scripts, as it may indicate where issues are within shell scripts. It works with different shells, such as Bash, and the Korn shell.

Visit www.shellcheck.net to get information about how to obtain the tool.

For Red Hat Enterprise Linux based systems (and its derivatives), it can simply be installed as follows:

# yum -y install epel-release
# yum install ShellCheck
Or on Fedora:
# dnf install ShellCheck
ShellCheck helps to identify potential issues with check scripts, and thus can be an important tool for shell script developers. Even if a shell script works fine, the ShellCheck tool can be used to identify potential improvements to shell scripts. Considering that UNIX Health Check software is completely written in shell scripts, it is a tool used by us quite a lot to aid in ensuring that our code is valid and correct.

To check a specific script, e.g. to check script test.sh, run:
# shellcheck test.sh
For more information about the use of ShellCheck, run:
# man shellcheck

Topics: Hardware, Red Hat / Linux, System Admin

Determining memory DIMMs for Linux systems

If you wish to determine how much memory is installed in a Linux system, or perhaps the maximum amount of memory configurable on a system and the exact number and size of the memory DIMMs installed in the system, then you should use the dmidecode command.

The dmidecode command has a type option ( -t ), that can be used to indicate the type of device you wish to see detailed information for, like bios, system, baseboard, chassis, processor, cache, connector, slot and ... memory.

To retrieve the memory information, run the following command:

# dmidecode -t memory
You should get an output similar like the one below, but it can obviously differ per Linux system, depending on the hardware (model) and the installed memory.
# dmidecode -t memory
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.

Handle 0x0047, DMI type 16, 23 bytes
Physical Memory Array
        Location: System Board Or Motherboard
        Use: System Memory
        Error Correction Type: None
        Maximum Capacity: 16 GB
        Error Information Handle: Not Provided
        Number Of Devices: 2

Handle 0x0048, DMI type 17, 34 bytes
Memory Device
        Array Handle: 0x0047
        Error Information Handle: Not Provided
        Total Width: 64 bits
        Data Width: 64 bits
        Size: 4096 MB
        Form Factor: SODIMM
        Set: None
        Locator: DIMM A
        Bank Locator: Not Specified
        Type: DDR3
        Type Detail: Synchronous
        Speed: 1600 MT/s
        Manufacturer: Hynix/Hyundai
        Serial Number: 3248A01B
        Asset Tag: 9876543210
        Part Number: HMT351S6CFR8C-PB
        Rank: 2
        Configured Memory Speed: 1600 MT/s

Handle 0x004A, DMI type 17, 34 bytes
Memory Device
        Array Handle: 0x0047
        Error Information Handle: Not Provided
        Total Width: 64 bits
        Data Width: 64 bits
        Size: 4096 MB
        Form Factor: SODIMM
        Set: None
        Locator: DIMM B
        Bank Locator: Not Specified
        Type: DDR3
        Type Detail: Synchronous
        Speed: 1600 MT/s
        Manufacturer: Kingston
        Serial Number: BA33020F
        Asset Tag: 9876543210
        Part Number: KFYHV1-HYC
        Rank: 2
        Configured Memory Speed: 1600 MT/s
In the output above you can see that this particular system has a "Maximum Capacity" of 16 GB, and up to two "Number of Devices" can be installed.

Below it, you see the currently installed Memory Devices, the first one with a size of 4096 MB in slot DIMM A, with the exact Manufacturer and the Part Number listed. In slot DIMM B, you can also see another DIMM of 4096 MB installed, which is of a different vendor.

This information tells you, that the system shown above can be configured up to 16 GB of memory, but currently has two 4 GB memory DIMMs, thus 8 GB installed. In this case, upgrading the system to 16 GB of memory would mean replacing the two 4 GB memory DIMMs with two 8 GB memory DIMMs.

Topics: Security, System Admin

Using a proxy for wget

If you need to use a proxy for wget, this can be configured as follows:

Either create a file $HOME/.wgetrc (for a specific user) or /etc/wgetrc (to set the proxy configuration for the entire system), and add in the following lines, while replacing the entries below with your HTTP proxy URL and port information:

# You can set the default proxies for Wget to use for http and https.
# They will override the value in the environment.
https_proxy = http://proxy.yoyodyne.com:3128
http_proxy = http://proxy.yoyodyne.com:3128

# If you do not want to use proxy at all, set this to off.
use_proxy = on
Alternatively, you may also use the -e option for wget:
# wget ... -e https_proxy=http://proxy.yoyodyne.com:3128 ...

Topics: Red Hat / Linux, Security

Renew self-signed SSL Certificate

Should the root user receive emails from certwatch about expiring self-signed certificates, like these:

 ################# SSL Certificate Warning ################

  Certificate for hostname 'yourhost', in file (or by nickname):
     /etc/pki/tls/certs/localhost.crt

  The certificate needs to be renewed; this can be done
  using the 'genkey' program.

  Browsers will not be able to correctly connect to this
  web site using SSL until the certificate is renewed.

 ##########################################################
                                  Generated by certwatch(1)

Then, you can run the following command to renew this self-signed certificate for a new year:
# openssl req -new -days 365 -x509 -nodes -out /etc/pki/tls/certs/localhost.crt \ 
-keyout /etc/pki/tls/private/localhost.key

More complete instructions can be found here: http://stevejenkins.com/blog/2010/08/renewing-a-self-signed-ssl-certificate-on-fedoracentos

Topics: Performance, Red Hat / Linux

How to view threads of a process on Linux

Threads are a popular programming abstraction for parallel execution on modern operating systems. When threads are forked inside a program for multiple flows of execution, these threads share certain resources (e.g., memory address space, open files) among themselves to minimize forking overhead and avoid expensive IPC (inter-process communication) channel. These properties make threads an efficient mechanism for concurrent execution.

In Linux, threads (also called Lightweight Processes (LWP)) created within a program will have the same "thread group ID" as the program's PID. Each thread will then have its own thread ID (TID). To the Linux kernel's scheduler, threads are nothing more than standard processes which happen to share certain resources. Classic command-line tools such as ps or top, which display process-level information by default, can be instructed to display thread-level information.

Here are several ways to show threads for a process on Linux:

Using the ps command

The "-T" option for the ps command enables thread views. The following command list all threads created by a process with :

# ps -T -p <pid>
For example to list the threads for the following java process:
# ps -ef | grep 97947
deploy   97947 97942  1 00:51 ?       00:13:51 java
Run the following command:
# ps -T -p 97947
  PID  SPID TTY      TIME CMD
97947 97947 ?    00:00:00 java
97947 97948 ?    00:00:00 java
97947 97949 ?    00:00:00 java
The "SID" column represents thread IDs, and "CMD" column shows thread names.

Using the top command

The top command can show a real-time view of individual threads. To enable thread views in the top output, invoke top with "-H" option. This will list all Linux threads. You can also toggle on or off thread view mode while top is running, by pressing 'H' key.
top - 14:43:25 up 6 days,  5:40,  2 users,  load average: 0.87, 0.33, 0.22
Threads: 684 total,   1 running, 683 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.3 us,  4.0 sy,  0.0 ni, 89.6 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7910136 total,   384812 free,  1603096 used,  5922228 buff/cache
KiB Swap:  8388604 total,  8239100 free,   149504 used.  5514264 avail Mem
Note how in the example above the number of threads on the system is listed.

To restrict the top output to a particular process and check all threads running inside the process:
# top -H -p <pid>
Using htop

A more user-friendly way to view threads per process is via htop, an ncurses-based interactive process viewer. This program allows you to monitor individual threads in tree views.

To enable thread views in htop, launch htop, and press F2 to enter htop setup menu. Choose "Display option" under "Setup" column, and toggle on "Tree view" and "Show custom thread names" options. Presss F10 to exit the setup.

Topics: Red Hat / Linux

Linux strace command

The strace command in Linux lets you trace system calls and signals. Or in other words: it allows you to see what a program or process is doing.

Following is its syntax:

strace [OPTIONS] command
In the simplest case, strace runs the specified command until it exits. It intercepts and records the system calls which are called by a process and the signals which are received by a process. The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option. For example:
# strace ls
If a process is already running, you can trace it by simply passing its PID as follows; this will fill your screen with continued output that shows system calls being made by the process. To end it, press [Ctrl + C].
# strace -p 3569

Number of results found: 461.
Displaying results: 1 - 10.