Topics: Monitoring, Red Hat / Linux

Monitoring a log file through Systemd

The following procedure describes how you can continuously monitor a log file through the use of SystemD on Red Hat Enterprise Linux (or similar operating systems).

Let's say you want to receive an email when a certain string occurs in a log file. For example, if the string "error" occurs in file /var/log/messages.

First, create a script that does a tail of /var/log/messages and searches for that string:

# cat /usr/local/bin/monitor.bash

tail -fn0 /var/log/messages | while read line ; do
   echo "${line}" | grep -i "error" > /dev/null
   if [ $? = 0 ] ; then
      echo "${line}" | mailx -s "error in messages file"
You can run that script, and be done with it. But if that script somehow gets cancelled or killed, for example when the system is rebooted, then the monitoring of the log file stops as well. That's where the use of Systemd comes in.

Create a file in folder /etc/systemd/system, such as "monitor.service", and add the following:
Description=My monitor script

ExecStart=/bin/bash /usr/local/bin/monitor.bash

This file is a description of a service managed by Systemd, and it basically tells SystemD to run the script that we created earlier, and to restart it, in case it fails (that's what "Restart=always" is for).

Next, you'll have to tell Systemd that you made some changes:
# systemctl daemon-reload
Now you can start the newly defined service:
# systemctl start monitor.service
And after starting it, you can query the status:
# systemctl status monitor.service
monitor.service - My monitor script
   Loaded: loaded (/etc/systemd/system/monitor.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-10-03 12:12:28 CDT; 2s ago
 Main PID: 1832 (bash)
   CGroup: /system.slice/monitor.service
           ??1832 /bin/bash /usr/local/bin/monitor.bash
           ??1833 tail -fn0 /var/log/messages
           ??1834 /bin/bash /usr/local/bin/monitor.bash

Oct 03 12:12:28 enemigo systemd[1]: Started My monitor script.
Oct 03 12:12:28 enemigo systemd[1]: Starting My monitor script...
As you can see in the output above, both the monitor.bash script and the tail command are running. To test if the service is actually restarted, you can try killing the tail process or the monitor.bash script, and then check for the status again. You'll see it is restarted.

You may also want to test that the new monitor script indeed is working. Considering that /var/log/messages is a file written to through rsyslog, you can log an entry with the string "error" to that file as follows:
# logger error
Next, you should receive an email saying that an "error" occurrence was found in the messages file.

Finally, you'll want to make sure this new monitor service is restarted also when the system boots:
# systemctl enable monitor.service

If you found this useful, here's more on the same topic(s) in our blog:

UNIX Health Check delivers software to scan Linux and AIX systems for potential issues. Run our software on your system, and receive a report in just a few minutes. UNIX Health Check is an automated check list. It will report on perfomance, capacity, stability and security issues. It will alert on configurations that can be improved per best practices, or items that should be improved per audit guidelines. A report will be generated in the format you wish, and the report includes the issues discovered and information on how to solve the issues as well.

Interested in learning more?