MultiTail, a Powerful Console Log Viewer

On UNIX systems, log files are everywhere. Application are often very verbose (which is good!) but keeping an eye on logs can quickly become a nightmare!

I’m a big fan of the tail command, specially the “-f” flag which does not stop the command once the end of file is reached but, instead, wait for additional data to be appended to the input. Even more powerful, the GNU version of tail allows to specify more than one file with the “-f” flag to keep track of changes in multiple logs at a time. Note that the BSD version tail does not support this feature (check a previous post for a patch). While googling for alternative solutions, I found MultiTail. This is a quite old tool, but so powerful.

The basic usage of MultiTail is to replace the “tail -f” command: It displays the content of log files in real time:

$ multitail /var/www/logs/host1.access /var/www/logs/host2.access

Your console will be split in two virtual windows and display the file contents into them (like the screen utility). But the features list is impressive. Here are the most interesting (from my point of view):

• Logfiles can be filtered with the help of one or more regular expressions
• An external tool can be executed when a regular expression matches
• Parts of the line can be highlighted
• Color schemes can be defined in a configuration file. A lot of standard log file formats are provided: Apache, Squid, Sendmail, Snort, …

Have a look at the multiple examples provided to have a good idea of the features. But how can MultiTail help you in your security business? Here are some examples:

First, a notification can be generated when something special is detected (via a regular expression) in a log file. In the following example, MultiTail will follow the authentication log file and generate an e-mail alert when a failed SSH login occurred:

# multitail -ex "sshd.*Failed password for invalid user" \
                       "echo SSH breaking attempt detected | \
                       mail -s Alert xavier@rootshell.be" \
                 -i /var/log/authlog"

The following example will tag blocked packets by a pf (packet filter) firewall on OpenBSD:

# multitail -cf 5 " " -l "tcpdump -n -e -ttt -i pflog0"

The fifth field (“accept” or “block”) and will be displayed in different colors when passing from one value to the other. Combined with the screen utility, a single SSH session to a server is enough to build powerful dashboards. Check out the following session:

A screen session (five virtual consoles on the bottom line – also with the server load average) are started and in one of them a MultiTail display three log: a firewall log (pflog), an Apache access log and the standard /var/log/messages. MultiTail is really a must have for all the command line addicts.