Getting Useful Info From the Log Hell with Awk

Getting useful info from log file should be piece of cake …if the file is properly formatted! Usually, one event is written on a single line with useful info delimited by a separator or extractable using regular expressions. But it’s not always the case, welcome to the log hell…

Sometimes, the log file contains the output of a script or a dump of another file and is split into multiple lines (think about a Java error –  known to be extremely verbose). If the application does a good job, the dump can be identified by “tags” at the beginning and end of the interesting data. Here is a quick tip to extract them from the UNIX command like. Very useful to parse them or just send the output via an email.

I’m a big fan of Security Onion. Amongst multiple tools to analyze your network traffic, it helps me to gather intelligence about new IDS signatures. One tool used by this distribution is Pulled Pork, which keeps a Snort / Suricata IDS rule base up-to-date. Executed daily, it generates a log file with the new and removed rules. Every day, the following data are appended to the file:

-=Begin Changes Logged for Mon Aug 29 07:26:33 2016 GMT=-
New Rules
    BROWSER-CHROME Google Chrome FileSystemObject clsid access (1:21446)
    ...

Deleted Rules
    BROWSER-CHROME Google Chrome FileSystemObject clsid access (1:0)
    ...

Set Policy: Disabled

Rule Totals
    New:-------46
    Deleted:---12
    Enabled:---27385
    Dropped:---0
    Disabled:--23867
    Total:-----51252

No IP Blacklist Changes

-=End Changes Logged for Mon Aug 29 07:26:33 2016 GMT=-

I like to get a notification with the daily added / removed IDS rules on my Security Onion box (I’m using the Emerging Threats feed). The power of the command line can help us to extract useful information from the log above. The goal is to search for the “Begin changes” line, the “End Changes” line and extract what’s in the middle. How? Thanks to the wonderful ‘awk‘ tool:

awk "
  /Begin Changes Logged for `date +'%a %b %d'`/ {echo=1}
  echo
  /End Changes Logged for `date +'%a %b %d'`/ {echo=0}
  " /var/log/nsm/sid_changes.log \
| mail -s "[SecurityOnion] Suricata Rules Update" foo@bar.com

How does it work? awk searches for the starting header line (with the current date properly formatted). Once found, it declares the variable ‘echo’. While ‘echo’ is declared, the next line read from the file is displayed. A second search is performed (the ending header). If found, ‘echo’ is set to ‘0’ and the next lines won’t be printed.