Improving File Integrity Monitoring with OSSEC

FIM or “File Integrity Monitoring” can be defined as the process of validating the integrity of operating system and applications files with a verification method using a hashing algorythm like MD5 or SHA1 and then comparing the current file state with a baseline. A hash will allow the detection of files content modification but other information can be checked too: owner, permissions, modification time. Implemeting file integrity monitoring is a very good way to detect compromized servers. Not only operating system files can be monitored (/etc on UNIX, registry on Windows, share libraries, etc) but also applications (monitoring your index.php or index.html can reveal a defaced website).

During its implementation, a file integrity monitoring project may face two common issues:

• The baseline used to be compared with the current file status must of course be trusted. To achieve this, it must be stored on a safe place where attacker cannot detect it and cannot alter it!
• The process must be fine tuned to react only on important changes otherwise they are two risks: The real suspicious changes will be hidden in the massive flow of false-positives. People in charge of the control could miss interesting changes.

There are plenty of tools which implement FIM, commercial as well as free. My choice went to OSSEC for a while. My regular followers know that I already posted lot of articles about it. I also contributed to the project with a patch to add Geolocatization to alerts. This time, I wrote another patch to improve the file integraty monitoring feature of OSSEC.

FIM has been part of the OSSEC features for a while and is handled by the syscheckd daemon running on all agents. How does OSSEC address the common issues reported above? To keep the baseline integrity, the databases of files (or registry for Windows agents) are stored on the manager itself. This manager is normally a well-protected server where all the OSSEC intelligence is stored. About false-positives, OSSEC implement several ways to prevent them. Some files can be ignored with an <ignore> XML tag in ossec.conf:

<syscheck>
    <ignore>/etc/mnttab</ignore>
</syscheck>

This is easy to exclude files but it’s a pain to manage! Some files can be excluded using specific OSSEC rules:

<rule id="100000" level="0" >
    <if_group>syscheck</if_group>
    <description>Ignored file changes</description>
    <match>/etc/mnttb|/etc/hosts|/etc/resolv.conf</match>
    <hostname>srv1</hostname>
</rule>

This rule will disable notification if any change is detected on srv1 in /etc/mnttab, /etc/hosts or /etc/resolv.conf. Note that another control exists: By default when a file has changed three times, new changes will be automatically ignored. Handy but… it could be improved!

When I’m deploying security tools and control, my goal is to reduce the “noise” as much as possible. A side effect of file integrity monitoring is the number of false positive alerts generated when patching your systems. Keeping the latest patch level is important but hundreds of files can be replaced only by one new package! That’s why I wrote the following patch for OSSEC (more precisely for the analysisd daemon which is responsible of the decoding and alerting of events generated by agents).

I added a SQLite3 DB which contains a list of MD5 hashes to ignore when reported by agents. When a file change is reported, its NEW MD5 hash is looked up in the DB. If found, the change is ignored. Why an external SQL database to store the hashes? To be easily populated by external tools as seen in the following schema:

To active this feature, apply the patch, create a SQLite3 database:

CREATE TABLE files (
    md5sum VARCHAR(32),
    file VARCHAR(256),
    time DATETIME
);
CREATE UNIQUE INDEX files_idx ON files(md5sum);

Then, just define the MD5 database in the main ossec.conf file on your OSSEC server:

<global>
    <md5db>/etc/md5.db</md5db>
</global>

This database must contains all the MD5 hashes that you want to ignore. On Ubuntu, it’s easy to find all hashes of installed files in /var/lib/dpkg/info/*.md5sums. I wrote a simple Python script to read those files and populate the SQL database.

#!/usr/bin/python

import fnmatch
import os
import sqlite3
import signal
import sys
def signal_handler(signal, frame):
    print "Interrupted!"
    if (conn):
        conn.commit()
        conn.close()
    sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
conn = sqlite3.connect('/opt/ossec/etc/md5db.db')
for file in os.listdir('/var/lib/dpkg/info'):
    if fnmatch.fnmatch(file, '*.md5sums'):
        c = conn.cursor()
        f = open('/var/lib/dpkg/info/' + file, 'r')
        l = f.readline()
        while l:
            array = l.split()
        try:
            c.execute('INSERT INTO files VALUES("' + array[0] + '","' + \
                      array[1] + '",date("now"))')
        except sqlite3.Error, e:
        print "%s: %s" % (array[0], e.args[0])
        l = f.readline()
        conn.commit()
        f.close()
conn.close()

After every new patch installation on my Ubuntu, the database is updated with new MD5’s. As the FIM process is executed every 6 hours (default setting) by OSSEC, you have time to update the database and reduce the false positives alerts.

The patch is available here.