Playing with IP Reputation with Dshield & OSSEC

[This blogpost has also been published as a guest diary on isc.sans.org]

When investigating incidents or searching for malicious activity in your logs, IP reputation is a nice way to increase the reliability of generated alerts. It can help to prioritize incidents. Let’s take an example with a WordPress blog. It will, sooner or later, be targeted by a brute-force attack on the default /wp-admin page. In this case, IP reputation can be helpful: An attack performed from an IP address reported as actively scanning the Internet will not (or less) attract my attention. On the contrary, if the same kind of attack is coming from an unkown IP address, this could be more suspicious…

By using a reputation system, our monitoring tool can tag an IP address with a label like “reported as malicious” based on a repository. The real value of this repository depends directly of the value of collected information. I’m a big fan of dshield.org, a free service provided by the SANS Internet Storm Center. Such service is working thanks to the data submitted by many people across the Internet. For years, I’m also pushing my firewall logs to dshield.org from my OSSEC server. I wrote a tool to achieve this: ossec2dshield. By contributing to the system, it’s now time to get some benefits from my participation: I’m re-using the database to automatically check the reputation of the IP addresses attacking me. We come full circle!

To achieve this, let’s use the API provided on isc.sans.org and the OSSEC feature called “Active-Response” which allows to trigger a script upon a set of conditions. In the example below, we call the reputation script with our attacker address for any alert with a level >= 6.

<command>
    <name>isc-ipreputation</name>
    <executable>isc-ipreputation.py</executable>
    <expect>srcip</expect>
    <timeout_allowed>no</timeout_allowed>
</command>

<active-response>
    <!-- Collect IP reputation data from ISC API
      -->
    <command>isc-ipreputation</command>
    <location>server</location>
    <level>6</level>
</active-response>

(Check the Active-Response documentation for details)

The ISC API can be used to query information about an IP address. The returned results are:

$ wget -O - -q https://isc.sans.edu/api/ip/195.154.243.219?json
{"ip":{"abusecontact":"unknown","number":"195.154.243.219","country":" FR ","as":"12876 ","asname":" AS12876 ONLINE S.A.S.,FR","network":" 195.154.0.0\/16 ","comment":null}}

The most interesting fields are:

• count – the number of times the IP address has been reported as an attacker
• attacks – the number of targeted IP addresses
• mindate  – the first report
• maxdata – the last report

The script “isc-ipreputation.py” can be used from the command line or from an OSSEC Active-Response configuration block. To reduce the requests against the API, a SQLite database is created and populated with a local copy of the data. Existing IP addresses will be checked again after a specified TTL (time-to-live), by default 5 days. Data are also dumped in a flat file or Syslog for further processing by another tool. Here is an example of entry:

$ tail -f /var/log/ipreputation.log
[2015-05-27 23:30:07,769] DEBUG No data found, fetching from ISC
[2015-05-27 23:30:07,770] DEBUG Using proxy: 192.168.254.8:3128
[2015-05-27 23:30:07,772] DEBUG Using user-agent: isc-ipreputation/1.0 (blog.rootshell.be)
[2015-05-27 23:30:09,760] DEBUG No data found, fetching from ISC
[2015-05-27 23:30:09,761] DEBUG Using proxy: 192.168.254.8:3128
[2015-05-27 23:30:09,762] DEBUG Using user-agent: isc-ipreputation/1.0 (blog.rootshell.be)
[2015-05-27 23:30:10,138] DEBUG Saving 178.119.0.173
[2015-05-27 23:30:10,145] INFO IP=178.119.0.173, AS=6848("TELENET-AS Telenet N.V.,BE"), Network=178.116.0.0/14, Country=BE, Count=148, AttackedIP=97, Trend=0, FirstSeen=2015-04-21, LastSeen=2015-05-27, Updated=2015-05-27 18:37:15

In this example, you can see that this IP address started to attack on the 21st of April. It was reported 148 times while attacking 97 different IP addresses (This IP is certainly part of a botnet).

The script can be configuration with a YAML configuration file (default to /etc/isc-ipreputation.conf) which is very easy to understand:

logging:
  debug: yes
database:
  path: '/data/ossec/logs/isc-ipreputation.db'
network:
  exclude-ip: '192\.168\..*|172\.16\..*|10\..*|fe80:.*'
  ttl-days: 5
http:
  proxy: '192.168.254.8:3128'
  user-agent: 'isc-ipreputation/1.0 (blog.rootshell.be)'

Finally, the SQLite database can use used to get interesting statistics. Example, to get the top-10 of suspicious IP addresses that attacked me (and their associated country):

$ sqlite3 isc-ipreputation.db 
SQLite version 3.8.2 2013-12-06 14:53:30
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select ip, count, attacks,country from ip order by count desc limit 10;
61.240.144.66|4507455|32533|CN
218.77.79.43|2947146|63295|CN
61.240.144.65|2408418|24185|CN
61.240.144.64|1947038|22054|CN
61.240.144.67|1759210|25421|CN
184.105.139.67|1678608|63055|US
61.160.224.130|1553361|62140|CN
61.183.128.6|1385025|13829|CN
61.160.224.129|1312580|15202|CN
61.160.224.128|1209176|61006|CN
sqlite>

It is also very easy to generate dynamic lists of IP addresses (or CDB as called by OSSEC). The following command will generate a CDB list with my top-10 of malicious IP addresses:

$ sqlite3 isc-ipreputation.db \
"select ip from ip order by count desc limit 10;"| \
while read IP;
do
echo "$IP:Suspicious";
done >/data/ossec/lists/bad-ips
$ cat /data/ossec/lists/bad-ips
61.240.144.66:Suspicious
218.77.79.43:Suspicious
61.240.144.65:Suspicious
61.240.144.64:Suspicious
61.240.144.67:Suspicious
184.105.139.67:Suspicious
61.160.224.130:Suspicious
61.183.128.6:Suspicious
61.160.224.129:Suspicious
61.160.224.128:Suspicious
$ ossec-makelists
* File lists/bad-ips.cdb needs to be updated

Based on this list, you can add more granularity to your alerts by correlating the attacks with the CDB list. Note that dshield.org proposes a recommended block list ready to be used. A few months ago, Richard Porter explained how to integrate one of them in a Palo Alto Networks firewall. This is a great resource but I think that both are complementary.

The script is available on my github repository.