Are You Playing “Cold Case” with Your Logs?
Two days ago, I attended an event about “big data” (yeah, another buzz word) and how to use it for security purposes. One of the presented talks was very interesting and almost changed my mind about our best friends (or nightmare)… logs!
When I’m talking about log management with customers, I always insist on the fact to work with “business cases“, for two major reasons. The first one is purely technical: to avoid a constant flood of events and to be able to process correctly the important ones. The second one is more focused on the business: to get a quick return on investment! Log management is often expensive and management will be happy if they get “something to eat” as soon as possible. Business cases can be related to internal requirements or to compliance requirements. PCI-DSS is a good example. Here are some reports that must be produced:
- IPS reports
- User accesses on devices
- Account changes
- Configuration changes
There is a big side effect with compliance requirements: organizations tend to do the strict minimum to be “compliant“. Still today, some companies see this as a stamp on a paper. Fail! Let’s put the “business” aside and let’s view the log management as a tool to perform incident handling or forensics researches. In this case, it will be more efficient if we would investigate across a broader amount of data. Sometimes investigations occur a long time after the incident. Incident handlers are playing like the Cold Case characters, trying to find who did what, when and where! And evidences could be located at so many places! Let’s take an IP address. We can find them in firewall, IDS or proxy logs but also:
- In SMTP headers
- In Excel sheets (low cost IPAM solution but commonly used
- In configuration files
To increase the chances to find relevant information, the log management solution must collect and index as much data as possible. It’s the opposite of the compliance requirements. Do I change sides? Not completely but I think that both points of view must be taken into account with their pros & cons.
Howto: Distributed Splunk Architecture
Implementing a good log management solution is not an easy task! If your organisation decides (should I add “finally“?) to deploy “tools” to manage your huge amount of logs, it’s a very good step forward but it must be properly addressed. Devices and applications have plenty of ways to generate logs. They could send SNMP traps, Syslog messages, write in a flat file, write in a SQL database or even send smoke signals (thanks to our best friends the developers). It’s definitively not an out-of-the-box solution that must be deployed. Please, do NOT trust $VENDORS who argue that their killing-top-notch-solution will be installed in a few days and collect everything for you! Before trying to extract the gold of your logs, you must correctly collect events. This mean first of all: do not loose some of them. It’s a good opportunity to remind the Murphy’s laws here: The lost event will always be the one which contained the most critical piece of information! In most cases, a log management solution will be installed on top of an existing architecture. This involves several constraints:
- From a security point of view, firewalls will for sure block flows used by the tools. Their policy must be adapted. The same applies to the applications or devices.
- From a performance point of view, the tools can’t have a negative impact on the “business” traffic.
- From a compliance point of view, the events must be properly handled in respect of the confidentiality, integrity and availability (you know the well-know CIA principle).
- From a human point of view (maybe the most important), you will have to fight with other teams and ask them to change the way they work. Be social!
To achieve those requirements, or at least trying to reach them, your tools must be deployed in a distributed architecture. By “distributed“, I mean using multiple software componants desployed in multiple places in your infrastructure. The primary reason for this is to collect the events as close as possible to their original source. If you do this, you will be able to respect the CIA principle and:
- To control the resources usage to process them and centralise them
- To get rid of proprietary or open multiple protocols
- To control the good processing of them from A to Z.
For those who are regular readers of my blog, you know that I’m a big fan of OSSEC. This solution implements a distributed architecture with agents installed on multiple collection points to grab and centralise the logs:
OSSEC is great but lack of a good web interface to search for events and generate reports. Lot of people interconnect their OSSEC server with a Splunk instance. There is a very good integration of both products using a dedicated Splunk app. Usually, Splunk is deployed on the OSSEC server itself. The classic way to let Splunk collect OSSEC events is to configure a new Syslog destination for alerts like this (in your ossec.conf file):
<syslog_output> <server>10.10.10.10</server> <port>10001</port> </syslog_output>
This configuration blog will send alerts (only!) to Splunk via Syslog messages sent to 10.10.10.10:10001 (where Splunk will listen for them). Note that the latest OSSEC version (2.7) can write native Splunk events over UDP. Personally, I don’t like this way of forwarding events because UDP remains unreliable and only OSSEC alerts are forwarded. I prefer to process the OSSEC files using the file monitor feature of Splunk:
[monitor:///data/ossec/logs] whitelist=\.log$
But what if you have multiple OSSEC server across multiple locations? Splunk has also a solution for this called the “Universal Forwarder“. Basically, this is a light Splunk instance which is installed without any console. This goal is just to collect events in the native format and forward them to a central Splunk instance (the “Indexer“):
If you have experience with ArcSight products, you can compare the Splunk Indexer with the ArcSight Logger and the Universal Forwarder with the SmartConnector. The configuration is pretty straight forward. Let’s assume that you already have a Splunk server running. In your $SPLUNK_HOME/etc/system/local/inputs.conf, create a new input:
[splunktcp-ssl:10002] disabled = false sourcetype = tcp-10002 queue = indexQueue [SSL] password = xxxxxxxx rootCA = $SPLUNK_HOME/etc/auth/cacert.pem serverCert = $SPLUNK_HOME/etc/auth/server.pem
Restart Splunk and it will now bind to port 10002 and wait for incoming traffic. Note that you can use the provided certificate or use your own. It’s of course recommended to encrypt the traffic over SSL! Now install an Universal Forwarder. Like the regular Splunk, packages are available for most modern OS. Let’s play with Ubuntu:
# dpkg -i splunkforwarder-5.0.1-143156-linux-2.6-intel.deb
Configuration can be achieved via the command line but it’s very easy to do it directly by editing the *.conf files. Configure your Indexer in the $SPLUNK_HOME/etc/system/local/outputs.conf:
[tcpout] defaultGroup = splunkssl [tcpout:splunkssl] server = splunk.index.tld:10003 sslVerifyServerCert = false sslCertPath = $SPLUNK_HOME/etc/auth/server.pem sslPassword = xxxxxxxx sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
The Universal Forwarder inputs.conf file is a normal one. Just define all your sources there and start the process. It will start forwarding all the collected events to the forwarder. This is a quick example which demonstrate how to improve your log collection process. The Universal Forwarder will take care of the collected events and send them safely to your central Splunk instance (compressed, encrypted) and will queue them in case of outage.
A final note, don’t ask me to compare Splunk, OSSEC or ArcSight. I’m not promoting a tool. I just gave you an example of how to deploy a tool, whatever your choice is
Attackers Geolocation in OSSEC
If you follow my blog on a regularly basis, you probably already know that I’m a big fan of OSSEC. I’m using it to monitor all my personal systems (servers, labs, websites, etc). Being a day-to-day user, I have always new ideas to extend the product , by using 3rd party tools or by adding features. One of the missing feature (at least for me), is the lack of information when an alert is generated. Tracking the attackers source IP addresses is very nice. Example: OSSEC can trigger active-response scripts to blacklist them during a short period but how can we benefit of more “visibility” with those addresses. When you think how to give more visibility to IP addresses, you immediately think: Geolocation! I already posted an article about the power of geolocation (link) to map alerts onto a Google map (example: a brute-force attack). This is very interesting but required manual actions. As IP addresses are already known by OSSEC (saved in the variable “srcip“), why not let OSSEC do the job for us in real time?
The problem solved by the geolocation is the following: How to convert IP addresses in coordinates (longitude/latitude) first and then map them to a country and/or city? MaxMind has the solution. This company maintains a database of all the assigned IP addresses (Over 99.5% on a country level and 78% on a city level for the US within a 40 kilometer radius – source: Maxmind) with their mapping to geographic locations. Note that they provide two databases: IPv4 and IPv6. They propose API‘s for several languages like C. Good news OSSEC is written in C! So, I wrote a patch which perform a geolocation of the ‘srcip’ attackers to display the location of the IP address. Let’s go…
Step 1: Install the MaxMind GeoIP API
This is easy as any other open source tool/library. Download the source code and install it. It should not a problem on classic Linux flavors.
# wget http://www.maxmind.com/download/geoip/api/c/GeoIP-1.4.8.tar.gz # tar xzvf GeoIP-1.4.8.tar.gz # cd GeoIP-1.4.8 # ./configure # make # make install
By default, everything will be installed under /usr/local but you are free to change the multiple paths via the configure script. The API provides C include files and (dynamic/static) libraries, very common.
Step 2: Install or recompile OSSEC with GeoIP localization
If you have already an OSSEC running, you could simply apply my patch to the original source tree. Take care if you already performed personal changes! I created the patch from a standard 2.6 tarball. To enable the GeoIP feature, you have to enable it (like the MySQL support).
# wget http://www.ossec.net/files/ossec-hids-2.6.tar.gz # tar xzvf ossec-hids-2.6.tar.gz # wget http://blog.rootshell.be/wp-content/uploads/2012/06/ossec-geoip.patch # cd ossec-hids-2.6 # patch -p1 < ../ossec-geoip.patch # cd src # make setgeoip # cd .. # ./install.sh
Step 3: Install the MaxMing GeoIP DBs
MaxMind provides different versions of the databases. I’m using the GeoCityLite. This is the free version which provides precision up to the city. This is far enough for me. Databases are not provided with the API, they must be installed manually:
# cd /var/ossec/etc # wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz # gzip -d GeoLiteCity.dat.gz # wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz # gzip -d -c - >GeoLiteCityv6.dat.gz
I suggest you to download both databases (v4 & v6) and don’t forget that databases are regularly updated! It’s interesting to setup a small cron to install new versions to keep a good accuracy in the results.
Step 4: Fix the OSSEC configuration files
Once the patch installed, new parameters are available to set the GeoIP environment. First, in the ossec.conf “global” section, define the path of your databases.
<global> <geoip_db_path>/etc/GeoLiteCity.dat</geoip_db_path> <geoip6_db_path>/etc/GeoLiteCityv6.dat</geoip6_db_path> </global>
In the “alerts” section, activate the GeoIP feature:
<alerts> <use_geoip>yes</use_geoip> </alerts>
Finally, in the internal_options.conf, enable (or disable) the display of GeoIP information in the notification emails:
# Maild GeoIP support (0=disabled, 1=enabled) maild.geoip=1
How does it work? The OSSEC process which performs the GeoIP lookups is “ossec-analysisd“. When an alert must be logged and if a “srcip” has been decoded, the IP address is passed to a new function GeoIPLookup() which calls the MaxMind API and returns a string with the geolocation data. The data is added to the alert text. The second component which has been patched is “ossec-maild” which parses the alerts and send emails also with the GeoIP data (if enabled).
During the configuration, don’t forget that “ossec-analysisd” runs chrooted (in the main OSSEC directory). Don’t forget to adapt the path to the GeoIP databases! (they must be defined relative to the chroot environment).
Here follow some examples of alerts with GeoIP data enabled:
** Alert 1338899194.500996: - apache,invalid_request, 2012 Jun 05 14:26:34 (xxx) x.x.x.x->/var/log/apache/error_log Rule: 30115 (level 5) -> 'Invalid URI (bad client request).' Src IP: 94.45.169.133 Src Location: RU,Moscow City,Moscow [Tue Jun 05 14:26:34 2012] [error] [client 94.45.169.133] [deleted] ** Alert 1338901319.507426: - syslog,postfix,spam, 2012 Jun 05 15:01:59 (xxx) x.x.x.x->/var/log/syslog Rule: 3303 (level 5) -> 'Sender domain is not found (450: Requested mail action not taken).' Src IP: 188.142.66.168 Src Location: NL,Zuid-Holland,Alphen Jun 5 15:01:43 xxx postfix/smtpd[7397]: NOQUEUE: reject: [deleted] Received From: (xxx) x.x.x.x->/var/log/apache/access_log Rule: 100106 fired (level 15) -> "PHP CGI-bin vulnerability attempt." Src Location: IL,Tel Aviv,Tel Aviv-yafo Portion of the log(s): 77.126.209.178 - - [05/Jun/2012:16:42:07 +0200] [deleted]
Note that GeoIP lookups will be successful only for alerts which have a valid “srcip” field! In all case, the returned location will be “(null)“! What about the impact on performances? I’m using this patch in production for a few days and I did not notice any performance degradation. The GeoIP lookup is performed only once and parsed later by ossec_maild. (I’ve an average of 2000 alerts logged per day).
My OSSEC server is compiled with the support for MySQL. I did not detect any incompatibility between MySQL and my patch. At this time, the geolocation data are not sent to the MySQL alert table. But this could be done easily: Storing the latitude/longitude could be helpful to map attacks in real time using a 3rd party tool.
My patch is available here. Feel free to download it, use it and maybe improve it. Comments and suggestions will be appreciated. Final disclaimer, I won’t be responsible of you break your current OSSEC setup…
Monitor your Monitoring Tools
We (and I’m fully part of it) deploy and use plenty of security monitoring tools daily. As our beloved data is often spread across complex infrastructures or simply across multiple physical locations, we have to collect interesting information and bring them in a central place for further analysis. That’s called “log management“. Based on your collected events, you can generate alerts, build reports. Nice! But… if systems and applications generate [hundreds|thousands|millions] of events, those ones are processed by the same kind of hardware running some piece of software. Hardware may fail (network outage, power outage, disk crash) and softwares have bugs (plenty of).
The value of HTTP 404 Errors
The HTTP protocol has a list of response status codes to help communication between the server and the browser. Everytime a server responds to a browser request, a status code is sent. The most common ones are: “200” which means “Everything is ok, here is some food!” and “404” which means “Not found“. The second error may be caused by the client (example: an error in the URL typed in the browser) or by the developer/administrator who forgot to copy files or also made typo errors in his code. That’s why the amount of 404 errors is directly related to the type of environment. During development and test phases, it’s common to have more errors. On the other side, in a production environment, the amount of 404 errors should be limited and the main source of errors will be the client/browser.
Sometimes, “404” errors are considered useless by webmasters and are simply ignored in their reports. After all, their goal is to know how many visitors browsed to their websites. From a security perspective, those errors could be very helpful to detect unusual traffic targeting a web sites.
I analyzed one year of my blog logs (yes, I’ve a long retention policy!). Some facts to start:
- Total hits: 9.534.062
- 404 errors: 343.606 (3.6%)
As you can see on the graph below, the 404 error code comes in the fifth position after the classic 200 and 3xx codes.
(Click to enlarge)
As I’m trying to keep the blog clean, this huge amount of “not found” errors looked strange to me. I decided to generate more statistics. What can we deduct? For a while, the big winner is the TimThumb vulnerability discovered in Augustus 2011. The exploit was released the 3rd of Augustus and the first attempt hit me on the 4th! Still today, I received plenty of probes (see this month):
(Click to enlarge)
The TimThumb scans are coming from three main sources as see on the Google map below (the live map is available here).
(Click to enlarge)
Another trend this month: more and more .rar archive files are tested. Especially this month. Why? I’ve absolutely no idea! If you have ideas, feel free to post your comments!
(Click to enlarge)
The top-10 of requested .rar files is:
- /mirserver.rar
- /web.rar
- /www.rar
- /mirserver1.rar
- /wwwroot.rar
- /youxi.rar
- /mh.rar
- /manhua.rar
- /mirserver2.rar
- /mirserver3.rar
Some of them look like performed by scanners which are looking for websites backups. But I did not see the same amount of requests for .tar.gz or .zip files! (Except for “www.zip“) I also saw request for files based on numbers: 5555.rar, 8888.rar, 444.rar, etc. Based on Google, those file are massively infected with malwares but why look for them on my server?
Finally, scanners are looking for .asp (Microsoft .Net) pages. Especially for the last two months:
(Click to enlarge)
The top-10 of requested .asp pages is:
- /save.asp
- /plug/save.asp
- /gmsave.asp
- /diy.asp
- /shell.asp
- /dama.asp
- /upfile_flash.asp
- /FCKeditor/editor/filemanager/connectors/asp/connector.asp
- /xiaoma.asp
- /up_BookPicPro.asp
And what about common tools or web interfaces? The top-10 is:
- /setup.php
- /scripts/setup.php
- /admin
- /login.php
- /phpmyadmin/
- /myadmin/
- /mysql/
- /db/
- /administrator/
- /db/
As you can see, there is plenty of useful information in your Apache (or any other webserver) log files! Keep an eye on your 404 errors to discover new trends! A temporary peak of 404 errors could mean that your server is under an attack…
Ranking People Like Domains or IP Addresses?
Real time events or network traffic analysis is interesting to track suspicious behaviors. And, if you add some external sources of information, you could increase even more the capability of detecting real events. Such ranking sources applies usually to IP addresses and domain names. They are plenty of online resources with huge lists of suspicious IP’s/domains (a good starting list is available here). You can of course create and maintain your own private lists. But can we implement the same ranking with “people” (humans)?
Tracking Tweets in your SIEM
My previous post about monitoring pastebin.com within your SIEM gave ideas to some of you. On aftershell.com, you will find a fork of my script to monitor WordPress RSS feeds. If pastebin.com may contain relevant piece of information as well as blogs, do not underestimate the value of social networks! They are plenty of them: Google+, LinkedIn, Twitter, Facebook, etc. Let’s focus on Twitter.
Vulnerability Management: OSSEC & Secunia PSI
“Vulnerability Management“… This is an important topic for your corporate security. One of the steps in this process is the monitoring of your applications and operating systems. With hundreds (thousands?) of devices connected to your network, how to keep an eye on the applications and patches installed on all of them? There exists plenty of vulnerability management tools which allow you to track/install patches from a central place. But again, most small organizations don’t have the resources or budget to deploy this kind software and users have to keep your laptops/workstations up-to-date. This article will show you how to implement a basic control of your pool of Windows workstations based on Secunia PSI and OSSEC.
Secunia PSI (“Personal Software Inspector“) is a free tool which scan your computer at regular intervals and reports uninstalled patches, unsafe (or end-of-life) applications.
(Source: secunia.com)
Secunia PSI can be integrated to their commercial product called “CSI” (“Corporate Software Inspector“). About OSSEC, I won’t present the product once more. If you follow my blog, you already know that it’s one of my favorite tool.
The goal will be to configure OSSEC to keep track of changes detected by Secunia PSI. How? Secunia stores all the data about your computer in their own “cloud”. Nothing is kept local. An API is available (examples here) but it’s not easy to use it from OSSEC. Fortunately, there is an interesting log file located in “C:\Program Files\Secunia\PSI\psialog.txt“. Just a remember: your system are full of logs! In our case, it contains all the required material to perform basic alerting like:
- Report applications changes (added, removed, clean, unsafe, …)
- Report of the PSI score is not 100%
A few words about how Secunia stores the scan results. You must be aware of this. Your computer profile is sent to the Secunia cloud (psi.secunia.com) via HTTPS:
POST /psi_api/2004/?type=data& \
scantype=4& \
tz=-3600& \
domain=0& \
uid=6X4EUbXhif39a59e330eea22c2d56acaxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx& \
ui=agent& \
langroup=LAB& \
host=WIN7LAB
Your UID (stored in the Windows registry) is randomly generated during the software installation. Keep this in mind: they know a lot of details about you and your company. Example: the workgroup or domain is sent to Secunia. As most organization use their domain name, it’s easy for Secunia to know which software is used in which company. Of course, based on all the collected data, they’re able to perform nice statistics.
Now, the recipe:
- An OSSEC server
- OSSEC Windows agents properly configured
- Secunia PSI properly deployed and configured
- Some decoders and rules
Let’s have a look at the PSI log file. Two types of events are important. A first one gives details about the application being scanned (new, unsafe, etc) and the second one gives details about the scan results (this one contains the percentage).
[11/25 19:17:54.288] Setting state of 'Adobe Flash Player 11.x' to clean
[11/26 14:19:24.791] server.showBalloon('Secunia PSI - Scan Completed Successfully', \
'You have insecure programs on your PC.
Secunia System Score: 67%
Based on the following detections:
5 Insecure programs
1 End-of-Life programs
12 Patched programs
')
First, let’s write decoders to handle those events:
<decoder name="secunia-psi-event"> <prematch>^[\d\d/\d\d \d\d:\d\d:\d\d.\d+] Setting state of</prematch> <regex>Setting state of '(\.+)' to (\S+)$</regex> <order>extra_data, action</order> </decoder>
<decoder name="secunia-psi-score"> <prematch>Secunia System Score:</prematch> <regex offset="after_prematch">(\d+)%</regex> <order>status</order> </decoder>
In the first rule, the variable “extra_data” will contain the application name and “action” its status. The different status I already detected are: “determining“, “clean“, “created“, “added“, “approved“, “downloading“, “downloaded“, “entry“, “queued“, “condfailt“, “started” (they are maybe others, let me know if you have more information). In the second decoder, the variable “status” will contain the integer representing the scan score (0-100). Now we can define some rules:
<rule id="110000" level="0"> <decoded_as>secunia-psi-score</decoded_as> <description>Secunia Scan Results</description> </rule> <rule id="110001" level="9"> <if_sid>110000</if_sid> <regex>: \d\d%</regex> <description>Vulnerabilities found. Secunia scan score not 100%</description> </rule> <rule id="110002" level="0"> <decoded_as>secunia-psi-event</decoded_as> <description>Secunia PSI Event</description> </rule> <rule id="110004" level="9"> <if_sid>110002</if_sid> <match>added</match> <description>New application detected by Secunia</description> </rule>
Results will look like:
OSSEC HIDS Notification. 2011 Nov 28 20:55:40 Received From: (win7lab) 192.168.254.240->\Program Files\Secunia\PSI\psialog.txt Rule: 110003 fired (level 9) -> "New application detected by Secunia" Portion of the log(s): [11/28 20:49:14.130] Setting state of 'Adobe Flash Player 11.x' to added --END OF NOTIFICATION
OSSEC HIDS Notification. 2011 Nov 28 20:59:27 Received From: (win7lab) 192.168.254.240->\Program Files\Secunia\PSI\psialog.txt Rule: 110001 fired (level 9) -> "Secunia scan score not 100%" Portion of the log(s): Secunia System Score: 98% --END OF NOTIFICATION
Simple and efficient! However, it’s not complete. At the moment, I don’t know where to find the status of the Windows patches. I’m trying to get more info from Secunia. I’m not aware of commercial SIEM nor log management solutions which interpret Secunia PSI results; this is good for OSSEC. You can imagine plenty of scenarios: hosts with a scan score below 100% can be stored to temporary tables. Dangerous applications can also be stored in a table and used to write correlation rules…
Detecting Defaced Websites with OSSEC
In the scope of the OSSEC Week, here is a quick contribution which can greatly help you to monitor suspicious changes on a website. Today, your corporate website is the very first contact you have with your customers, partners, press, etc. It’s your window to the world. Nobody can pretend being fully protected against defacement or intrusions. It’s important to be alerted as soon as possible when something “suspicious” occurs. It’s never a good story to be alerted by a third party that you’ve been hacked…
OSSEC integrates by default a FIM (“File Intregrity Monitoring“) feature which can be used to detect changes in files on your web servers. But sometimes, those servers are outsourced or not fully controlled from A to Z by you or your team. Anyway, OSSEC can detect changes in remote files via another feature called “full_command“. How?
In your ossec.conf, define a new “file” entry like this:
<localfile>
<log_format>full_command</log_format>
<command>wget -o /dev/null -O - http://www.company.com | sha1sum</command>
</localfile>
This command will grab the homepage of www.company.com and compute its SHA1 digest. Now, define a new alert in local_rules.xml:
<rule id="123456" level="8">
<if_sid>530</if_sid>
<match>ossec: output: 'wget -o /dev/null -O - http://www.company.com</match>
<check_diff />
<description>Change detected on www.company.com.</description>
</rule>
Of course, you can detect changes on specific files:
<command>wget -o /dev/null -O - http://www.company.com/file.xml | sha1sum</command>
Or on a group of files read from another one:
<command>for I in `cat files.tmp`; do wget -O - -o /dev/null http://site.be/$I | sha1sum; done</command>
Do you need to track changes related to your BGP Autonomous System? Use this one:
<command>whois -h whois.ripe.net as12345 | sha1sum</command>
Use your imagination! There are tons of other examples with the “full_command” OSSEC feature. Remember: As soon as you detect a problem, as soon as you fix it! Just one last remark, most websites integrated dynamic content like banners, newsfeed. Select carefully which changes you need to track otherwise, you will be flooded by false positive alerts!
