Last week, I went to London to attend the RSA Conference Europe (my wrap up is here). One of the sessions I followed was presented by Eric Vyncke about “forensics in a post IPv4 exhaustion“. You should live on another planet if you’re not aware of the coming IPv4 exhaustion. Today, the big challenge for Internet Service Provider is to handle IPv6 deployments in parallel to the implementation of new ways to limit the lack of IPv4 addresses. Once of these techniques is called “Carrier Grade NAT“. As Eric mentioned in his talk, ISP will start (some already started – mainly mobile operators) to share IPv4 addresses between customers. This will have a huge impact on forensics investigations!
Why? When you are looking for evidences (example: to track a security incident), your first reflex is to? Have a look at your logs! Good! Let’s have a look at a regular Apache access log entry:
12.34.56.78 - - [17/Oct/2011:18:52:22 +0200] \ "GET /favicon.ico HTTP/1.1" 200 1173 "-" \ "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110921 Ubuntu/10.04 (lucid) Firefox/3.6.23"
The IP address ‘12.34.56.78’ is the machine which made an HTTP request on your server. A simple whois lookup will give you the ISP or company which “owns” this IP address with usually a way to contact to report an abuse. Example:
NetRange:Â Â Â Â Â Â 12.0.0.0 - 12.255.255.255
CIDR:Â Â Â Â Â Â Â Â Â Â 12.0.0.0/8
OriginAS:Â Â Â Â Â Â
NetName:Â Â Â Â Â Â Â ATT
NetHandle:Â Â Â Â Â NET-12-0-0-0-1
Parent:Â Â Â Â Â Â Â Â
NetType:Â Â Â Â Â Â Â Direct Allocation
Comment:Â Â Â Â Â Â Â For abuse issues contact abuse@att.net
RegDate:Â Â Â Â Â Â Â 1983-08-23
Updated:Â Â Â Â Â Â Â 2010-11-18
Ref:Â Â Â Â Â Â Â Â Â Â Â http://whois.arin.net/rest/net/NET-12-0-0-0-1
OrgName:Â Â Â Â Â Â Â AT&T Services, Inc.
OrgId:Â Â Â Â Â Â Â Â Â ATTW-Z
Address:Â Â Â Â Â Â Â 200 S. Laurel AVE.
City:Â Â Â Â Â Â Â Â Â Â MIDDLETOWN
StateProv:Â Â Â Â Â NJ
PostalCode:Â Â Â Â 07748
Country:Â Â Â Â Â Â Â US
RegDate:Â Â Â Â Â Â Â 2009-12-18
Updated:Â Â Â Â Â Â Â 2010-08-30
Comment:Â Â Â Â Â Â Â Contact AT&T Abuse ( abuse@att.net ) for policy abuse issues.
Comment:Â Â Â Â Â Â Â All policy abuse issues sent to other POCs will be disregarded.
Now, if this ISP uses Carrier Grade Nat, it cannot point to the right customer immediately! One thousand (or more) customers are potential hackers. To uniquely identify the bad guy, the following critical information is mandatory:
- The ISP must keep a track of all the IP flows generated by its customers INCLUDING THE SOURCE PORT!
- You have to provide the IP address INCLUDING THE SOURCE PORT!
Based on those points, correlation is possible to find the attacker. From the ISP perspective, this could have huge impacts on the tools it uses to store the flows. From your perspective, if you have access to the firewall logs in from of your application, just have a look at the logs. If your source of information is only your application log, it’s time to upgrade it!
Start logging the source port NOW! Here is an example (source: Eric Vyncke) with Apache logs. Replace the default format:
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
With this one:
LogFormat "%h:%{remote}p %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
Of course, if you change the format of your log, you have to replicate this change to your log management tools! My Apache servers are monitored by OSSEC, so my new Apache decoder is now:
<decoder name="web-accesslog"> Â <type>web-log</type> Â <prematch>^\d+.\d+.\d+.\d+:\d+ </prematch> Â <regex>^(\d+.\d+.\d+.\d+):(\d+) \S+ \S+ [\S+ \S\d+] </regex> Â <regex>"\w+ (\S+) HTTP\S+ (\d+) </regex> Â <order>srcip, srcport, url, id</order> </decoder>
This is an excellent example which proves that log management is complex and must be deployed with the help of strong procedures. I quickly made a round of classic open source applications almost none is able to log the source port! To resume:
- Start to log the source port now!
- Change your application log format.
- Adapt your log management solution
- Train your developers to write correct information into log files
- And (last but not least), implement IPv6!
Bon à savoir