Logs are important in your security policy. Each devices in your infrastructure generate events and write them to log files. Log files are stored locally and can be reviewed via the tool provided by the device manufacturer. However, it quickly becomes a pain to manage if you’ve hundreds or thousands of devices to keep under control. Another reason to implement a logging solution is the limited storage available on devices themselves. A good retention policy must be defined: An event generated today can be useful in the future to investigate an incident. The solution is to centralize all logs in a unique repository. To perform this, a well-known solution is to use the Syslog protocol. This protocol is widely available on almost all devices (if not natively, like on Windows operating systems, the local events format can be converted via third-party tools). But Syslog lacks of features to make the logging policy robust enough.

Let’s go back to our classic CIA triad! (Confidentiality, Integrity, Availability). How to apply the third principles to logging? Let’s review each of them but I will focus later on “Integrity”.

Availability

First, the Syslog protocol relies on UDP. This protocol is session less and no warranty of a correct delivery to the remote host is given! To solve this problem, we have several solutions:

• Send logs to multiple destinations (lot of devices allow multiple destinations). The side effect is an increased network traffic.
• Use an alternative Syslog protocol over TCP. rsyslog tries to make Syslog more reliable using TCP instead of UDP.

The received data must be stored on a safe storage (RAID based). A dedicated LUN on a SAN can be useful if the amount of logs to handle is important. A RAID system will only prevent you against technical errors (disk failure) but not against human errors or bugs. That’s why data must be part of a strong backup policy with backup media stored on a remote location (to prevent from theft or natural disasters).

Confidentiality

Protecting the packets transmitted over the network can also be a requirement. Syslog messages can be sent through a VPN tunnel or directly encrypted using SSL or TLS. Here again, rsyslog can be helpful.

Once received by the syslog concentrator, messages may contain sensitive or confidential data. Access to them must be protected against non-authorized users. Encryption of the file systems and the backup media will prevent unexpected disclosure.

Regarding the server, it will be installed in a safe place with limited physical access and the operating system will be hardened.

Integrity

Probably the most important aspect of your logging policy. In a previous post, I explained what is a valid evidence. First keep your devices internal clock in sync via the NTP protocol. Then make sure that data are not altered. Alteration is possible at two levels:

• During the transit over the network.
• On the concentration server.

During the packet transit, the packet can be captured by a malicious host (using some kind of MitM attack) and sent back to the logger. Remember, Syslog is based on UDP. It’s very easy to spoof packets or alter them. Syslog-Sign is a protocol designed to avoid integrity problems during packets transit. An IETF draft describes the protocol:

“This document describes a mechanism to add origin authentication, message integrity, replay resistance, message sequencing, and detection of missing messages to the transmitted syslog messages. This specification is intended to be used in conjunction with the work defined in [RFC5424], “The syslog Protocol”.”

Compared to the standard protocol, Syslog-Sign adds:

• origin authentication,
• message integrity,
• replay resistance,
• message sequencing,
• detection of missing messages

But Syslog-sign is facing a major problem: it remained in draft status (expiration in June 2009). It will never be really implemented if not described by an RFC! There exists another RFC (3195) which describe a reliable delivery mechanism for Syslog messages (based on TCP).

Syslog-Sign does not protect against log manipulations on the Syslog concentrator. An evidence must be properly handled. If any suspicious alteration or suspect manipulation is discovered by a Court, the judge will dematerialize the evidence immediately. To keep the evidence valuable, we will use the following tools:

• syslog-ng – will help the administrator to manage its log files. Filter per source, per message types and split logs across several files.
• GnuPg will help to sign the messages to prevent any alteation in the future.
• Truecrypt will encrypt the file system where are stored the log files

Note: there exists commercial Syslog servers which propose extra features like encryption and time-stamping. In the example below, we will use only free software to sign the received messages.

All the operations will be performed on the Syslog concentration server (running Linux). First create a dedicated user with a GnuPg key-pair to sign the Syslog messages (do NOT work with the root account). [Note: This post will be cover the tools installation, refer to the document for details]

# useradd syslog
# passwd syslog

Switch to the newly created user and generate it’s GnuPG key:

# su - syslog
$ gnupg --gen-ke
[follow the dialog]

syslog-ng replaces the standard syslog daemon and is configured to split all received messages in files like “YYYYMMDD.log”. Messages from different devices are stored in dedicated directories (based on the host name). To achieve this, let’s use the following configuration directive:

destination hosts { 
  file("/data/log/$HOST/$YEAR/$FACILITY_$YEAR$MONTH$DAY.log"
  owner(syslog) group(syslog) perm(0640) dir_perm(0750)
  create_dirs(yes)); };

The following script will sign the message files.

#!/usr/local/bin/bash
umask 077
TODAY=`date +"%Y%m%d"`
for FILE in `find /data/log/ -type f -name "*.log"`
do
  BASENAME=`basename $FILE`
  # Process if no sig found and not the current file
  if [ ! -f $FILE.gpg -a \
       "$BASENAME.log" != "$TODAY.log" ]; then
    gzip -9 $FILE
    gpg -b --homedir /home/syslog --passphrase-fd 0 \
             $FILE.gz </root/.passphrase
  fi
done

Execute the script once a day via a cron:

0 * * * * /usr/local/bin/sign_logs.sh

Security notice: the argument –passphrase fd 0 helps to read the private key pass-phrase automatically (“0” means STDIN). Of course, this is not the safest method to use a pass-phrase! Take care to store and protect the .passphrase file in a safe directory and make it readable only by root! Integrity will depend on your private key security.

The script above will create a signature for each daily log (with an extension .sig). Using the following command, you’ll be able to check the file integrity:

# cd /data/log/host1
# gpg --verify 20090304.gz.sig 
gpg: Signature made Mon Mar 09 22:51:19 2009 CET \
     using DSA key ID 54BD8F2F
gpg: Good signature from "Syslog Concentrator "

Now, imagine that the original message has been altered:

# gpg --verify 20090306.gz.sig 
gpg: Signature made Mon Mar 09 22:51:19 2009 CET \
    using DSA key ID 54BD8F2F
gpg: BAD signature from "Xavier Mertens "

This method is not perfect but can increase your logs integrity for long term retention.