All Your Logs are Belong to the Cloud…

Ever heard of Loggly? This is a new cloud service which presents itself as “Logs Made Easy“. I’ll not come back on the definition of cloud computing, its benefits and issues. If you are looking for interesting information about this topic, I suggest you to visit Craig Balding’s blog cloudsecurity.org.

At the moment, Loggly is still in development and proposes beta accounts. I’m currently using a friend’s account and hope to receive my own access soon (Dear Loggly administrators, if you read my blog… 0:-) ). The purpose of this new service is simple and sexy: It will take care of your beloved logs! Everybody agrees to say that logs are painful to manage: they require lot of resources (disk, cpu) and must be reliable to investigate incidents when required. Loggly can help you to achieve this. How? Just be configuring your servers to send their logs to the Loggly cloud. They will be indexed and available for search actions via a simple web browser.

Loggly did not reinvent the wheel and those who have experience with log management solutions will easily find their way. To parse your logs, it’s easy as 1, 2, 3!

1. Define an “input” which bind to a specific port on your hostname (defined with your account – <something>.loggly.com)
2. Configure your server/device to send its events to the hostname/port specified above.
3. Start search your logs!

At the moment, Loggly accepts Syslog events over TCP & UDP or HTTP. For reliability reasons, I suggest you to use only Syslog over TCP. It means that Loggly is able to accept events from almost every applications or device “speaking” Syslog. For my personal tests, I’m sending a full Linux box server (two to four MBytes/day) to Loggly for a few days:

Once the events are collected and indexed by Loggly, you can search them using a “shell” with simple commands:

I’ll not give more information about the shell here. It’s very intuitive and works quite well. As you can imagine, Loggly will never replace a real log management or SIEM solution but it can be very helpful in some cases.

The big question is of course, can we trust a solution of log management in the cloud. There are several issues that must be addressed (following the CIA principle)

• The logs confidentiality must be conserved at no price. Events generated by applications contain often sensitive data like IP addresses, FQDN, logins, executed commands, paths, etc. The events are sent to Loggly via the wild Internet. They must be sent using encrypted protocols. Once received, they must be processed in the right way and encrypted on disks. Access to the web console must be properly secured and monitored (who accessed the console to search for which kind of information).
• The log integrity must be controlled. How to prevent the events stored in the cloud to be altered in the cloud or in transit to the cloud (MiTM attacks)
• Finally, the availability is critical too. I suppose that Loggly will offer commercial services with a business model based on MB/day or something similar. A log management solution must be implemented on a long term view. Remember: some events have no value TODAY but may have a bigger value in a few months or years when you’ll have to investigate an old incident or upon a request from the authorities.
• As usual, a cloud solution is synonym of questions about the physical location of the data. In some cases, the storage of data in foreign countries could be a business issue.

If you keep these consideration in mind, I think that Loggly could be interesting for some very small companies which don’t have resources (time, knowledge, budget). I’m in contact with a lot of such companies who have nothing in place to manage their logs. The clould could be better than nothing… (Reminder: you must take care of your logs!)

Finally, something interesting for the developers… As said above, Loggly accepts also logs over HTTP and  an API is available to send and search events. Example:

curl -d "127.0.0.1 - there's no place like home" http://logs.loggly.com/inputs/83e527d7-fad3-4d93-89da-0c2d8c0bcd6c

I’m curious to see the future of “log management in the cloud“!