A guide to better logging

To build a good security monitoring program, logs are critical.

They feed almost everything we do in monitoring from event correlation to auditing.  We need logs from things such as security tools, network devices, servers, databases and applications in order effectively monitoring networks.

Any SIEM, analyst, or auditor is only as good as the data available for analysis. Think garbage in and garbage out.

Networks and systems today are growing ever larger and more complex.  With that, almost all devices can generate logs and there are many different levels of logging that can be configured for each device. 

It is crucial to ensure you get the right logs from the right devices and at the right level.  Without this you can and will miss events of interest.

However, getting logs from all devices and systems can lead into thousands of devices sending logs and requiring storage of those logs. 

Here are some key points to consider when setting up or evaluating your logging infrastructure:

1.  Know your budget.
The more important the data, the more money companies are generally going to spend on it.  I know money is tight and sadly security is often times the first place for cuts.

However, you can only work within the means of the funding you have available.  This is important because the more logs you take in requires a more robust logging infrastructure as well as storage for those logs. 

You may have to choose smartly about the logs you can process from which devices.  Know the budget you have to work with when starting up.  For those doing an evaluation, its a good time to see if you need to grow and how much that will cost.

2.  Determine the devices you should have logging.  
There is no simple answer to that question.  Ideally its everything, but realistically that won't work.

Ask yourself how much storage do you have available? How many messages per second can your infrastructure support?  How big are the different logs from the different systems you'll be receiving? 

You also need to know what you are trying to protect and where it lives. Once you answer those questions, it becomes easier to look at your infrastructure for the key devices/systems that you will need logs from. You want logs coming in that will allow you to paint as complete of a picture as possible.

3.  Determine what level of logging is needed and document it.
This should be a group decision.  What groups in your organization use the logs to support their various jobs? 

It is those groups who need to have a say in what that level would be be. It is equally important to document this and the logging level for the different device types for future reference.  

For example, a Cisco router has eight different logging levels and each of these will provide more granular information resulting in more log entries.  If you set the level to debug, you can fill your central log storage up pretty fast if you have alot of devices logging.

If you set it to alerts then you may not get the information you want.  You can even mix and match the logging levels by having devices that are forward facing have more detailed logging levels and those devices that are more protected farther back in the network having less detailed logging.

Remember, that this can be changed as necessary.

4.  Know the log retention policy.
Not only do you have to have enough log storage capacity on your logging infrastructure, but you may also face the requirement of retaining those logs for a specified length of time.

If you have a log retention policy, you need to know what it is to ensure you have enough SAN or offline storage available to retain the logs you generate.

Just because your logging infrastructure may be able to capture the logs you are generating, doesn't mean that you have the necessary long term storage capabilities to meet the retention capabilities.  

5.  Monitor your log submissions.
How do you know are still getting the logs you asked for, at the level you want and nothing has changed? 

This is probably one of the toughest areas and the one most often overlooked  My experience has been that people hand folks a SOP with how to send their logs, confirm they are getting logs and nothing more.

This is tough, especially in a large organisation where you can have thousands of devices sending you logs.  How do you know if anything has changed?  You can't afford not to know when so much is riding on the logs you recieve.

6.  Plan for the future.
If your network is going to grow, you need to ensure your logging infrastructure can grow with it.  This can include many areas such as log servers, appliances, SAN storage, offline storage, capacity of tools that are going to ingest these logs, additional personnel, and so on.  You need to plan well in advance of when you are going to need to expand.

Your logging team has to have a pulse on everything going on with the network.  A logging infrastructure takes a lot of work, but the benefit is worth it. 

It provides the foundation of your security monitoring and deserves more time than it is often given.

If your auditors check logs for a specific issue and report that all is normal, ask yourself if it is because the log information is lacking.

This blog appeared at SANS ICS.

Copyright © SC Magazine, Australia