The Article 29 Working Party has recently highlighted the importance of detecting and mitigating information security breaches. One of the key tools in doing this is logfiles: the European Court of Justice in Breyer v Germany recognised the role of web server logs, the Article 29 Working Party guidelines mention logs and network flow data. Common questions are “what logs do I need?” and “how long should I keep them?”. But that’s actually the wrong place to start.
Instead, look at your processes for detecting and responding to breaches. If you don’t have such processes, write them. Those processes will tell you which logs you need: for attacks from outside, logs from externally visible servers and network flows are likely to be the foundation; for internal attacks, and to deal with complaints, you need to be able to translate the local IP and email addresses in those logs and reports into the identity of the individuals responsible for the problematic activity. Jisc’s technical guide to logfiles has more detail. Exercises are a good way to test processes and logging: start from a (theoretical) report or alert (NIST Appendix A is a good source of example scenarios), and apply your response process to confirm that it works. That may highlight that you are missing some logs, tools or processes; or that you are unnecessarily keeping logs that no process will ever use.
As to how long to keep logs: clearly you need to keep them long enough that you’ll be able to investigate the incidents that are discovered. Getting better at discovering your own incidents should mean that happens more quickly. Monitoring how long it takes you to detect incidents and then to fix them can be a useful indicator of the effectiveness of your incident response, as well as a guide to appropriate retention times. However surveys still show that incident detection can take several months, particularly in sectors or organisations where the majority of incidents are detected by third parties. If you are one of those, improving your own detection is a better approach than keeping logs longer in the hope that someone else will eventually find your incidents for you.
Under the General Data Protection Regulation, the justification for retaining logs is that the benefits to users of detection are greater than the risks caused to them by retention. But the longer an intruder stays in your systems undetected, the less benefit there will from a detailed investigation. Once the intruder has taken all the data accessible from a system, backdoored it to give them a full and persistent control, and modified applications and logs to conceal these activities, the benefit of all those logfiles is pretty small.
Many years ago, the UK Data Protection Commissioner suggested keeping routine logs for three to six months. Logs for incidents being investigated can obviously be kept for longer if required. At the time, we hoped that improved detection and investigation techniques would gradually let us reduce that period but, sadly, it still seems about right for most organisations.
