There is an interesting analogy between between debugging software and debugging the security of your systems.
As Brian W. Kernighan and Rob Pike wrote in “The Practice of Programming”
As personal choice, we tend not to use debuggers beyond getting a stack trace or the value of a variable or two. One reason is that it is easy to get lost in details of complicated data structures and control flow; we find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is. More important, debugging statements stay with the program; debugging sessions are transient.
In programming, it is faster to examine the contents of a couple of variables than to single-step through entire sections of code.
Collecting security logs is key to information security management not only for understanding what and why an event happened but also in order to prove regulatory compliance with regulations such as the HIPAA security rule. The business requirements are that security logs should be both relevant and effective.
- Relevant content of audit controls: For example, providing a detailed trace of an application whenever it elevates privilege in order to execute a system level function.
- Effective audit reduction and report generation: Given the large amount of data that must be analyzed in security logs, its crucial that critical events are separated from normal traffic and that concise reports can be produced in real-time to help understand what happened, why it happened and how it was mediated and how to mitigate similar risks in the future.
In security log analysis, it is faster and definitely more effective for a security analyst to examine the contents of a few real time events than to process gigabytes or terabytes of security logs (the equivalent of stepping through or placing watch points in sections of of a sub-modules with hundreds or thousands of lines of code.
When you have to analyze security logs, it is easy to get lost in details of complicated data and flows of events and find yourself drifting off into all kinds of directions even as the bells go on in the back of your mind that you are chasing ghosts in a futile and time-consuming exercise of investigation and security event debugging.
In order to understand this better, consider another analogy, this time from the world of search engines.
Precision and recall are key to effective security log analysis and effective software debugging.
In pattern recognition and information retrieval, precision is the fraction of retrieved instances that are relevant, while recall is the fraction of relevant instances that are retrieved. Both precision and recall are therefore based on an understanding and measure of relevance. When a program for recognizing the dogs in a scene correctly identifies four of the nine dogs but mistakes three cats for dogs, its precision is 4/7 while its recall is 4/9. When a search engine returns 30 pages only 20 of which were relevant while failing to return 40 additional relevant pages, its precision is 20/30 = 2/3 while its recall is 20/60 = 1/3. See Precision and recall in the Wikipedia.
In other words – it doesn’t really matter if you have to analyze a program with 100,000 lines of code or a log file with a terabyte of data – if you have good precision and good recall.
The problem is however, that the more data you have, the more difficult it is to achieve high precision and recall and that is why real-time events (or debugging statements) are more effective in day-to-day security operations.
