Been a couple weeks since I blogged – have my head down on a few medical device projects and a big PCI DSS audit where I’m helping the client improve his IT infrastructure and balance the demands of the PCI auditors.
Last year I gave a talk on quantitative methods for estimating operational risk of information systems in the annual European GRC meeting in Lisbon – you can see the presentation below.
As a I noted in my talk, one of the crucial phases in estimating operational risk is data collection: understanding what threats, vulnerabilities you have and understanding not only what assets you have (digital, human, physical, reputational) but also how much they’re worth in dollars.
Many technology people interpret data collection as some automatic process that reads/scans/sniffs/profiles/processes/analyzes/compresses log files, learning and analyzing the data using automated  algorithms like ANN (adaptive neural networks).
The automated log profiling tool will then automagically tell you where you have vulnerabilities and using “an industry best practice database of security countermeasures”,  build you a risk mediation plan. Just throw in a dash of pie charts and you’re good to go with the CFO.
This was in fashion about 10 years ago (Google automated audit log analysis and you’ll see what I mean) for example this reference on automated audit trail analysis,  Automated tools are good for getting a quick indication of trends, and  tend to suffer from poor precision and recall that  improve rapidly when combined with human eyeballs.
The PCI DSS council in Europe (private communication) says that over 80% of the merchants/payment processors with data breaches  discovered their data breach  3 months or more after the event. Yikes.
So why does maintaining 3 years of log files make sense – quoted from PCI DSS 2.0

10.7 Retain audit trail history for at least
one year, with a minimum of three
months immediately available for
analysis (for example, online, archived,
or restorable from back-up).
10.7.a Obtain and examine security policies and procedures and
verify that they include audit log retention policies and require
audit log retention for at least one year.
10.7.b Verify that audit logs are available for at least one year and
processes are in place to immediately restore at least the last
three months’ logs for analysis

Wouldn’t it be a lot smarter to say –
10.1 Maintain a 4 week revolving log with real-time exception reports as measured by no more than 5 exceptional events/day.
10.2 Estimate the financial damage of the 5 exceptional events in a weekly 1/2 meeting between the IT manager, finance manager and security officer.
10.3 Mitigate the most severe threat as measured by implementing 1 new security countermeasure/month (including the DLP and SIEM systems you bought last year but haven’t implemented yet)


I’m a great fan of technology, but the human eye and brain does it best.