Be careful what you measure

Almost everyone, from health services to primary schools, seems to be singing the metrics mantra these days. It seems logical enough; we need to measure performance and then we can see how well things are going and, if necessary, try to fix any problems. Certainly, objective measurements are useful but, unfortunately, all too often metrics tend to reflect what is easy to measure rather than what is a good indicator of performance.

The IT business is, of course, a great place to collect numbers. Almost every device on a network can churn out statistics at will. But relating these to your business is not always as easy as it seems.

Take virus protection. The typical measure of its effectiveness is the number of virus incidents affecting the business. But taken in isolation, this measure ignores the changing nature of the threat. I strongly suspect most readers will have suffered more virus incidents since the rapid growth of macro and email-borne threats. Does that mean that our virus protection ten years ago was better? No, it just means that things have moved on.

Along with every metric comes a target, and these can be equally troublesome. I once saw a target for security of "zero security incident reports". Easy to meet: simply don't report incidents. Now, does anyone seriously believe a large company with no security incidents reported is secure? The same applies to software; I'm much happier to see faults found and fixed.

Of course, quality-assurance metrics aren't the only problem areas with numbers. In the risk-analysis process, various numbers are often applied almost arbitrarily to risks. To security professionals this isn't a problem; we know that it isn't as simple as saying 10.4 is safe but 10.3 is a worry. But to others, the presence of arbitrary measurements can give an air of scientific validity.

Metrics aren't a bad thing per se, and simply assuming everything is all right without checking is as foolish as putting too much faith in arbitrary numbers. Unfortunately, however, the trend appears to be making sure that the graphs look right, not that the protection itself is adequate. While this might get you through an audit, it won't protect your business.