Beware of gamed research

Scandal has emerged over the past two weeks after two Chinese antivirus vendors were busted cheating in independent product testing.

The first investigation into Qihoo 360, China’s largest security vendor, revealed it had substituted its own, slower scanning engine for the faster alternative from Romanian company Bitdefender in independent trials of its software.

The second discovery was that tech heavyweight Tencent had “submitted products with optimisations designed to improve their [sic] ratings in independent third-party testing.”

Both companies will lose their awards and certifications and undoubtedly now be discredited in the industry.

The security market is not a forgiving place, so these two will suffer significant setbacks. It will be interesting to see how Qihoo 360’s share price fairs on the New York Stock Exchange when this news is fully comprehended by its investors.

These stories highlight that there is value to be garnered from independent testing and benchmarking: if you happen to be in the market for an antivirus solution, you’ll certainly now know to stay away from those two companies.

But how do you know you can trust the testers? How can you be sure that the testing organisation is not on the payroll of the security vendor?

This kind of testing could easily be used as a marketing stunt to create a biased view of the competition, therefore it’s vital that you understand who is funding the tests and whether independent is truly independent.

Done well, independent testing can be used to help us with our purchasing decision-making to ensure products are doing what they purport, and that the vendor is a trustworthy supplier.

The fact is, security products are selected to mitigate risks to which your systems and information may be subject.

For example, if you think you might be infected by malware, get yourself an antivirus product. If you are concerned about a hacker attacking your internet connection, get yourself a firewall. But how do you select the best product for the job, given the myriad of products in the market?

If someone in your contracts department puts together a tender, or your purchasing department takes a look at the market, selection will undoubtedly be driven by cost. But how much due diligence goes into the evaluation, taking into account the utility and warranty that the product can do what it says on the label?

I remember a couple of years ago I was at a security conference where a well-respected database penetration tester scared the bejesus out of everyone in the audience with tales of his exploits.

The floor opened to questions, and someone sked what the speaker’s opinion was of Common Criteria testing. The response was curt and damning: “it’s not worth the paper it’s written on.”

That is a typical response from the perspective of a brilliant penetration tester, given the black and white nature of that profession: either you can break a system or you can’t.

Nevertheless, security is neither black nor white; it’s full of shades of grey where colours represent risk, impact and likelihood rather than absolutes.

The purpose of Common Criteria is to "ensure that evaluations of IT products and protection profiles are performed to high and consistent standards and are seen to contribute significantly to confidence in the security of those products and profiles.”

Common Criteria is an independent organisation, trusted by countries and governments around the world, including Australia, to evaluate security products to prove they do what they are supposed to do.

The Australian Signals Directorate (ASD) maintains an Evaluated Product List (EPL) that allows government agencies (and anyone that wants to take heed) to consider evaluated products trustworthy in comparison to products that have not passed muster.

ASD has a partner arrangement with Common Criteria, which also has partners with the UK, USA, and many other countries, where similar evaluation facilities exist.

Information security gives us assurance that we are safe. Software assessments completed by government and independent agents should factor into your decision-making when designing security architectures, as this gives you some assurance.

You won’t have a solution that is perfect, but it is closer to being trustworthy than a product that no one has looked at.

Remember, just because you have bought software labelled as an antivirus product, it doesn’t always mean it will do the job.

These two Chinese companies have proved this: their customers had considered the products trustworthy, but testing proved them not to be so.

The lesson is to do as much due diligence as you can and use all the resources you have at hand to ensure you validate the claims of the vendor.

If they don’t have the benchmarks and are unwilling to be independently tested, consider closing the door on that conversation and looking elsewhere.