Security vendor Tencent found cheating in tests

Chinese anti-virus developer Tencent will lose its certifications after it was found to have submitted products with optimisations designed to improve their ratings in independent third-party testing.

Tencent is the second Chinese security vendor to be caught cheating recently. Last week, rival anti-virus developer Qihoo 360 was stripped of its awards after it was found to have submitted products for testing with its default detection engine disabled, instead using BitDefender for improved results.

AV-Test chief executive Andreas Marx told iTnews Tencent's performance was significantly better from one month to another, which rang alarm bells for the test lab.

"As part of our research, we found that all the tools we're using for the performance benchmark were "whitelisted" using their name (eg "AcroRd32.exe"), so the Tencent scan engine won't check them anymore," Marx said.

"This only applied to exactly the programs we're using (some are well know, like Acrobat Reader, but some are less commonly used).

"The "whitelist" was even extended from month to month by exactly the tools we used for the next benchmark, so this optimisation was clearly focusing on our tests only."

Once the tools were renamed, Tencent product performance decreased significantly, confirming the test-specific optimisations.

US-based industry forum Anti-Malware Testing Standards Organisation (AMTSO) strongly condemned the vendor's actions.

"Unethical behavior cannot be tolerated and decisions need to be made about how to penalise those who have bypassed the rules of engagement in order to falsely achieve a better testing result," AMTSO said.

AMTSO pointed out that the product testing drives anti-malware vendors to improve their solutions, leading to better threat protection for users.

Conversely, bad testing not only misleads consumers and businesses, it tarnishes the reputation of everyone involved, AMTSO said.

Competing anti-malware tester Virus Bulletin also revoked all 2015 certifications for Tencent, saying it expects test participants to behave in an open and ethical manner, and considers gaming of results unhelpful to both developers and users.

Virus Bulletin said it would impose stricter controls on test participants to reduce the opportunities for cheating.