Research offers software salvation from AV friendly-fire

In the war on malware, legitimate software can be caught in the crossfire. But a Melbourne researcher is developing a method that could reduce the incidents of anti-virus false-positives.

Silvio Cesare has developed a method, Automated Static Unpacking Using Speculative Decompression, which serves as an alternative means to unwrap the obfuscation tricks coders use to hide malware.

Before malware could be identified, anti-virus vendors must detect and unravel so-called packing techniques used to conceal malicious code.

But malware writers lace packing with countermeasures that prevent anti-virus from running the packed malware in emulated sandboxes, a process used by most systems.

Cesare said anti-virus systems unable to bypass the counter-measures would mark all unknown packed code as malicious, meaning legitimate software could be purged.

"Sometimes legitimate commercial software is packed which means without analysis of the hidden code anti-virus would incorrectly label it as malicious,” Cesare wrote.

“Unpacking code can be a challenging problem and non-traditional packing techniques such as instruction virtualisation are quickly becoming more and more used by malware authors.”

Cesare's method was an alternative that jettisoned the need to unpack malware in emulated environments.

Static unpacking aimed to identify malware in its packed state by utilising packing algorithms, which eliminated the need to unpack malware in a virtual or sandbox environment.

It was invulnerable to anti-emulation countermeasures.

“Our system can easily unpack these types of malware.”

The system could determine the type of packing used on a given malware sample with entropy analysis and packer classification.

While the Melbourne security researcher had dropped further development of the system, he may build it into a new online analysis tool.

Cesare said the free tool would compare malware samples to reveal the heritage of code, and determine if it was plagiarised or borrowed.

“This can allow you to detect malware variants, or to detect if a sample belongs to a known
malware author's work,” he said.

The malware binaries would be unpacked using the Reboot emulator Cesare created during his Masters degree at the Central Queensland University in 2008.

Cesare said static unpacking system was promising but too underdeveloped to best existing methods.

“It seems that the only safe solution for anti-virus is to perform packer detection and flag all such occurrences as potential malware. For legitimate software, white listing and co-ordination with anti-virus vendors may be the only secure way forward.”

Copyright © SC Magazine, Australia