The vulnerability, deemed "critical" by the company, is found in the ARJ archive file format parser. The AntiVirus Library handles compressed files using the ARJ standard. The company said in a statement that this format is "too flexible, especially in the file name field in the local header."
This file name is stored as a null-terminated string and limited only by the overall size of the local header.
Therefore it is possible to create an ARJ archive file that overwrites data after the allocated 512-byte buffer. An attacker could use this specially-crafted file to execute arbitrary code.
A spokeswoman for the company said there had been "no reports of this type of specially crafted file seen among our customer base."
"This issue has been resolved through the most recent Scan Engine version, version 7.51, that is available for download for customers through the Trend Micro Update Center," she added.
Security company, ISS, discovered the flaw and in its advisory said "several ISPs and vendors also use Trend Micro AntiVirus Library and are likely vulnerable."