
The study (PDF download) compared the number of flaws during the 90 days after the application's launch for Windows Vista; Windows XP; Red Hat Enterprise Linux 4 (REL4) workstation; Ubuntu 6.06 LTS; Novell's Suse Linux Enterprise Desktop 10 (SLED10); and Apple's OS X. (Also see the table on the next page)
Vista beat the other operating systems on nearly all fronts. The software logged the fewest fixed vulnerabilities and the least repairs with a severity rating of 'high'. It ranked second in the number of unpatched flaws after 90 days, trailing behind only Windows XP.
Apple's OS X ranked third behind the two Windows versions, followed by Ubuntu, SLED10 and REL4.
Comparing the number of patched and disclosed vulnerabilities is a controversial method of comparing the security between products. Different operating systems have different features, offering attackers diverse ways to attack the software.
Jones attempted to pre-empt criticism over features by including a tweaked version of the three Linux distributions that he included in his test. The adapted version had been stripped of bundled applications that aren't found in Windows or OS X, such as the Openoffice productivity suite, as well as graphics and developer tools.
The number of fixes also fails to consider the popularity with attackers and security researchers. Because Windows is the predominant operating system, users run a greater risk of getting target. But that also has caused the software to get closely scrutinised by both Microsoft and independent security researchers as they attempt to protect their clients.
Researchers meanwhile have started to closely track Apple software. This is sparked both by frustration over the firm's arrogant attitude towards outside researchers as well as the refusal by so-called Mac fan boys to acknowledge that Apple software isn't bullet-proof. This for instance prompted the disclosure of a slew of security flaws in the days after the firm launched its Safari 3 beta for Windows.
Jones' report is bound to receive flame for his security claims, but he seemed well aware of that risk. In closing the 14 page study, he wrote:
"Jeff actively encourages readers to challenge his assumptions, analysis and conclusions and provide critical feedback – but asks for equal (or better) rigor in methodology and analysis to support the challenges, as opposed to enthusiastic espousal of unsupported evangelistic fervor."
Vulnerabilities in the first 90 days after launch:
flaws pre-launch1 (high severity)* | flaws fixed in first 90 days (high severity)* | unpatched after 90 days (high severity)* | |
Windows Vista | 0 | 12 (10) | 15 (1) |
Windows XP | 3 (0) | 36 (23) | 3 (2) |
REL4ws | 129 (40) | 281 (86) | 65 (12) |
REL4ws reduced** | n/a | 214 (62) | 59 (12) |
Ubuntu 6.06 LTS | 29 (9) | 145 (47) | 20 (n/a) |
Ubuntu 6.06 reduced** | n/a | 74 (28) | 11 (2) |
SLED10 | 23 (5) | 159 (50) | 27 (6) |
SLED10 reduced** | n/a | 123 (44) | 20 (6) |
OS X 10.4 | 10 (3) | 60 (18) | 16 (3) |
1: vulnerabilities that were disclosed prior to the software release. In most cases a patch was available, but had to be applied by the user after installation
* high severity rating assigned by the National Vulnerabilitiy Database of the National Institute of Standards and Technology
** Distribution tweaked to mimick the functionality of Windows by stripping bundled components such as OpenOffice and development tools