NVIDIA calls the kettle black

COMMENTARY: Graphics chip maker NVIDIA called on the graphics card industry to halt the common practice of optimising graphics drivers for benchmarks only, and only to optimise drivers where benefits are seen in both computer games and benchmarks.

'In the past we've made clear mistakes that have caused us grief,' said Nick Triantos, NVIDIA's Director of Software Engineering, at last week's NVIDIA Editor's Day in San Francisco. He referred to the incident earlier in the year (http://www.pcauthority.com.au/Index.asp?PageType=ArticleDetail&CatID=1&ID=13334) where NVIDIA were caught cheating via driver optimisations in FutureMark Corporation's 3Dmark2003 benchmark.

3D benchmarks are a way of computing a graphics card's ability to perform by rendering complex 3D scenes, with an emphasis on lighting, detail rendering and other important graphical elements. Each test returns a score that can be used to compare a new graphics card against an older one, or more importantly against the competitor's products.

NVIDIA's driver cheating for 3Dmark2003 only came to light in a developer version of the benchmark where you can pause and mouse-look around the rendered landscape. Doing this revealed that (in the most obvious benchmark abuse from NVIDIA), that a full sky-box had not been rendered. This means that the far-off box containing the sky textures was not fully rendered where the 'camera' could not see it, saving the graphics card from processing erroneous visual data that would be immediately discarded after processing and therefore freeing up more of the graphics card's resources. While the final graphical output appeared identical to the user, it resulted in a falsely higher 3Dmark2003 score.

As an optimisation, it's not a bad one provided all gaming engine code was badly written to process invisible or out-of-frame visual data, but the sticking points are that not all games are written that way, and that NVIDIA's drivers would only activate the sky-box clipping mode when they detected that a pre-scripted benchmarking run of 3Dmark2003 was being performed.

Therefore the optimisation was simply there to falsely inflate 3Dmark2003 - and only 3Dmark2003 - scores. Optimisations other than this for 3Dmark2003 were not as obvious, but many were used by both NVIDIA and ATI to falsely inflate the 3Dmark2003 benchmark result.

While optimisations like the sky-box clipping can, and are, made in a game-by-game basis, doing so just for a benchmark is cheating, no matter how many times you plead that it isn't.

Since being caught with their hand in the cookie-jar, NVIDIA has turned the tables and are now positioning themselves as the arbiter of all things just and fair with 3D card benchmarking, headlined by a new driver optimisation process that has a basis in some simple (and some might argue, obvious) rules.

Firstly, NVIDIA driver optimisations must be simple and straightforward. Secondly, an optimisation must produce the correct image. Thirdly (and most importantly we feel), any optimisation must do more than just accelerate a benchmark. Fourthly, an optimisation must not contain pre-computed states.

This is followed by a multi-step optimisation validation process which checks and rechecks new drivers against the guidelines. Triantos was bullish about this system, claiming that 'Only NVIDIA has this type of process,' and that 'no one else has ethics on what we put in our optimisations.'

In a FAQ handed out to press, the 3Dmark2003 cheating issue was addressed thus:

'Some of the visual anomalies turned out to be bugs; others were aggressive optimizations. We make no secret of the fact that we were doing optimizations for 3Dmark2003. We have always maintained we would, but we feel it is an unfortunate waste of time as it doesn't benefit games. We would certainly prefer to spend all of our energy optimizing games people play, as it ultimately benefits the users. Unfortunately, it is safe to say that all GPU manufacturers tune for benchmarks.'

It's worth reading those last two sentences again, as they're a strict counterpoint to NVIDIA's new anti benchmark-optimisation stance.

This is interesting, considering NVIDIA's head of Windows Software (read: drivers) also admitted to me that 'People are getting so good at sniffing out our optimisations that we've had to obfuscate them in the system registry with alphanumeric keys'.

Furthermore, when I pointed out to Triantos that a few of the slides in the presentation seemed to indicate that optimisations were made first for benchmarks, then checked to see if they benefited games, instead of being made in general for games and benchmarks, he responded: 'No, not at all. We give what the customer wants. If they want UT2003 (Unreal Tournament 2003) to go faster then we'll optimise for it. If the customer wants 3Dmark2003 we'll optimise for that.'

'But only if it also benefits games,' he quickly added.