Analysis: Broadband Expert speed test relies on trust

A controversial report that pegged average broadband speeds on Telstra's NextG network below that of Vodafone included data where the identity of the user's network couldn't be verified.

The British company behind the report, Broadband Expert, has admitted it "wouldn't know which [broadband] provider" some customers used during their speed test, unless it asked them to first manually input the provider's name.

Using Broadband Expert's speed test tool, users are asked to manually select their connection type, the name of their provider, the speed they are supposed to receive under their broadband plan and their geographic state from a series of drop-down menus.

The fact that some of this user-input information cannot be verified by Broadband Expert's speedtest software casts doubt over the report's accuracy. 

The results in the report rely on users to input the correct information about their connection.

Broadband Expert commercial director Rob Webber told iTnews that host information [the ISP's name] was "generally" captured automatically when a user took a speed test.

"I mentioned that the ISP name is generally included in the host information, it is not always included," he later clarified.

"If we didn't ask for the provider name [upfront] we would have to exclude a percentage of the results simply because we wouldn't know which provider they were attached to.

"So we felt it made more sense to ask the question [upfront]."

Tests by iTnews revealed the ease with which users could ‘game' the system by lying about the identity of their ISP.

For example, it was possible to select the name of any telco and any peak speed before conducting a test, or to test a wired connection but identify it as a 3G service.

Webber said that in the "rare instances where the host details and the ISP selected by the user do not match, the results are excluded."

But that matching exercise wasn't possible for everyone included in the report.

iTnews does not allege the results were 'gamed'; only that the system setup made gaming possible.

One test a day?

Another potential cause for concern was the low sample size of test results that make up the report.

The report found Vodafone 3G users experienced the fastest average line speeds of any mobile telco - 2.2 Mbps. This was based on 238 speed tests.

Telstra came in a distant fourth with users experiencing an average 1.8 Mbps line speed. This was based on 476 speed tests.

The report did not distinguish between the number of tests and the number of individuals conducting them.

"If multiple tests are carried out by one person, these stats are not removed," Webber said.

Webber speculated that the "nature of mobile broadband meant that one user could be running a number of tests from multiple locations and the results could be different every time, so we choose to include these".

"Due to the amount of tests in the results, it [was] unlikely one person running multiple tests would have a huge impact on the overall results," he said.

There were 184 days in the test period of July to December 2009. That's an average of 1.3 speed tests per day for Vodafone; and 2.6 tests per day by Telstra users.

Put simply, a single Vodafone user that conducted one daily speed test on their connection would account for 184 of the 238 speed tests in the test period.

Webber admitted the numbers were low but stood by the sample size, believing it was "reasonable".

"We'll have a much larger sample next time we update our stats," he said.

He also believed it was "highly unlikely someone would want to run a speed test on our site every day and lie about their ISP and type of connection".

"During the testing period, no one was aware that we were going to publish these stats by ISP averages, so your suggestion that someone at Vodafone was deliberately trying to improve their speed averages is not logical," he said.

iTnews did not suggest Vodafone had taken any action to affect the outcome of the report.

Hopeful

Despite these concerns, Webber hoped the report would "help consumers by uncovering the best and worst performers from an independent point of view."

"Up to speeds have never been a good indicator and with recent industry regulations, some ISPs are now not quoting speeds at all," he said.

"We believe the ISP's should quote the average speeds that they're existing users achieve at the point of sign."