iTnews
  • Home
  • News
  • Technology
  • Telco/ISP

Analysis: Broadband Expert speed test relies on trust

By Ry Crozier on Apr 15, 2010 6:18AM
Analysis: Broadband Expert speed test relies on trust

Can it be gamed?

A controversial report that pegged average broadband speeds on Telstra's NextG network below that of Vodafone included data where the identity of the user's network couldn't be verified.

The British company behind the report, Broadband Expert, has admitted it "wouldn't know which [broadband] provider" some customers used during their speed test, unless it asked them to first manually input the provider's name.

Using Broadband Expert's speed test tool, users are asked to manually select their connection type, the name of their provider, the speed they are supposed to receive under their broadband plan and their geographic state from a series of drop-down menus.

The fact that some of this user-input information cannot be verified by Broadband Expert's speedtest software casts doubt over the report's accuracy. 

The results in the report rely on users to input the correct information about their connection.

Broadband Expert commercial director Rob Webber told iTnews that host information [the ISP's name] was "generally" captured automatically when a user took a speed test.

"I mentioned that the ISP name is generally included in the host information, it is not always included," he later clarified.

"If we didn't ask for the provider name [upfront] we would have to exclude a percentage of the results simply because we wouldn't know which provider they were attached to.

"So we felt it made more sense to ask the question [upfront]."

Tests by iTnews revealed the ease with which users could ‘game' the system by lying about the identity of their ISP.

For example, it was possible to select the name of any telco and any peak speed before conducting a test, or to test a wired connection but identify it as a 3G service.

Webber said that in the "rare instances where the host details and the ISP selected by the user do not match, the results are excluded."

But that matching exercise wasn't possible for everyone included in the report.

iTnews does not allege the results were 'gamed'; only that the system setup made gaming possible.

One test a day?

Another potential cause for concern was the low sample size of test results that make up the report.

The report found Vodafone 3G users experienced the fastest average line speeds of any mobile telco - 2.2 Mbps. This was based on 238 speed tests.

Telstra came in a distant fourth with users experiencing an average 1.8 Mbps line speed. This was based on 476 speed tests.

The report did not distinguish between the number of tests and the number of individuals conducting them.

"If multiple tests are carried out by one person, these stats are not removed," Webber said.

Webber speculated that the "nature of mobile broadband meant that one user could be running a number of tests from multiple locations and the results could be different every time, so we choose to include these".

"Due to the amount of tests in the results, it [was] unlikely one person running multiple tests would have a huge impact on the overall results," he said.

There were 184 days in the test period of July to December 2009. That's an average of 1.3 speed tests per day for Vodafone; and 2.6 tests per day by Telstra users.

Put simply, a single Vodafone user that conducted one daily speed test on their connection would account for 184 of the 238 speed tests in the test period.

Webber admitted the numbers were low but stood by the sample size, believing it was "reasonable".

"We'll have a much larger sample next time we update our stats," he said.

He also believed it was "highly unlikely someone would want to run a speed test on our site every day and lie about their ISP and type of connection".

"During the testing period, no one was aware that we were going to publish these stats by ISP averages, so your suggestion that someone at Vodafone was deliberately trying to improve their speed averages is not logical," he said.

iTnews did not suggest Vodafone had taken any action to affect the outcome of the report.

Hopeful

Despite these concerns, Webber hoped the report would "help consumers by uncovering the best and worst performers from an independent point of view."

"Up to speeds have never been a good indicator and with recent industry regulations, some ISPs are now not quoting speeds at all," he said.

"We believe the ISP's should quote the average speeds that they're existing users achieve at the point of sign."

Got a news tip for our journalists? Share it with us anonymously here.
Tags:
3gbroadbanddownloadexpertleaguelinereportspeedtabletelco/isptelstratestvodafone

Partner Content

The Great Resignation has intensified insider security threats
Promoted Content The Great Resignation has intensified insider security threats
Why Genworth Australia embraced low-code software development
Promoted Content Why Genworth Australia embraced low-code software development
Security "mindset shift" needed to protect organisations
Promoted Content Security "mindset shift" needed to protect organisations
Security: Understanding the fundamentals of governance, risk & compliance
Promoted Content Security: Understanding the fundamentals of governance, risk & compliance

Sponsored Whitepapers

Extracting the value of data using Unified Observability
Extracting the value of data using Unified Observability
Planning before the breach: You can’t protect what you can’t see
Planning before the breach: You can’t protect what you can’t see
Beyond FTP: Securing and Managing File Transfers
Beyond FTP: Securing and Managing File Transfers
NextGen Security Operations: A Roadmap for the Future
NextGen Security Operations: A Roadmap for the Future
Video: Watch Juniper talk about its Aston Martin partnership
Video: Watch Juniper talk about its Aston Martin partnership

Events

  • Micro Focus Information Management & Governance (IM&G) Forum 2022
  • CRN Channel Meets: CyberSecurity Live Event
  • IoT Insights: Secure By Design for manufacturing
  • Cyber Security for Government Summit
By Ry Crozier
Apr 15 2010
6:18AM
0 Comments

Related Articles

  • TPG Telecom puts fresh targets on IT simplification
  • NBN Co says TPG tie-up could help Telstra sidestep spectrum limits
  • Telstra, Optus, Vodafone ready multi-factor authentication
  • Northern Rivers locals still angry with telcos over flood response
Share on Twitter Share on Facebook Share on LinkedIn Share on Whatsapp Email A Friend

Most Read Articles

Qantas calls time on IBM, Fujitsu in tech modernisation

Qantas calls time on IBM, Fujitsu in tech modernisation

Service NSW hits digital services goal two years early

Service NSW hits digital services goal two years early

NBN Co taking orders for 'non-premises' connections

NBN Co taking orders for 'non-premises' connections

Australian scientists build world's first quantum computer IC

Australian scientists build world's first quantum computer IC

Digital Nation

Crypto experts optimistic about future of Bitcoin: Block
Crypto experts optimistic about future of Bitcoin: Block
Integrity, ethics and board decisions in the digital age
Integrity, ethics and board decisions in the digital age
The security threat of quantum computing
The security threat of quantum computing
COVER STORY: Operationalising net zero through the power of IoT
COVER STORY: Operationalising net zero through the power of IoT
IBM global chief data officer on the rise of the number crunchers
IBM global chief data officer on the rise of the number crunchers
All rights reserved. This material may not be published, broadcast, rewritten or redistributed in any form without prior authorisation.
Your use of this website constitutes acceptance of nextmedia's Privacy Policy and Terms & Conditions.