Dodo cops blame for national internet outages

 

Telstra routers downed for 35 minutes.

Dodo has revealed a "minor hardware issue" was behind a Telstra outage that impacted multiple service providers and internet services nationwide.

The outage, which lasted approximately 35 minutes this afternoon, impacted an international link used by major service providers Telstra, Optus and iiNet for ADSL, cable and 3G data services.

Telstra said it had solved the issue but was still investigating what caused it.

Network engineers took to web forums with suspicions that a routing issue originating in Dodo's network had caused the issue.

Industry sources said the network issue came as a result of Dodo mistakenly issuing new IP route addresses from its system that confused Telstra's systems and caused blackouts on the AS1221 upstream router.

A memo purportedly from Optus, and posted to Whirlpool, indicated Dodo had "decided to advertise all the global routes it knows to Telstra and for some unknown reason Telstra then accepted these as 'best path' which in effect meant ALL traffic originating from the Telstra network would try and route traffic via Dodo".

Dodo chief executive Larry Kestelman confirmed the outage could be sourced to Dodo as the result of a "minor hardware issue".

"Telstra and us are investigating why this had such an overstated effect," he told iTnews.

A statement circulated to media last night by Kestelman clarified it was a "hardware issue with a Cisco border router".

It remains unclear, however, why the new IP routes issued by Dodo's network were accepted by Telstra routers.

Though the problem has been fixed in the meantime, it is expected engineers at both telcos will spend the next 48 hours investigating the outage before concluding the actual cause in Dodo's routers.

Widespread effect

Users of multiple ISPs flooded Twitter and broadband forum Whirlpool to seek information on the outage.

A post on the Whirlpool forums of a partial notification by wholesale service provider Vocus appeared to point to a routing problem.

"Vocus network operations staff have been made aware of a routing incident that has had significant impact on some domestic networks," the post read.

"This incident has impacted most of the Telstra domestic peering points and caused a large amount of route churn and some sub-optimal routing. 

"Engineers from the affected network are working to restore traffic back to its original paths but this may take some time."

Other network operators and ISPs reported downstream issues that appeared to point back to the issue at a specific Telstra router.

"Looks like a major outage in Telstra network. Appears nation wide," Spectrum Networks tweeted, saying its network was operating as normal. 

ISP iiNet also pointed to an "upstream routing issue" as the cause of a nationwide fault it described as having a "major" impact on services. The fault had since been resolved, engineers said.

An iiNet representative said it was "intentionally avoiding Telstras [sic] network where possible at the moment".

"You may see higher than usual latency to hosts in Australia (other than via the IX [internet exchange] points)," Gavin Tweedie posted to Whirlpool.

"We'll slowly bring back up the domestic Telstra capacity during the course of this afternoon if performance is acceptable."

TPG also noted issues it attributed to a "3rd party carrier fault".

AusWeb said the outage was "causing major routing table changes throughout the network."

The outage affected the websites of banks including Westpac. "There is a @Telstra issue affecting westpac.com.au & other sites," the bank tweeted.

"We're looking into to ensure it's resolved as soon as possible."

NAB reported similar issues on their Twitter feed.

Copyright © iTnews.com.au . All rights reserved.


Dodo cops blame for national internet outages
 
 
 
Top Stories
First look: Microsoft Outlook for iOS
[Update] Office productivity suite for iOS completed with Outlook.
 
NewSat defaults on $26m in overdue Lockheed payments
Jabiru-1 satellite build hits further hurdles.
 
IBM denies plans to cut 112k jobs
But admits to further restructuring.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...
Latest articles on BIT Latest Articles from BIT
Microsoft Outlook is now on iPhone and iPad: why could this be useful?
Jan 30, 2015
Microsoft today released Office for Android and Outlook for iOS - complementing the other Office ...
Franchisees, here's something you should know about
Jan 23, 2015
You need to know the Code if you are a franchisee or franchisor as the penalties are significant.
Xero users rejoice! Quoting has finally arrived
Jan 23, 2015
It has taken years, but Xero has at last added integrated quoting to its online accounting software.
You can now get a no-contract wi-fi tablet from Telstra
Jan 17, 2015
Telstra has began selling wi-fi tablets out of contract without paying extra for cellular ...
Get your business ready for 2015: mobile payments
Jan 2, 2015
These handy apps from MYOB, Xero and others can reduce your administrative load and improve ...
Latest Comments
Polls
Who do you trust most to protect your private data?







   |   View results
Your bank
  36%
 
Your insurance company
  5%
 
A technology company (Google, Facebook et al)
  9%
 
Your telco, ISP or utility
  8%
 
A retailer (Coles, Woolworths et al)
  4%
 
A Federal Government agency (ATO, Centrelink etc)
  18%
 
An Australian law enforcement agency (AFP, ASIO et al)
  14%
 
A State Government agency (Health dept, etc)
  7%
TOTAL VOTES: 3112

Vote
Do you support the abolition of the Office of the Information Commissioner?

   |   View results
I support shutting down the OAIC.
  27%
 
I DON'T support shutting the OAIC.
  73%
TOTAL VOTES: 994

Vote