Telstra blames network outage on DNS bungle, modem firmware bug

Telstra has revealed a fixed broadband outage last week was triggered by a bungled software update to a domain name server, which in turn triggered a previously unknown fault in some customer modems.

Last Thursday night, reports began flooding in from Telstra customers across the country who were unable to access their broadband services, with the telco slowly bringing its NBN and ADSL customers back onto its network from Monday morning.

In a conference call this morning, Telstra chief operations officer Kate McKenzie and group managing director of networks Mike Wright said a software update to a DNS server caused its fixed broadband services to go down.

The resulting service disruption interrupted the ‘heartbeat’ signal to customer modems, causing them to reset.

It was during this reset that a previously unknown firmware bug was triggered in some end user gateways, leaving them unable to reconnect to the network.

Telstra estimates that around 370,000 users, or roughly 10 percent of its fixed broadband customer base, was impacted by the initial DNS-related outage, with around 33,000 customers still experiencing issues last Friday morning.

In the vast bulk of cases, restoring a modem to its original factory settings was enough to resolve the issue, with half of all customers reconnected by the weekend, Telstra said.

Despite this, a number of customers McKenzie estimates as being in the “low single thousands” are currently still experiencing issues with modems, and will require a replacement modem.

Telstra said despite some reports, it did not attempt to roll out a firmware update to customer modems either on the night of the outage or since then.

However, Wright said the telco was working with vendors on fixing the firmware bug, and expects it to be patched in a firmware update once the software has been fully tested.

McKenzie claimed “customer feedback” had shown that users would not be interested in another free data day offer, as had happened after the carrier’s recent mobile outages. However, selected customers will receive $25 on their bill, to be judged on a case-by-case basis.

A separate issue unconnected to the fixed line outage caused some customers to lose mobile broadband internet access in NSW on the following Sunday morning. That issue was caused by a network failover as a result of a faulty card in the carrier’s wireless network equipment.

The latest outage is the fourth major network problem for Telstra this year. The three earlier outages - to the mobile network - forced Telstra to pledge $50 million to improve its network.

The telco has also been rocked by a series of smaller network problems, including to NBN services in NSW and to its domain name servers, while a third-party routing issue in March locked Telstra customers out of Amazon Web Services' Sydney data centre.