Optus traces mobile outage to database software update glitch

Optus has traced the cause of a mobile outage that impacted 220,000 customers earlier this week to a software update glitch that impaired one of its four nationally distributed subscriber databases.

An Optus spokesperson told iTnews that its technology partner, Ericsson, was carrying out routine software updates on the databases, which store subscriber information that is used to register mobile devices on its network, when one of them developed a fault.

Optus operates four of the databases, known as Home Location Registers (HLR), across its network for load balancing and resiliency purposes.

The remaining three HLRs were not affected by the update snafu, explaining the unpredictable way that the outage manifested.

“Optus network engineers have been working with our partner, Ericsson, to remove a software fault from the mobile network," Optus' spokesperson said.

"The fault, which causes invalid data to be passed to an individual’s mobile device, prevented a number of customers from being able to connect to the Optus network.

“This software fault does not affect anything else on the customer’s device except its ability to connect to the Optus mobile network. It did not affect the customer’s ability to call triple zero,” the spokesperson added.

It’s understood that Optus did not expect the disruption and had no opportunity to warn customers of the outage.

Customers were able to diagnose whether their device was caught up by examining whether it displayed ‘No Service’ or ‘SOS’ – a condition often referred to SOS mode – without other cause.

At its peak, the glitch impacted 220,000 devices which, iTnews understands, was not isolated to mobile handsets but, in up to 50 percent cases, also other devices containing mobile SIMs, such as security cameras, tablet computers, watches and IoT equipment.

Optus advised customers that the fault could be fixed by restarting the affected devices and the number that remain impacted has gradually decreased since Optus first started issuing statements about the issue late Monday.

By Monday evening, the number of devices impacted stood at around 120,000, a 100,000 decrease from its peak of 220,000 which was reported early in the outage.

By around 5.30pm yesterday, Optus was reporting that the number of remaining devices without service was hovering around 35,000.

It’s understood that most of the remaining devices are no longer attended to or turned on, suggesting that number of impacted devices is highly unlikely ever to reach zero.

With a parliamentary inquiry into Australia’s triple zero services underway, in part triggered by a major Optus network outage last September linked to two fatalities, the outage was raised in senate estimates yesterday.

Under questioning from the senate’s environment and communications committee, the Australian Communications and Media Authority chair Nerida O'Loughlin said that the regulator would work with federal government’s new triple zero guardian to see whether the incident warranted investigation for breaches of the regulatory framework.

O’Loughlin told the committee that, given the incident had only taken place the day before the hearing, the regulator had too little information to go on to decide whether to investigate.

“There's certainly nothing in what I heard [about the incident] which causes me to immediately think there has been compliance issues, but there will certainly be further enquiries before we finalise any view around that,” O’Loughlin said.

The triple zero inquiry committee was expected to report its findings this week.

However, the committee’s chair Senator Sarah Hanson-Young has confirmed that she has requested that the date be pushed back to March 30.

It’s understood that the committee is hoping to hold more hearings before finalising its report.