Optus firewall upgrade behind 13-hour Triple Zero outage

Optus CEO Stephen Rue said that a “regular” firewall upgrade at 12.30am on Thursday caused Triple Zero calls on the network to fail for 13 hours, before it appears the change was rolled back.

Speaking at a media conference on Saturday afternoon, Rue confirmed an earlier report by the Australian Financial Review that a “firewall misconfiguration” was the root cause.

Rue said that the telco’s telemetry did not show anything abnormal after the “regular” change was made, although he did note that the carriage of emergency calls is not specifically monitored.

“When the upgrades and changes were implemented, initial testing and monitoring did not indicate there were any issues with calls connecting,” Rue said.

“Normal calls were connecting as they should, and call volumes at a national level did not raise any red flags. 

“Further, there were no alarms to alert us that some emergency calls were not making it through to emergency services.”

The telco was first made aware that there were problems just after 9am Thursday when a customer called the Optus contact centre directly. A second customer then made a similar report.

Inexplicably, neither of these calls was red-flagged or escalated.

“Two individual calls to the Optus contact centre …complained that the Triple Zero service was not working,” Rue said.

“Early reviews suggest that we had not handled these calls as would be expected.This information was not surfaced with the relevant escalation at that time.”

The telco remained largely in the dark until 1.30pm, when it was contacted “directly” by another customer, followed by SA Police at 1.50pm on Thursday, some 13 hours after the problems started.

“Once notified, we stopped the upgrade, restoring Triple Zero and began to confirm with relevant stakeholders such as police and other regulatory and government agencies and departments that we had experienced an outage impacting Triple Zero,” Rue said.

“We also confirmed that once we had the call details that we would be conducting welfare checks. 

“Welfare checks commenced later that evening and into the following day. 

“The delay in this timing was due to the complexity of pulling records from the network.”

Rue was grilled heavily as to why it took Optus until Friday afternoon - some 24 hours later - to publicly communicate that the incident had occurred.

Governments, including at the federal level, were also only notified of the extent of the incident on Friday evening, and this has not been well-received.

On the topic of delayed notification, Rue would only say that “we wanted to get the facts and the information together.”

Federal communications minister Anika Wells blasted the slow and late communication of the incident, and the continuing dearth of detail.

“We’ve all been taken aback by how long it took for us to find out the gravity of what’s happened here,” Wells said.

“Mr Rue would say they themselves are still gathering that information, but these are serious questions for them to answer, particularly: Why didn’t they know? Why didn’t alarms go off? Why did the testing fail? How long did it take them to work that out?

“Optus themselves don’t appear to know enough about what’s happened here. 

“I don’t think [it’s] good enough that 24 hours on, that they don’t have good answers.”

Wells noted the difference between this incident and the last time Optus had a major outage around Triple Zero call carriage, which occurred in 2023, and for which it was fined $12 million.

She said that in 2023, the incident came to public attention while it was still in-train.

“In 2023, it came to the public attention before it had been resolved, so there was national fear and scrutiny because there was a live question of whether you could call Triple Zero and get through,” Wells said.

“What has happened here is that we were advised late yesterday [Friday] afternoon that there had been an issue and that it had been rectified and we were advised at the same time. 

“I guess there is a question for Optus to answer there about whether they should have advised both emergency authorities on the ground in places like South Australia and Western Australia, and national authorities, when it was happening and when it was rectified.”

In response, Rue said that Optus "take full accountability for the technical failure and that we were unaware of this for a period of time, an unacceptable gap which I’ll ensure is fully investigated."

In any event, Wells said she found it “hard to put into words how this could have possibly been allowed to happen again, given what happened in 2023.”

“Many of the things that happened in this outage are failures to implement some of the recommendations [made after the earlier incident], including alerting the public and emergency services authorities,” she said.

“That is what I find to be particularly disappointing.”

Optus is running its own review of the incident, and will appoint an “independent person” in the coming days to oversee the process, Rue said.

The Australian Communications and Media Authority (ACMA) will conduct its own investigation, Wells confirmed.

She also suggested other investigations could come from state policing and health agencies, owing to fatalities in households that could not reach Triple Zero services during the outage.