Optus fast-tracks network operations insourcing from Nokia

Optus has revealed that it will accelerate plans bring network functions currently outsourced to Nokia back in-house in response to September's triple zero outage on its network.

Chief executive Stephen Rue revealed the move as part of his opening remarks to a parliamentary inquiry into the incident today.

“In May the board approved a proposal to take back in-house network capabilities currently provided by Nokia. At the time the aim was to transition this work progressively to May 2027,” Rue said.

“I have now asked the network team to accelerate the move".

Rue also said that the carrier is taking on 150 additional new staff to fill roles in a “process centre of excellence” in Australia to accelerate the carrier’s ongoing business transformation and overhaul procedures across the business.

In a related paper submission [pdf]. Optus confirmed technical details of the outage cause.

It also confirmed that 605 individual numbers attempted to place calls but did not succeed. This could have included calls from automated systems, such as those that are placed automatically from publicly distributed defibrillator boxes when they are opened.

The carrier provided details on how a firewall upgrade at its Regency Park exchange in South Australia went wrong leading to the outage.

It explained that during the work, the carrier need to apply a “soft lock/hard lock” change to its session border gateway (SBG). The soft lock stops new calls but allows established calls to continue.

“Where a lock is applied, the correct process is to first redirect the traffic at the SBG before applying the lock.

“When the change above was implemented, voice traffic continued to be directed to the SBG but failed to be routed through the SBG as it had not been redirected. This meant that calls were not able to be processed through that part of the network,” the carrier wrote.

The company said that Nokia staff used an “outdated” manual when planning the upgrade.

“During the process of planning the upgrade, an outdated method of procedure document was selected for the implementation of the upgrade by Nokia personnel and was peer-reviewed by three Nokia personnel.

“The method of procedure did not include making changes in the evolved packet gateway (EPG) to ‘offload’ traffic from the SBG prior to implementing the change to lock the SBG,” it said.

Optus also revealed in its submission that some of its core network engineer staff were not present at a planning meeting ahead of the Regency Park upgrade.

Nokia has been contacted for comment.

Rue told the committee that the upgrade was one of only 10 points of failure in the carrier’s handling of the outage, which was later linked to at least three fatalities.

The committee focused its questions on when Optus knew about the fatalities and when it informed the government.

Greens Senator Sarah Hanson-Young was particularly critical of Rue's decision to prioritise contacting Singtel's board and Optus' senior leadership about the incident before notifying the government.

More to come.