Telstra broke its network with undocumented time fix

Telstra has revealed twin causes behind its nationwide mobile outage earlier this month, comprising an unrecorded configuration change on a Symmetricom SSU-2000 Network Time Synchronisation System unit, combined with a years-old outstanding firmware update.

The explanation, provided at a senate hearing on Friday, introduces additional complexity to what was previously understood to be the root cause.

The telco previously attributed the problem to a node in its mobile network used for timekeeping.

Mobile networks rely on precise timekeeping to function, and that timing infrastructure typically spans several layers - "stratums" - between the network core and the edge. 

Like other telcos, Telstra uses the hierarchical Network Time Protocol (NTP) which groups references from Stratum 0 which have atomic clocks, in 15 steps with decreasing authority.

In that model, Stratum 3 servers see Stratum 2 ones as authoritative; Stratum 1 are even higher up, and take their reference from Stratum 0 devices with atomic clocks using a GNSS card.

Telstra's standard network design has the SSU-2200 at Stratum 3 in the network time hierarchy. However, the carrier gave evidence that the unit developed a problem connecting with Stratum 2 servers last October.

CEO Vicki Brady said that the unspecified issue with a unit in Melbourne meant it failed to take time from its usual Stratum 2 time source.

To fix this, at some point in the past six months, engineers made an undocumented change to this specific unit, promoting it to Stratum 1 so that it used its own onboard GPS card as the authority on time and date instead of querying the network.

"This particular server, when it comes to timing and synchronisation, is designed [such] that it reaches out to another server in the network. It is not the authority on time and date. It sits at Stratum 3 in the network, [and] it goes to Stratum 2 to get that [authority]," Brady said.

"There was an issue that appeared where it wasn’t doing that successfully so it was then redesigned to be at Stratum 1 so it used the GPS card that was sitting in the server as the authority on time and date.

"That was the design change that was made to this particular server, whereas others [in Sydney and Perth] continued to operate in the design as standard."

Telstra's problem was that the GPS card was unpatched – because it was not previously being used. The undocumented workaround changed this.

"Our investigation so far has shown that ... it’s not a case of [technical teams] just ignoring that update. They did consider [implementing] it. The way this was designed and the way we were using this particular server and this card, this update was affecting a feature that we were not using at that time.

"What has happened, however, is we made a design change on this particular server to address another issue, which meant that became relevant.

"We didn’t document that design change and we hadn’t implemented then the software update," Brady told the hearing.

The carrier conceded in the hearing that it had received two reminders to apply the software patch the GPS card: one was sent in 2020 and another in 2022.

Timekeeping issue explained

Publicly available documentation for the SSU-2000 suggests the platform is around 24 years old.

Devices of that age contain GPS cards that use a legacy time counter which resets to zero every 1024 weeks which is just under 20 years.

Once the SSU 2000 was elevated to a Stratum 1 NTP server and began trusting its own GPS card as the authority, that rollover reportedly reset the network's internal clock to November 2006.

Telstra customers posted on social media that their devices displayed 2006 instead of 2026, which is approximately 20 years back in time, once they re-registered on the mobile network.

Devices presenting the wrong time could fail access negotiations using transport layer security (TLS) for authenticated and secured links, due to large date difference.

Telstra operates NTP nodes across Australia, including Sydney and Perth.

Compensation claims

Telstra chief financial officer Michael Ackland today told the hearing that the carrier had received around 8000 requests for compensation for the outage. At the time of the hearing, he said that the company had already paid out around $100,000 compensation.

Larger claims were still being processed, he said.

Brady said that the carrier had engaged Technology Audit Partners to carry out a full investigation into the incident.