Telstra routing flub affects hundreds of networks worldwide

A misconfiguration by Telstra is causing problems for hundreds of network providers worldwide, sparking accusations of border gateway protocol (BGP) hijacking by the Aussie telco.

BGP is used to determine traffic routes between internet connected network providers based on agreements and policies.

The routing protocol is prone to misconfigurations, at times with major consequences as traffic is sent by large networks to smaller ones that don't have the capacity to handle the volume, for example.

Swiss privacy-oriented email provider ProtonMail complained on social media that Telstra was announcing its 185.70.40.0/24 subnet with authorisation, which meant that traffic to that network took a detour via Melbourne.

This caused delays for incoming and outgoing messages, which ProtonMail had to queue up and route via slower secondary paths.

Several other providers fell victim to erroneously advertised routes for their networks as well, although some have managed to contact Telstra and have the problem resolved.

Some Quad9 prefixes are currently being announced without permission by @Telstra causing outages for some end users. Other firms report similar hijacking of routes. Investigation and mitigation is ongoing. Please ensure 149.112.112.112 is in your resolver config to avoid faults.

— Quad9 (@Quad9DNS) September 29, 2020

Packet's Sydney facility that hosts Google's local region was also impacted for three hours.

Reverse proxy and content delivery network Cloudflare's BGP routes page showed Telstra's Autonomous System 1221 this morning as announcing 102 networks on the internet, with invalid route origin authorisation.

A Telstra spokesperson told iTnews that the issues were the result of a "technical error" in which "a number of internet prefixes were incorrectly advertised as Telstra’s."

"This meant some internet traffic may have been routed to Telstra incorrectly, although our systems indicate negligible traffic was actually received," the spokesperson said.

"The overnight change has now been reversed.

"We’re working with ProtonMail and other impacted customers to ensure a full recovery of their services."

Outside of this morning's specific issue, a collective effort called Resource Public Key Infrastructure (RPKI) is underway among internet connected networks to prevent similar BGP configurations mistakes and ensuing outages.

However, RPKI only works if networks filter on the validity of cryptographically signed route objects, and drop invalid ones without the appropriate authorisation.