CloudFlare burnt by network routing mishap

Maintenance work triggers downtime.

CloudFlare has suffered a service outage after its engineers erred in routing optimisation work during a data centre maintenance window in Hong Kong.

The CloudFlare service, which optimises the speed of websites and mitigates attacks levelled against them, was offline for up to 15 minutes.

The outage impacted about 75 percent of all traffic to the CloudFlare network.

Traffic to Hong Kong was intended to be diverted to data centres in Singapore or Japan during the maintenance window, but a routing configuration error meant this did not occur.

"At some point, the outbound routes were entered into the inbound interface. The outbound routes describe our entire net range so the net effect was the router in Hong Kong was announcing that it was the correct place to send all traffic bound for CloudFlare's IP space," the company said in a post-incident report.

"Our upstream provider trusts our routes so, via BGP, they were quickly relayed throughout their network and to their upstreams."

The company said it realised the error and re-announced the corrected routes.

It said it would initiate changes to prevent a similar occurrence in future.

"We are implementing systems to run all routing changes through a verification layer that double check before any routes are announced," it said.

"We are also talking with all our upstream providers to enable additional checks on their networks that do not automatically propagate major routing changes without confirmation."

The quick post-incident report won praise from customers on Twitter.

CloudFlare shot to fame recently after it was revealed they had helped mitigate attacks against LulzSec.

It is not the first time a procedural routing error has led to service outages. ISP Dodo effectively "advertised the entire internet" - made up of approximately 400,000 routing prefixes - in February, which was accepted and propagated by Telstra.

That error caused a widespread 35-minute internet service outage. Steps were also taken in that instance to prevent a similar occurrence.