CloudFlare burnt by network routing mishap


Maintenance work triggers downtime.

CloudFlare has suffered a service outage after its engineers erred in routing optimisation work during a data centre maintenance window in Hong Kong.

The CloudFlare service, which optimises the speed of websites and mitigates attacks levelled against them, was offline for up to 15 minutes.

The outage impacted about 75 percent of all traffic to the CloudFlare network.

Traffic to Hong Kong was intended to be diverted to data centres in Singapore or Japan during the maintenance window, but a routing configuration error meant this did not occur.

"At some point, the outbound routes were entered into the inbound interface. The outbound routes describe our entire net range so the net effect was the router in Hong Kong was announcing that it was the correct place to send all traffic bound for CloudFlare's IP space," the company said in a post-incident report.

"Our upstream provider trusts our routes so, via BGP, they were quickly relayed throughout their network and to their upstreams."

The company said it realised the error and re-announced the corrected routes.

It said it would initiate changes to prevent a similar occurrence in future.

"We are implementing systems to run all routing changes through a verification layer that double check before any routes are announced," it said.

"We are also talking with all our upstream providers to enable additional checks on their networks that do not automatically propagate major routing changes without confirmation."

The quick post-incident report won praise from customers on Twitter.

CloudFlare shot to fame recently after it was revealed they had helped mitigate attacks against LulzSec.

It is not the first time a procedural routing error has led to service outages. ISP Dodo effectively "advertised the entire internet" - made up of approximately 400,000 routing prefixes - in February, which was accepted and propagated by Telstra.

That error caused a widespread 35-minute internet service outage. Steps were also taken in that instance to prevent a similar occurrence.

Copyright © . All rights reserved.

CloudFlare burnt by network routing mishap
Top Stories
Myer CIO named retailer's new chief executive
Richard Umbers to lead data-driven retail strategy.
Empty terminals and mountains of data
Qantas CIO Luc Hennekens says no-one is safe from digital disruption.
BoQ takes $10m hit on Salesforce CRM
Regulatory hurdles end cloud pilot.
Sign up to receive iTnews email bulletins
Latest Comments
Who do you trust most to protect your private data?

   |   View results
Your bank
Your insurance company
A technology company (Google, Facebook et al)
Your telco, ISP or utility
A retailer (Coles, Woolworths et al)
A Federal Government agency (ATO, Centrelink etc)
An Australian law enforcement agency (AFP, ASIO et al)
A State Government agency (Health dept, etc)

Do you support the abolition of the Office of the Information Commissioner?

   |   View results
I support shutting down the OAIC.
I DON'T support shutting the OAIC.