CloudFlare burnt by network routing mishap

 

Maintenance work triggers downtime.

CloudFlare has suffered a service outage after its engineers erred in routing optimisation work during a data centre maintenance window in Hong Kong.

The CloudFlare service, which optimises the speed of websites and mitigates attacks levelled against them, was offline for up to 15 minutes.

The outage impacted about 75 percent of all traffic to the CloudFlare network.

Traffic to Hong Kong was intended to be diverted to data centres in Singapore or Japan during the maintenance window, but a routing configuration error meant this did not occur.

"At some point, the outbound routes were entered into the inbound interface. The outbound routes describe our entire net range so the net effect was the router in Hong Kong was announcing that it was the correct place to send all traffic bound for CloudFlare's IP space," the company said in a post-incident report.

"Our upstream provider trusts our routes so, via BGP, they were quickly relayed throughout their network and to their upstreams."

The company said it realised the error and re-announced the corrected routes.

It said it would initiate changes to prevent a similar occurrence in future.

"We are implementing systems to run all routing changes through a verification layer that double check before any routes are announced," it said.

"We are also talking with all our upstream providers to enable additional checks on their networks that do not automatically propagate major routing changes without confirmation."

The quick post-incident report won praise from customers on Twitter.

CloudFlare shot to fame recently after it was revealed they had helped mitigate attacks against LulzSec.

It is not the first time a procedural routing error has led to service outages. ISP Dodo effectively "advertised the entire internet" - made up of approximately 400,000 routing prefixes - in February, which was accepted and propagated by Telstra.

That error caused a widespread 35-minute internet service outage. Steps were also taken in that instance to prevent a similar occurrence.

Copyright © iTnews.com.au . All rights reserved.


CloudFlare burnt by network routing mishap
 
 
 
Top Stories
iiNet facing new copyright battle with Hollywood
Fighting to protect customer details.
 
The CISO’s dilemma: Do you trust your partner’s partner?
[Blog post] How far down the chain do you check?
 
Microsoft confirms Australian Azure launch
Available from next week.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...
Latest Comments
Polls
In which area is your IT shop hiring the most staff?




   |   View results
IT security and risk
  25%
 
Sourcing and strategy
  12%
 
IT infrastructure (servers, storage, networking)
  23%
 
End user computing (desktops, mobiles, apps)
  15%
 
Software development
  26%
TOTAL VOTES: 302

Vote
Would your InfoSec team be prepared to share threat data with the Australian Government?

   |   View results
Yes
  59%
 
No
  41%
TOTAL VOTES: 114

Vote