CloudFlare burnt by network routing mishap

 

Maintenance work triggers downtime.

CloudFlare has suffered a service outage after its engineers erred in routing optimisation work during a data centre maintenance window in Hong Kong.

The CloudFlare service, which optimises the speed of websites and mitigates attacks levelled against them, was offline for up to 15 minutes.

The outage impacted about 75 percent of all traffic to the CloudFlare network.

Traffic to Hong Kong was intended to be diverted to data centres in Singapore or Japan during the maintenance window, but a routing configuration error meant this did not occur.

"At some point, the outbound routes were entered into the inbound interface. The outbound routes describe our entire net range so the net effect was the router in Hong Kong was announcing that it was the correct place to send all traffic bound for CloudFlare's IP space," the company said in a post-incident report.

"Our upstream provider trusts our routes so, via BGP, they were quickly relayed throughout their network and to their upstreams."

The company said it realised the error and re-announced the corrected routes.

It said it would initiate changes to prevent a similar occurrence in future.

"We are implementing systems to run all routing changes through a verification layer that double check before any routes are announced," it said.

"We are also talking with all our upstream providers to enable additional checks on their networks that do not automatically propagate major routing changes without confirmation."

The quick post-incident report won praise from customers on Twitter.

CloudFlare shot to fame recently after it was revealed they had helped mitigate attacks against LulzSec.

It is not the first time a procedural routing error has led to service outages. ISP Dodo effectively "advertised the entire internet" - made up of approximately 400,000 routing prefixes - in February, which was accepted and propagated by Telstra.

That error caused a widespread 35-minute internet service outage. Steps were also taken in that instance to prevent a similar occurrence.

Copyright © iTnews.com.au . All rights reserved.


CloudFlare burnt by network routing mishap
 
 
 
Top Stories
Coalition's NBN cost-benefit study finds in favour of MTM
FTTP costs too much, would take too long.
 
Telcos finally briefed on data retention details
Update: AGD offers list of data to be stored.
 
Qld Health hires short-term CIO, CTO
Ray Brown leaves after five years at IT helm.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...
Latest Comments
Polls
Which is the most prevalent cyber attack method your organisation faces?




   |   View results
Phishing and social engineering
  67%
 
Advanced persistent threats
  3%
 
Unpatched or unsupported software vulnerabilities
  12%
 
Denial of service attacks
  7%
 
Insider threats
  11%
TOTAL VOTES: 557

Vote