CloudFlare burnt by network routing mishap

By on
CloudFlare burnt by network routing mishap

Maintenance work triggers downtime.

CloudFlare has suffered a service outage after its engineers erred in routing optimisation work during a data centre maintenance window in Hong Kong.

The CloudFlare service, which optimises the speed of websites and mitigates attacks levelled against them, was offline for up to 15 minutes.

The outage impacted about 75 percent of all traffic to the CloudFlare network.

Traffic to Hong Kong was intended to be diverted to data centres in Singapore or Japan during the maintenance window, but a routing configuration error meant this did not occur.

"At some point, the outbound routes were entered into the inbound interface. The outbound routes describe our entire net range so the net effect was the router in Hong Kong was announcing that it was the correct place to send all traffic bound for CloudFlare's IP space," the company said in a post-incident report.

"Our upstream provider trusts our routes so, via BGP, they were quickly relayed throughout their network and to their upstreams."

The company said it realised the error and re-announced the corrected routes.

It said it would initiate changes to prevent a similar occurrence in future.

"We are implementing systems to run all routing changes through a verification layer that double check before any routes are announced," it said.

"We are also talking with all our upstream providers to enable additional checks on their networks that do not automatically propagate major routing changes without confirmation."

The quick post-incident report won praise from customers on Twitter.

CloudFlare shot to fame recently after it was revealed they had helped mitigate attacks against LulzSec.

It is not the first time a procedural routing error has led to service outages. ISP Dodo effectively "advertised the entire internet" - made up of approximately 400,000 routing prefixes - in February, which was accepted and propagated by Telstra.

That error caused a widespread 35-minute internet service outage. Steps were also taken in that instance to prevent a similar occurrence.

Copyright © iTnews.com.au . All rights reserved.
Tags:

Most Read Articles

You must be a registered member of iTnews to post a comment.
| Register

Poll

New Windows 10 users, are you upgrading from...
Windows 8
Windows 7
Windows XP
Another operating system
Windows Vista
How should the costs of Australia's piracy scheme be split?
Rights holders should foot the whole bill
50/50
ISPs should foot the whole bill
Government should chip in a bit
Other
View poll archive

Whitepapers from our sponsors

What will the stadium of the future look like?
What will the stadium of the future look like?
New technology adoption is pushing enterprise networks to breaking point
New technology adoption is pushing enterprise networks to breaking point
Gartner names IBM a 'Leader' for Disaster Recovery as a Service
Gartner names IBM a 'Leader' for Disaster Recovery as a Service
The next era of business continuity: Are you ready for an always-on world?
The next era of business continuity: Are you ready for an always-on world?

Log In

Username:
Password:
|  Forgot your password?