Google explains Apps data centre failure

Powered by SC Magazine
 

Honest assessment of February outage.

Google has published a post-mortem of an incident in February in which Google Apps went down for over two hours.

All Google App Engine applications were "degraded" from 7:48am to 10:09am PST on 24 February after a power failure at the company's main data centre, the firm said.

About 25 percent of the servers failed within five minutes owing to a delay in back-up power generation. Google's message boards started showing questions from users almost immediately.

"By this time, our primary on-call engineer had determined that App Engine is down," the report said.

"The on-call engineer, according to procedure, paged our product managers and engineering leads to handle communicating the outage to users. A few minutes later, the first post from the App Engine team about this outage is made on the external group."

There was confusion about the instructions for switching to a back-up data centre and the decision-maker for the crossover could not be found. The team then received data suggesting that the data centre was recovering and that a changeover was not neccesary.

However, the data turned out to be inaccurate and this extended the outage considerably. By the time the move to the backup servers had been made, Google Apps had been down for more than two hours.

The report found that Google had not developed plans for a partial data centre failure, nor for determining whether the data centre was able to continue running on such a reduced server count.

The company will now hold regular drills for failure, with a wider spectrum of possible situations, and a bi-monthly audit of all operations documents.

Google claimed that a similar failure today would cause a service slowdown for a maximum of 20 minutes with the new procedures, rather than a complete outage.

Copyright ©v3.co.uk


Google explains Apps data centre failure
 
 
 
Top Stories
Beyond ACORN: Cracking the infosec skills nut
[Blog post] Could the Government's cybercrime focus be a catalyst for change?
 
The iTnews Benchmark Awards
Meet the best of the best.
 
Telstra hands over copper, HFC in new $11bn NBN deal
Value of 2011 deal remains intact.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...
Latest Comments
Polls
Who do you trust most to protect your private data?







   |   View results
Your bank
  38%
 
Your insurance company
  4%
 
A technology company (Google, Facebook et al)
  8%
 
Your telco, ISP or utility
  8%
 
A retailer (Coles, Woolworths et al)
  3%
 
A Federal Government agency (ATO, Centrelink etc)
  19%
 
An Australian law enforcement agency (AFP, ASIO et al)
  14%
 
A State Government agency (Health dept, etc)
  6%
TOTAL VOTES: 1869

Vote
Do you support the abolition of the Office of the Information Commissioner?