Growing pains: Amazon EC2 suffers huge outage

The cloud compute operator said it was contacting a percentage of customers hosted in this zone whose services could not immediately be restored.

Customers were particularly critical of Amazon’s refusal to reveal what percentage of customers or precisely which availability zones were back online at any stage of the outage.

Amazon Web Services staff - usually quite chatty on social networks - did not respond to any questions or requests for help during the outage.

At the time of writing Amazon was yet to publish a post-mortem on what network event caused the system to malfunction.

The last message on Amazon's service status page said that once customers were "fully back up and running" the company would “post a detailed account of what happened, along with the corrective actions we are undertaking to ensure this doesn’t happen again.”

Thorsten von Eicken, CTO and Founder of Right Scale described it as the “worst outage in cloud computing history” and a “wake up call” to the industry – suggesting that the “ripple effects” from the initial failure through to other Amazon services “should not have happened.”

Public cloud customers would be wise to review backup and restore processes and ensure they straddle multiple geographies and services, von Eicken advised in a blog post.

Amazon customers had found themselves victims of vendor lock-in, he said, as its database-as-a-service product does not allow customers to failover to third party providers.

“The obvious failure here is compounded by the fact that Amazon has made it difficult for users to backup their databases outside of RDS, leaving them no choice but to wait for someone at Amazon to work on their database. This lock-in is one reason many of our customers prefer to use our MySQL master-slave setup or to architect their own,” von Eiken said.

Lorenzo Modesto, COO at Bulletproof Networks said the outage reinforced fears in the Australian business community about the use of large-scale public compute clouds. 

"Ironically, the outage was sparked by the very high-availability features built into the services themselves," he noted.

"The additional complexity Amazon has architected into their services to deliver the availability required by mission critical applications has introduced an exceptionally long and complex outage, which they're still struggling with after several days."

Telcos building their own cloud computes should take note, Modesto said, that a "cookie cutter approach" to delivering services won't suffice for the enterprise, which requires closer collaboration between hosting providers and application developers.