Stress tests rain on Amazon's cloud

 

Availability an issue for Amazon EC2, Google AppEngine and Microsoft Azure.

Stress tests conducted by Sydney-based researchers have revealed that the infrastructure-on-demand services offered by Amazon, Google and Microsoft suffer from regular performance and availability issues.

The team of researchers, led by the University of New South Wales (UNSW) and in collaboration with researchers at NICTA (National ICT Australia) and the Smart Services Cooperative Research Centre (CRC), have spent seven months stress testing Amazon's EC2, Google's AppEngine and Microsoft's Azure cloud computing services.

The analysis simulated 2000 concurrent users connecting to services from each of the three providers, with researchers measuring response times and other performance metrics.

Anna Liu, Associate Professor in services engineering at the UNSW School of Computer Science told iTnews she was excited by Cloud Computing as it could potentially enable organisations to "outsource a certain amount of their risks and costs and tap into new economies of scale."

"We saw a lot of hype and confusion, and decided to lead a team of researchers and actually get our hands dirty with this stuff," she said. "We put these services to a stress test to see if these cloud platforms really perform the way their vendors say they do."

The research found some merit to vendors' claims of "perceived infinite scalability."

"With a simulation of 2000 concurrent users, we watched the cloud services scale up and respond dynamically to that demand," Liu said.

But researchers also found that the three platforms delivered wildly variable performance results as Amazon, Google and Microsoft trialled, added and dropped new features.

Response times on the service also varied by a factor of twenty depending on the time of day the services were accessed, she said.

The response times collated in Sydney were tested against measurement instruments loaded onto the cloud platform to isolate whether delays were attributable to the service itself or the latency involved with accessing US-based data centres from Australia.

Liu said the study revealed which of the three cloud infrastructures were suited to particular applications.

"Using Google AppEngine, none of your data processing tasks can last any longer than thirty seconds, or it throws an exception back at you," she said.

"This is very consistent with the Google business model - they want to enable simple web applications to thrive on the Internet. AppEngine is there to enable the rapid development of simple web applications that don't include intense compute at the back end."

Amazon's EC2, by contrast, provided "base cloud computing support" with value-added services supplied by third parties.

Microsoft, Liu said, has the enterprise and the ISV firmly in its sights with the Azure platform.

"Microsoft is looking to support the transformation or migration of in-premise apps with the cloud."

Liu, a former Microsoft employee, said that while Azure is limited to the .Net development platform, adapters are being released for Java and PHP among others. 

Liu said all three services lack the monitoring tools large organisations require to check on whether the platform is meeting service level agreements.

"None of the platforms have the kind of monitoring required to have a reasonable conversation about performance," she said. "They provide some level of monitoring, but what little there is caters for developers, not business users. And while Amazon provides a dashboard of how much it is costing you so far, for example, there is nothing in terms of forecasts about what it will cost you in the future.

This story has since been updated here.


Stress tests rain on Amazon's cloud
 
 
 
Top Stories
Toll Group to go Google
Poaches Woolworths project manager.
 
How News Corp's CIO tackled skills in his race to the cloud
What to do when your team’s talents are no longer needed.
 
Photos: How Thodey transformed Telstra
From turbulent Trujillo to Australia's leading telco.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...
Latest articles on BIT Latest Articles from BIT
Microsoft is offering Azure for Disaster Recovery to Australian SMBs
Feb 10, 2015
If you haven't talked to your IT provider about disaster recovery, it might be worth discussing ...
The 2015 Xero Roadshow is on: here are the locations and dates
Feb 6, 2015
The 2015 Xero Roadshow kicked off this week - see where you can attend at locations around ...
Microsoft Outlook is now on iPhone and iPad: why could this be useful?
Jan 30, 2015
Microsoft today released Office for Android and Outlook for iOS - complementing the other Office ...
Franchisees, here's something you should know about
Jan 23, 2015
You need to know the Code if you are a franchisee or franchisor as the penalties are significant.
Xero users rejoice! Quoting has finally arrived
Jan 23, 2015
It has taken years, but Xero has at last added integrated quoting to its online accounting software.
Latest Comments
Polls
Who do you trust most to protect your private data?







   |   View results
Your bank
  35%
 
Your insurance company
  5%
 
A technology company (Google, Facebook et al)
  8%
 
Your telco, ISP or utility
  8%
 
A retailer (Coles, Woolworths et al)
  4%
 
A Federal Government agency (ATO, Centrelink etc)
  18%
 
An Australian law enforcement agency (AFP, ASIO et al)
  15%
 
A State Government agency (Health dept, etc)
  7%
TOTAL VOTES: 3927

Vote
Do you support the abolition of the Office of the Information Commissioner?

   |   View results
I support shutting down the OAIC.
  27%
 
I DON'T support shutting the OAIC.
  73%
TOTAL VOTES: 1335

Vote