More data released on cloud stress tests

 

Throttling, power failures responsible for unpredictable clouds.

A Sydney-based university researcher leading a study of the cloud computing platforms of Amazon, Google and Microsoft yesterday revealed some of the reasons cloud services provide unpredicatable results under stress testing.

Anna Liu, associate professor in services engineering at the school of computer science and engineering at the University of NSW (UNSW) told iTnews last week that response times to queries sent to Amazon's EC2, Google's AppEngine and Microsoft's Azure platform varied by a factor of 20 depending on the service and time of day the cloud was accessed.

Liu today told the Architecture Australia Forum that throttling, power failures and deliberate design constraints were among the reasons for the unpredictability.

"While scalability works nicely for these platforms, availability is an interesting question," she told the Sydney audience. "We have noticed outages."

Liu said that with the effect of network latency issues included, stress tests revealed error rates of up to 12 percent - these being requests from the Sydney research headquarters to the U.S.-based cloud provider that didn't make the round trip back.

Once network latency was removed as a factor (due to instrumentation loaded onto the cloud platform itself), error rates were far lower, but still of concern to enterprise application developers.

While stressing that the numbers presented were only 'snapshots in time', Liu showed two slides to illustrate occasions in which availability became an issue.

In the first slide (below), tests of Google's AppEngine showed average failure rates of 0.04 per cent, along with some examples of the error messages received when the service failed.

click to view full size image
Snapshot of Google AppEngine stress test (Source: Anna Liu)

The horizontal axis represents time samples (time of day), while the vertical axis records a gradual increase in the number of concurrent users connecting to the service.

Liu said that after comparing notes with technicians at Google's engineering labs, she discovered that these failures were in fact a design feature of the AppEngine service.

They were designed, she said, to cope with potential denial of service attacks.

"It was a feature they needed to avoid potential attacks," she said. "They found they could fix it easily by digging in to their back-end infrastructure and changing a few parameters to accept a greater volume of requests."

Liu said such design features should be expected and that they were "consistent with [Google's] simple programming model for your average web developer."

In a second slide [below], Liu showed much higher failure rates for tests of Amazon.com's SimpleDB service.

click to view full size image
Snapshot of Amazon SimpleDB stress test (Source: Anna Liu)

Liu said these results correlated with a "power and back-up generator failure" Amazon.com experienced at the time.

"You will have some unpredictability in large-scale environment, at least until we get all of these issues sorted out," Liu said.

Liu later told iTnews that she felt the challenge for developers wasn't so much about availability but unpredictability around when outages will hit.

"From an architecture perspective, you need to have a plan for when outages hit," she said.

Liu said that it was difficult to do apple-for-apple comparisons of the three platforms considering their different use cases.

"It is not like the (database benchmarking organisation) TPCC, which measures the performance of relational databases," she said. "There is a well-defined set of capabilities and usage scenarios for a relational database, not so for a cloud service."

But Liu said one of her eventual aims is to build up a historical view of the performance of the three cloud platforms and create a simple web application to help application architects choose which suits their requirements.

"Ideally, you would be choosing an application profile, and be given the response time of a query using x service on y cloud platform," she said.


More data released on cloud stress tests
 
 
 
Top Stories
Frugality as a service: the Amazon story
Behind the scenes, Amazon Web Services is one lean machine.
 
Negotiating with the cloud email megavendors
[Blog post] Lessons from Woolworths’ mammoth migration.
 
Qld govt to move up to 149k staff onto Office 365
Australia's largest deployment, outside of the universities.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

The great data centre opportunity on Australia's doorstep
The great data centre opportunity on Australia's doorstep
Scott Noteboom, CEO of LitBit speaking at The Australian Data Centre Strategy Summit 2014 in the Gold Coast, Queensland, Australia. http://bit.ly/1qpxVfV Scott Noteboom is a data centre engineer who led builds for Apple and Yahoo in the earliest days of the cloud, and who now eyes Asia as the next big opportunity. Read more: http://www.itnews.com.au/News/372482,how-do-we-serve-three-billion-new-internet-users.aspx#ixzz2yNLmMG5C
Interview: Karl Maftoum, CIO, ACMA
Interview: Karl Maftoum, CIO, ACMA
To COTS or not to COTS? iTnews asks Karl Maftoum, CIO of the ACMA, at the CIO Strategy Summit.
Susan Sly: What is the Role of the CIO?
Susan Sly: What is the Role of the CIO?
AEMO chief information officer Susan Sly calls for more collaboration among Australia's technology leaders at the CIO Strategy Summit.
Meet the 2014 Finance CIO of the Year
Meet the 2014 Finance CIO of the Year
Credit Union Australia's David Gee awarded Finance CIO of the Year at the iTnews Benchmark Awards.
Meet the 2014 Retail CIO of the Year
Meet the 2014 Retail CIO of the Year
Damon Rees named Retail CIO of the Year at the iTnews Benchmark Awards for his work at Woolworths.
Robyn Elliott named the 2014 Utilities CIO of the Year
Robyn Elliott named the 2014 Utilities CIO of the Year
Acting Foxtel CIO David Marks accepts an iTnews Benchmark Award on behalf of Robyn Elliott.
Meet the 2014 Industrial CIO of the Year
Meet the 2014 Industrial CIO of the Year
Sanjay Mehta named Industrial CIO of the Year at the iTnews Benchmark Awards for his work at ConocoPhillips.
Meet the 2014 Healthcare CIO of the Year
Meet the 2014 Healthcare CIO of the Year
Greg Wells named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at NSW Health.
Meet the 2014 Education CIO of the Year
Meet the 2014 Education CIO of the Year
William Confalonieri named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at Deakin University.
Meet the 2014 Government CIO of the Year
Meet the 2014 Government CIO of the Year
David Johnson named Government CIO of the Year at the iTnews Benchmark Awards for his work at the Queensland Police Service.
Q and A: Coalition Broadband Policy
Q and A: Coalition Broadband Policy
Malcolm Turnbull and Tony Abbott discuss the Coalition's broadband policy with the press.
AFP scalps hacker 'leader' inside Australia's IT ranks.
AFP scalps hacker 'leader' inside Australia's IT ranks.
The Australian Federal Police have arrested a Sydney-based IT security professional for hacking a government website.
NBN Petition Delivered To Turnbull's Office
NBN Petition Delivered To Turnbull's Office
UTS CIO: IT teams of the future
UTS CIO: IT teams of the future
UTS CIO Chrissy Burns talks data.
New UTS Building: the IT within
New UTS Building: the IT within
The IT behind tomorrow's universities.
iTnews' NBN Panel
iTnews' NBN Panel
Is your enterprise NBN-ready?
Introducing iTnews Labs
Introducing iTnews Labs
See a timelapse of the iTnews labs being unboxed, set up and switched on! iTnews will produce independent testing of the latest enterprise software to hit the market after installing a purpose-built test lab in Sydney. Watch the installation of two DL380p servers, two HP StoreVirtual 4330 storage arrays and two HP ProCurve 2920 switches.
The True Cost of BYOD
The True Cost of BYOD
iTnews' Brett Winterford gives attendees of the first 'Touch Tomorrow' event in Brisbane a brief look at his research into enterprise mobility. What are the use cases and how can they be quantified? What price should you expect to pay for securing mobile access to corporate applications? What's coming around the corner?
Ghost clouds
Ghost clouds
ACMA chair Chris Chapman says there is uncertainty over whether certain classes of cloud service providers are caught by regulations.
Was the Snowden leak inevitable?
Was the Snowden leak inevitable?
Privacy experts David Vaile (UNSW Cyberspace Law and Policy Centre) and Craig Scroggie (CEO, NextDC) claim they were not surprised by the Snowden leaks about the NSA's PRISM program.
Latest articles on BIT Latest Articles from BIT
Another phone with Telstra's Blue Tick: The Samsung Galaxy S5
Apr 8, 2014
Samsung's latest flagship phone joins Telstra's list of recommended handsets for customers in ...
Run an online shop? This might be worth bookmarking
Mar 28, 2014
Things like Australian safety standards are probably the last thing on your mind, but just ...
Vodafone switches on 4G in Tasmania: list of locations
Mar 28, 2014
See a list of locations in Tasmania that now have access to 4G via Vodafone's network.
Samsung Galaxy S5 on sale from Telstra next month for $912
Mar 27, 2014
It's not cheap, but if you are looking to upgrade your phone then the Samsung Galaxy S5 could be ...
What Australian workplaces actually rely on tablet computers?
Mar 14, 2014
If you're curious about where tablets are being used at work, here are three examples.
Latest Comments
Polls
Which bank is most likely to suffer an RBS-style meltdown?





   |   View results
ANZ
  21%
 
Bankwest
  9%
 
CommBank
  11%
 
National Australia Bank
  17%
 
Suncorp
  24%
 
Westpac
  19%
TOTAL VOTES: 1442

Vote