Ninefold outage disables server provisioning

 

Spurs investment in second availability zone.

Australian cloud provider Ninefold has been unable to pinpoint the exact cause of a five-hour outage last week that led to virtual server provisioning in its cloud environment being disabled.

The outage started at 9.20am on Thursday February 16 and was resolved at 2.05pm the same day, according to a post-incident report obtained by iTnews.

The report initially pinned the incident on "an unexpected failure" of an NFS server that "automatically restarted itself and resumed normal operation within minutes".

"A number of physical host servers (which support customer VMs) were performing operations to the NFS server at the time of the unexpected Network File System server failure," the report stated.

"Due to the nature of the NFS server failure, this caused provisioning on these specific physical host servers to become unresponsive.

"Eventually, some customer VMs on these particular host servers became unavailable and unable to restart on alternate physical hosts."

Ninefold managing director Peter James told iTnews today that the fault was in the NFS server hardware "which ultimately transmitted into a software failure".

James could not say how this occurred. "We're still working with engineers to determine the absolute underlying events," he said.

He said patches had been applied to the NFS storage system and that further upgrades were in the pipeline.

Provisioning taken offline

Ninefold was careful in describing the outage, stating it was "not... system wide".

The outage was confined to part of the virtual server side of Ninefold's business, where "some" virtual server instances stored on "a number" of physical host servers connected to the problematic NFS server were impacted. The outage did not touch a "large" number of virtual servers or physical hosts and had no impact on Ninefold's cloud storage business.

However, the outage also meant that no customers - across the board - could spin up new virtual servers over the five hour period.

James confirmed that provisioning for all customers was disabled so that engineers could restore the VMs that had failed.

"We took a decision based on advice from our engineers," James said.

"The issue is that if we've got customers, albeit a small number of them, [that] are down, they're the ones that we absolutely focus on.

"We took a decision to take the appropriate steps to get them back up and running, albeit that on a Thursday it did mean that a number of customers couldn't provision but it did mean that we were able to reasonably quickly get that small number of customers who were more affected back up and running."

The post-incident report does not mention the number of physical host servers or customer VMs impacted and James declined to elaborate.

New availability zone

In response to the outage, James said Ninefold has doubled its investment in an already-planned core infrastructure enhancement project to $1 million. Installation work is expected "over the next few weeks".

The provider also plans to launch a second availability zone hosted in Macquarie Telecom's forthcoming Intellicentre 2 facility in North Ryde.

The zone is expected to be live in May, a month ahead of earlier expectations.

James said Ninefold would make a "significant investment as the anchor client" in the data centre.

He said Ninefold was "yet to work the detail out" on whether it might run its presence in Macquarie's two data centres in an active-active configuration.

"We're already underway with our planning, but we're yet to make that call," James said.

He said the launch of the zone was "largely aimed at increasing the resilience" of the service as it scaled up to meet customer growth.

Ninefold previously suffered a host server outage in August last year and another incident a few months earlier in May 2011.

Copyright © iTnews.com.au . All rights reserved.


Ninefold outage disables server provisioning
 
 
 
Top Stories
Frugality as a service: the Amazon story
Behind the scenes, Amazon Web Services is one lean machine.
 
Negotiating with the cloud email megavendors
[Blog post] Lessons from Woolworths’ mammoth migration.
 
Qld govt to move up to 149k staff onto Office 365
Australia's largest deployment, outside of the universities.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

The great data centre opportunity on Australia's doorstep
The great data centre opportunity on Australia's doorstep
Scott Noteboom, CEO of LitBit speaking at The Australian Data Centre Strategy Summit 2014 in the Gold Coast, Queensland, Australia. http://bit.ly/1qpxVfV Scott Noteboom is a data centre engineer who led builds for Apple and Yahoo in the earliest days of the cloud, and who now eyes Asia as the next big opportunity. Read more: http://www.itnews.com.au/News/372482,how-do-we-serve-three-billion-new-internet-users.aspx#ixzz2yNLmMG5C
Interview: Karl Maftoum, CIO, ACMA
Interview: Karl Maftoum, CIO, ACMA
To COTS or not to COTS? iTnews asks Karl Maftoum, CIO of the ACMA, at the CIO Strategy Summit.
Susan Sly: What is the Role of the CIO?
Susan Sly: What is the Role of the CIO?
AEMO chief information officer Susan Sly calls for more collaboration among Australia's technology leaders at the CIO Strategy Summit.
Meet the 2014 Finance CIO of the Year
Meet the 2014 Finance CIO of the Year
Credit Union Australia's David Gee awarded Finance CIO of the Year at the iTnews Benchmark Awards.
Meet the 2014 Retail CIO of the Year
Meet the 2014 Retail CIO of the Year
Damon Rees named Retail CIO of the Year at the iTnews Benchmark Awards for his work at Woolworths.
Robyn Elliott named the 2014 Utilities CIO of the Year
Robyn Elliott named the 2014 Utilities CIO of the Year
Acting Foxtel CIO David Marks accepts an iTnews Benchmark Award on behalf of Robyn Elliott.
Meet the 2014 Industrial CIO of the Year
Meet the 2014 Industrial CIO of the Year
Sanjay Mehta named Industrial CIO of the Year at the iTnews Benchmark Awards for his work at ConocoPhillips.
Meet the 2014 Healthcare CIO of the Year
Meet the 2014 Healthcare CIO of the Year
Greg Wells named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at NSW Health.
Meet the 2014 Education CIO of the Year
Meet the 2014 Education CIO of the Year
William Confalonieri named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at Deakin University.
Meet the 2014 Government CIO of the Year
Meet the 2014 Government CIO of the Year
David Johnson named Government CIO of the Year at the iTnews Benchmark Awards for his work at the Queensland Police Service.
Q and A: Coalition Broadband Policy
Q and A: Coalition Broadband Policy
Malcolm Turnbull and Tony Abbott discuss the Coalition's broadband policy with the press.
AFP scalps hacker 'leader' inside Australia's IT ranks.
AFP scalps hacker 'leader' inside Australia's IT ranks.
The Australian Federal Police have arrested a Sydney-based IT security professional for hacking a government website.
NBN Petition Delivered To Turnbull's Office
NBN Petition Delivered To Turnbull's Office
UTS CIO: IT teams of the future
UTS CIO: IT teams of the future
UTS CIO Chrissy Burns talks data.
New UTS Building: the IT within
New UTS Building: the IT within
The IT behind tomorrow's universities.
iTnews' NBN Panel
iTnews' NBN Panel
Is your enterprise NBN-ready?
Introducing iTnews Labs
Introducing iTnews Labs
See a timelapse of the iTnews labs being unboxed, set up and switched on! iTnews will produce independent testing of the latest enterprise software to hit the market after installing a purpose-built test lab in Sydney. Watch the installation of two DL380p servers, two HP StoreVirtual 4330 storage arrays and two HP ProCurve 2920 switches.
The True Cost of BYOD
The True Cost of BYOD
iTnews' Brett Winterford gives attendees of the first 'Touch Tomorrow' event in Brisbane a brief look at his research into enterprise mobility. What are the use cases and how can they be quantified? What price should you expect to pay for securing mobile access to corporate applications? What's coming around the corner?
Ghost clouds
Ghost clouds
ACMA chair Chris Chapman says there is uncertainty over whether certain classes of cloud service providers are caught by regulations.
Was the Snowden leak inevitable?
Was the Snowden leak inevitable?
Privacy experts David Vaile (UNSW Cyberspace Law and Policy Centre) and Craig Scroggie (CEO, NextDC) claim they were not surprised by the Snowden leaks about the NSA's PRISM program.
Latest Comments
Polls
Which bank is most likely to suffer an RBS-style meltdown?





   |   View results
ANZ
  21%
 
Bankwest
  9%
 
CommBank
  11%
 
National Australia Bank
  17%
 
Suncorp
  24%
 
Westpac
  19%
TOTAL VOTES: 1451

Vote