Datacom data centre flooded by Melbourne storm

Powered by SC Magazine
 

Carriers and customers hit as operator denies outage.

Datacom suffered a major outage in its Melbourne co-location facility after heavy rain broke a ceiling panel and poured into customers' equipment on Saturday afternoon.

The data centre operator's managing director Michael Browne denied there was an outage.

"There's no weight in that," he told iTnews. "The centre was not down and not ever flooded."

But his version is at odds with his own network operations centre (NOC), a series of photos sighted by iTnews and first-hand accounts from furious customers, who said the dead equipment included storage area networks, servers and routers.

"People had racks full of equipment that was dead. Some techs in there were close to tears," a national voice and data carrier who spoke on condition of anonymity said.

"We only had two servers and a router die. Others came off far worse. In some neighbouring racks, people were pouring water out of power supplies and SAN equipment."

Another customer confirmed they had been off-the-air for 13 hours and was aware of yet another customer - a business associate - with water-damaged kit.

Photos emerged showing a large hole and water-damaged ceiling above a caged area and buckets sitting on top of overhead cable trays, with customer racks clearly visible beneath.

(Photo: jason_one via Flickr).

Customers who did not suffer water damage lost power and cross-connects for up to 12 hours.

They were also unable to gain access to their racks until at least Sunday morning.

At least one customer was collecting information from others affected with a view to launching a class action against Datacom.

Mixed reports from majors

A number of major carriers and wholesalers were understood to have kit in Datacom that was affected by the outage.

An Optus spokesman confirmed to iTnews the telco rented space in Datacom and was affected by the power outage.

Wholesaler Vocus - which has a point-of-presence there - was able to work around the issue, chief executive James Spenceley said.

And AAPT - which had "a minor amount of traffic out of Datacom" - "didn't experience any notable impact to services for our customers as a result of the weather in Melbourne", a spokesman said.

They had only seen "unofficial reports" of the damage to Datacom.

NOC messages

iTnews has obtained the string of SMS outage messages sent to affected customers as the flood drama unfolded.

The first was sent on Saturday at 3.52PM titled: "Sbj: Flooding on the data floor."

At 5.28PM, network operations centre (NOC) staff told customers: "Datacom GC site has been affected by major storms. Some systems are currently shutdown until further notice, the remainder is stable."

At 6.40PM, "customers with single power supplies on DB7 (open data floor)" were advised they were affected by the flood. The "facility [was] otherwise stable", the update again said.

At 7.47PM, an update said the fault had been isolated and estimated power would resume "by midnight".

Both messages advised customers: "Do not come to site until further notice."

By 2.14AM Sunday, the situation had worsened. The site was "still experiencing issues" and the floor remained closed.

A "staged power-up" started at 3.49AM. A half-hour later, customers were told power had been restored to "99 percent of affected circuits". They were also invited to "please check your devices".

An official post-incident report did not appear to have been released. Information flows were complicated by a public holiday in Melbourne today.

At least one customer said they had been advised by NOC staff of a power surge in the facility's power distribution system.

Customers attached to newer power distribution units (PDUs) were saved by overload switches. Others were understood to have not been as lucky.

Lawsuit in the wings

Customers, including web hosts and ISPs, last night received an anonymous email seeking information from those affected for the purpose of commencing a class action.

The organiser told iTnews that parties interested in joining the suit should email datacom (dot) lawsuit (at) gmail (dot) com.

A further formal statement was requested from Datacom.

Were you impacted by the Datacom outage? Did you lose equipment? Email rcrozier (at) itnews (dot) com (au) and tell us your experience.

Copyright © iTnews.com.au . All rights reserved.


Datacom data centre flooded by Melbourne storm
Photo courtesy of customer.
 
 
 
Top Stories
Frugality as a service: the Amazon story
Behind the scenes, Amazon Web Services is one lean machine.
 
Negotiating with the cloud email megavendors
[Blog post] Lessons from Woolworths’ mammoth migration.
 
Qld govt to move up to 149k staff onto Office 365
Australia's largest deployment, outside of the universities.
 
 
Photo courtesy of customer.
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

The great data centre opportunity on Australia's doorstep
The great data centre opportunity on Australia's doorstep
Scott Noteboom, CEO of LitBit speaking at The Australian Data Centre Strategy Summit 2014 in the Gold Coast, Queensland, Australia. http://bit.ly/1qpxVfV Scott Noteboom is a data centre engineer who led builds for Apple and Yahoo in the earliest days of the cloud, and who now eyes Asia as the next big opportunity. Read more: http://www.itnews.com.au/News/372482,how-do-we-serve-three-billion-new-internet-users.aspx#ixzz2yNLmMG5C
Interview: Karl Maftoum, CIO, ACMA
Interview: Karl Maftoum, CIO, ACMA
To COTS or not to COTS? iTnews asks Karl Maftoum, CIO of the ACMA, at the CIO Strategy Summit.
Susan Sly: What is the Role of the CIO?
Susan Sly: What is the Role of the CIO?
AEMO chief information officer Susan Sly calls for more collaboration among Australia's technology leaders at the CIO Strategy Summit.
Meet the 2014 Finance CIO of the Year
Meet the 2014 Finance CIO of the Year
Credit Union Australia's David Gee awarded Finance CIO of the Year at the iTnews Benchmark Awards.
Meet the 2014 Retail CIO of the Year
Meet the 2014 Retail CIO of the Year
Damon Rees named Retail CIO of the Year at the iTnews Benchmark Awards for his work at Woolworths.
Robyn Elliott named the 2014 Utilities CIO of the Year
Robyn Elliott named the 2014 Utilities CIO of the Year
Acting Foxtel CIO David Marks accepts an iTnews Benchmark Award on behalf of Robyn Elliott.
Meet the 2014 Industrial CIO of the Year
Meet the 2014 Industrial CIO of the Year
Sanjay Mehta named Industrial CIO of the Year at the iTnews Benchmark Awards for his work at ConocoPhillips.
Meet the 2014 Healthcare CIO of the Year
Meet the 2014 Healthcare CIO of the Year
Greg Wells named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at NSW Health.
Meet the 2014 Education CIO of the Year
Meet the 2014 Education CIO of the Year
William Confalonieri named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at Deakin University.
Meet the 2014 Government CIO of the Year
Meet the 2014 Government CIO of the Year
David Johnson named Government CIO of the Year at the iTnews Benchmark Awards for his work at the Queensland Police Service.
Q and A: Coalition Broadband Policy
Q and A: Coalition Broadband Policy
Malcolm Turnbull and Tony Abbott discuss the Coalition's broadband policy with the press.
AFP scalps hacker 'leader' inside Australia's IT ranks.
AFP scalps hacker 'leader' inside Australia's IT ranks.
The Australian Federal Police have arrested a Sydney-based IT security professional for hacking a government website.
NBN Petition Delivered To Turnbull's Office
NBN Petition Delivered To Turnbull's Office
UTS CIO: IT teams of the future
UTS CIO: IT teams of the future
UTS CIO Chrissy Burns talks data.
New UTS Building: the IT within
New UTS Building: the IT within
The IT behind tomorrow's universities.
iTnews' NBN Panel
iTnews' NBN Panel
Is your enterprise NBN-ready?
Introducing iTnews Labs
Introducing iTnews Labs
See a timelapse of the iTnews labs being unboxed, set up and switched on! iTnews will produce independent testing of the latest enterprise software to hit the market after installing a purpose-built test lab in Sydney. Watch the installation of two DL380p servers, two HP StoreVirtual 4330 storage arrays and two HP ProCurve 2920 switches.
The True Cost of BYOD
The True Cost of BYOD
iTnews' Brett Winterford gives attendees of the first 'Touch Tomorrow' event in Brisbane a brief look at his research into enterprise mobility. What are the use cases and how can they be quantified? What price should you expect to pay for securing mobile access to corporate applications? What's coming around the corner?
Ghost clouds
Ghost clouds
ACMA chair Chris Chapman says there is uncertainty over whether certain classes of cloud service providers are caught by regulations.
Was the Snowden leak inevitable?
Was the Snowden leak inevitable?
Privacy experts David Vaile (UNSW Cyberspace Law and Policy Centre) and Craig Scroggie (CEO, NextDC) claim they were not surprised by the Snowden leaks about the NSA's PRISM program.
Latest Comments
Polls
Which bank is most likely to suffer an RBS-style meltdown?





   |   View results
ANZ
  21%
 
Bankwest
  9%
 
CommBank
  11%
 
National Australia Bank
  17%
 
Suncorp
  24%
 
Westpac
  19%
TOTAL VOTES: 1455

Vote