Datacom calls in structural engineers for data centre

Powered by SC Magazine
 

Post incident report on flood damaged data centre.

Datacom brought in structural engineers yesterday to assess the integrity of its data centre roof after a freak storm caused flooding and a power surge in part of the facility last weekend.

Engineers were also expected to "review the effectiveness of roof drainage", according to the official post-incident report (PIR) sighted by iTnews.

The PIR shed new light on the carrier issues caused by the water leak and outage.

Telstra network connectivity was lost for "approximately 45 minutes", the report said.

Optus connectivity was lost "for approximately nine hours" and was restored at 1.20AM the following day.

Nextgen was out for almost 11 hours with services restored after the operator replaced switches.

The network operations centre (NOC) chronology showed that emergency power remediation was implemented for Telstra, Optus, Pipe and Nextgen within 1.5 hours of the water leakage alert at 3.12PM.

"Optus and Nextgen were notified to assess on-site equipment," the report said.

Leak source

The storm brought severe rain and hail to the Melbourne metropolitan area. Hail blocked drains on the data centre roof and caused "extensive external street-level flooding."

"The blockage, combined with the heavy rain, resulted in water leakage entering the facility," the PIR said.

"Customers were advised initially to not attend the site due to safety concerns whilst containment activities were completed."

Within 15 minutes of the storm, Datacom staff conducted a visual inspection of the roof, where they discovered hail had blocked gutters on the facility's north side.

The first alarm from the building automation system identifying water leakage was at 3.12PM.

Three minutes later NOC engineers found "water intrusion through the roof on the north side open data area through a smoke detector ceiling fitting."

The leak was immediately plugged. A critical incident management process was enacted and customers were informed with a request to remotely check their systems.

Power outage

About an hour later, the water leakage caused a short circuit of the automatic static transfer switch that fed power distribution board number seven.

"Customer equipment that was only serviced by one electricity supply was affected for up to 12 hours while the circuit board was replaced or workarounds put in place," the PIR said.

Spare power circuits were installed for the Telstra media converter and switch and Optus racks, the PIR said.

The failed circuit breaker was disconnected, removed and replaced to restore electricity to affected cages.

The replacement took slightly longer than first thought because Datacom was "overly cautious with restoration of service focusing on safety and potential flow on effects."

A staged power-up of distribution board seven started at 3.40AM and was completed 25 minutes later.

From today, Datacom account managers were approaching customers to "review redundant electricity and network supply options."

Copyright © iTnews.com.au . All rights reserved.


Datacom calls in structural engineers for data centre
 
 
 
Top Stories
Frugality as a service: the Amazon story
Behind the scenes, Amazon Web Services is one lean machine.
 
Negotiating with the cloud email megavendors
[Blog post] Lessons from Woolworths’ mammoth migration.
 
Qld govt to move up to 149k staff onto Office 365
Australia's largest deployment, outside of the universities.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

The great data centre opportunity on Australia's doorstep
The great data centre opportunity on Australia's doorstep
Scott Noteboom, CEO of LitBit speaking at The Australian Data Centre Strategy Summit 2014 in the Gold Coast, Queensland, Australia. http://bit.ly/1qpxVfV Scott Noteboom is a data centre engineer who led builds for Apple and Yahoo in the earliest days of the cloud, and who now eyes Asia as the next big opportunity. Read more: http://www.itnews.com.au/News/372482,how-do-we-serve-three-billion-new-internet-users.aspx#ixzz2yNLmMG5C
Interview: Karl Maftoum, CIO, ACMA
Interview: Karl Maftoum, CIO, ACMA
To COTS or not to COTS? iTnews asks Karl Maftoum, CIO of the ACMA, at the CIO Strategy Summit.
Susan Sly: What is the Role of the CIO?
Susan Sly: What is the Role of the CIO?
AEMO chief information officer Susan Sly calls for more collaboration among Australia's technology leaders at the CIO Strategy Summit.
Meet the 2014 Finance CIO of the Year
Meet the 2014 Finance CIO of the Year
Credit Union Australia's David Gee awarded Finance CIO of the Year at the iTnews Benchmark Awards.
Meet the 2014 Retail CIO of the Year
Meet the 2014 Retail CIO of the Year
Damon Rees named Retail CIO of the Year at the iTnews Benchmark Awards for his work at Woolworths.
Robyn Elliott named the 2014 Utilities CIO of the Year
Robyn Elliott named the 2014 Utilities CIO of the Year
Acting Foxtel CIO David Marks accepts an iTnews Benchmark Award on behalf of Robyn Elliott.
Meet the 2014 Industrial CIO of the Year
Meet the 2014 Industrial CIO of the Year
Sanjay Mehta named Industrial CIO of the Year at the iTnews Benchmark Awards for his work at ConocoPhillips.
Meet the 2014 Healthcare CIO of the Year
Meet the 2014 Healthcare CIO of the Year
Greg Wells named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at NSW Health.
Meet the 2014 Education CIO of the Year
Meet the 2014 Education CIO of the Year
William Confalonieri named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at Deakin University.
Meet the 2014 Government CIO of the Year
Meet the 2014 Government CIO of the Year
David Johnson named Government CIO of the Year at the iTnews Benchmark Awards for his work at the Queensland Police Service.
Q and A: Coalition Broadband Policy
Q and A: Coalition Broadband Policy
Malcolm Turnbull and Tony Abbott discuss the Coalition's broadband policy with the press.
AFP scalps hacker 'leader' inside Australia's IT ranks.
AFP scalps hacker 'leader' inside Australia's IT ranks.
The Australian Federal Police have arrested a Sydney-based IT security professional for hacking a government website.
NBN Petition Delivered To Turnbull's Office
NBN Petition Delivered To Turnbull's Office
UTS CIO: IT teams of the future
UTS CIO: IT teams of the future
UTS CIO Chrissy Burns talks data.
New UTS Building: the IT within
New UTS Building: the IT within
The IT behind tomorrow's universities.
iTnews' NBN Panel
iTnews' NBN Panel
Is your enterprise NBN-ready?
Introducing iTnews Labs
Introducing iTnews Labs
See a timelapse of the iTnews labs being unboxed, set up and switched on! iTnews will produce independent testing of the latest enterprise software to hit the market after installing a purpose-built test lab in Sydney. Watch the installation of two DL380p servers, two HP StoreVirtual 4330 storage arrays and two HP ProCurve 2920 switches.
The True Cost of BYOD
The True Cost of BYOD
iTnews' Brett Winterford gives attendees of the first 'Touch Tomorrow' event in Brisbane a brief look at his research into enterprise mobility. What are the use cases and how can they be quantified? What price should you expect to pay for securing mobile access to corporate applications? What's coming around the corner?
Ghost clouds
Ghost clouds
ACMA chair Chris Chapman says there is uncertainty over whether certain classes of cloud service providers are caught by regulations.
Was the Snowden leak inevitable?
Was the Snowden leak inevitable?
Privacy experts David Vaile (UNSW Cyberspace Law and Policy Centre) and Craig Scroggie (CEO, NextDC) claim they were not surprised by the Snowden leaks about the NSA's PRISM program.
Latest Comments
Polls
Which bank is most likely to suffer an RBS-style meltdown?





   |   View results
ANZ
  21%
 
Bankwest
  9%
 
CommBank
  11%
 
National Australia Bank
  17%
 
Suncorp
  24%
 
Westpac
  19%
TOTAL VOTES: 1453

Vote