Datacom calls in structural engineers for data centre

Powered by SC Magazine
 

Post incident report on flood damaged data centre.

Datacom brought in structural engineers yesterday to assess the integrity of its data centre roof after a freak storm caused flooding and a power surge in part of the facility last weekend.

Engineers were also expected to "review the effectiveness of roof drainage", according to the official post-incident report (PIR) sighted by iTnews.

The PIR shed new light on the carrier issues caused by the water leak and outage.

Telstra network connectivity was lost for "approximately 45 minutes", the report said.

Optus connectivity was lost "for approximately nine hours" and was restored at 1.20AM the following day.

Nextgen was out for almost 11 hours with services restored after the operator replaced switches.

The network operations centre (NOC) chronology showed that emergency power remediation was implemented for Telstra, Optus, Pipe and Nextgen within 1.5 hours of the water leakage alert at 3.12PM.

"Optus and Nextgen were notified to assess on-site equipment," the report said.

Leak source

The storm brought severe rain and hail to the Melbourne metropolitan area. Hail blocked drains on the data centre roof and caused "extensive external street-level flooding."

"The blockage, combined with the heavy rain, resulted in water leakage entering the facility," the PIR said.

"Customers were advised initially to not attend the site due to safety concerns whilst containment activities were completed."

Within 15 minutes of the storm, Datacom staff conducted a visual inspection of the roof, where they discovered hail had blocked gutters on the facility's north side.

The first alarm from the building automation system identifying water leakage was at 3.12PM.

Three minutes later NOC engineers found "water intrusion through the roof on the north side open data area through a smoke detector ceiling fitting."

The leak was immediately plugged. A critical incident management process was enacted and customers were informed with a request to remotely check their systems.

Power outage

About an hour later, the water leakage caused a short circuit of the automatic static transfer switch that fed power distribution board number seven.

"Customer equipment that was only serviced by one electricity supply was affected for up to 12 hours while the circuit board was replaced or workarounds put in place," the PIR said.

Spare power circuits were installed for the Telstra media converter and switch and Optus racks, the PIR said.

The failed circuit breaker was disconnected, removed and replaced to restore electricity to affected cages.

The replacement took slightly longer than first thought because Datacom was "overly cautious with restoration of service focusing on safety and potential flow on effects."

A staged power-up of distribution board seven started at 3.40AM and was completed 25 minutes later.

From today, Datacom account managers were approaching customers to "review redundant electricity and network supply options."

Copyright © iTnews.com.au . All rights reserved.


Datacom calls in structural engineers for data centre
 
 
 
Top Stories
The iTnews Benchmark Awards
Meet the best of the best.
 
Telstra hands over copper, HFC in new $11bn NBN deal
Value of 2011 deal remains intact.
 
Photos: iTnews Benchmark 2015 finalists revealed
Awards alumni gather to celebrate.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...
Latest Comments
Polls
Who do you trust most to protect your private data?







   |   View results
Your bank
  39%
 
Your insurance company
  4%
 
A technology company (Google, Facebook et al)
  8%
 
Your telco, ISP or utility
  7%
 
A retailer (Coles, Woolworths et al)
  2%
 
A Federal Government agency (ATO, Centrelink etc)
  20%
 
An Australian law enforcement agency (AFP, ASIO et al)
  14%
 
A State Government agency (Health dept, etc)
  6%
TOTAL VOTES: 1751

Vote
Do you support the abolition of the Office of the Information Commissioner?