Datacom calls in structural engineers for data centre

Powered by SC Magazine

Post incident report on flood damaged data centre.

Datacom brought in structural engineers yesterday to assess the integrity of its data centre roof after a freak storm caused flooding and a power surge in part of the facility last weekend.

Engineers were also expected to "review the effectiveness of roof drainage", according to the official post-incident report (PIR) sighted by iTnews.

The PIR shed new light on the carrier issues caused by the water leak and outage.

Telstra network connectivity was lost for "approximately 45 minutes", the report said.

Optus connectivity was lost "for approximately nine hours" and was restored at 1.20AM the following day.

Nextgen was out for almost 11 hours with services restored after the operator replaced switches.

The network operations centre (NOC) chronology showed that emergency power remediation was implemented for Telstra, Optus, Pipe and Nextgen within 1.5 hours of the water leakage alert at 3.12PM.

"Optus and Nextgen were notified to assess on-site equipment," the report said.

Leak source

The storm brought severe rain and hail to the Melbourne metropolitan area. Hail blocked drains on the data centre roof and caused "extensive external street-level flooding."

"The blockage, combined with the heavy rain, resulted in water leakage entering the facility," the PIR said.

"Customers were advised initially to not attend the site due to safety concerns whilst containment activities were completed."

Within 15 minutes of the storm, Datacom staff conducted a visual inspection of the roof, where they discovered hail had blocked gutters on the facility's north side.

The first alarm from the building automation system identifying water leakage was at 3.12PM.

Three minutes later NOC engineers found "water intrusion through the roof on the north side open data area through a smoke detector ceiling fitting."

The leak was immediately plugged. A critical incident management process was enacted and customers were informed with a request to remotely check their systems.

Power outage

About an hour later, the water leakage caused a short circuit of the automatic static transfer switch that fed power distribution board number seven.

"Customer equipment that was only serviced by one electricity supply was affected for up to 12 hours while the circuit board was replaced or workarounds put in place," the PIR said.

Spare power circuits were installed for the Telstra media converter and switch and Optus racks, the PIR said.

The failed circuit breaker was disconnected, removed and replaced to restore electricity to affected cages.

The replacement took slightly longer than first thought because Datacom was "overly cautious with restoration of service focusing on safety and potential flow on effects."

A staged power-up of distribution board seven started at 3.40AM and was completed 25 minutes later.

From today, Datacom account managers were approaching customers to "review redundant electricity and network supply options."

Copyright © . All rights reserved.

Datacom calls in structural engineers for data centre
Top Stories
Earning the right to innovate
Breaking down the barriers to innovation is a long, but rewarding process, says Bank of Queensland Group CIO, Julie Bale.
A call for timely reporting
[Blog post] Businesses need incentives to keep customer data secure.
Doubts cast on Queensland's ICT Dashboard
Opposition, former Govt CIO say it can't be trusted.
Sign up to receive iTnews email bulletins
Latest Comments
What is delaying adoption of public cloud in your organisation?

   |   View results
Lock-in concerns
Application integration concerns
Security and compliance concerns
Unreliable network infrastructure
Data sovereignty concerns
Lack of stakeholder support
Protecting on-premise IT jobs
Difficulty transitioning CapEx budget into OpEx