Datacom calls in structural engineers for data centre

Post incident report on flood damaged data centre.

Datacom brought in structural engineers yesterday to assess the integrity of its data centre roof after a freak storm caused flooding and a power surge in part of the facility last weekend.

Engineers were also expected to "review the effectiveness of roof drainage", according to the official post-incident report (PIR) sighted by iTnews.

The PIR shed new light on the carrier issues caused by the water leak and outage.

Telstra network connectivity was lost for "approximately 45 minutes", the report said.

Optus connectivity was lost "for approximately nine hours" and was restored at 1.20AM the following day.

Nextgen was out for almost 11 hours with services restored after the operator replaced switches.

The network operations centre (NOC) chronology showed that emergency power remediation was implemented for Telstra, Optus, Pipe and Nextgen within 1.5 hours of the water leakage alert at 3.12PM.

"Optus and Nextgen were notified to assess on-site equipment," the report said.

Leak source

The storm brought severe rain and hail to the Melbourne metropolitan area. Hail blocked drains on the data centre roof and caused "extensive external street-level flooding."

"The blockage, combined with the heavy rain, resulted in water leakage entering the facility," the PIR said.

"Customers were advised initially to not attend the site due to safety concerns whilst containment activities were completed."

Within 15 minutes of the storm, Datacom staff conducted a visual inspection of the roof, where they discovered hail had blocked gutters on the facility's north side.

The first alarm from the building automation system identifying water leakage was at 3.12PM.

Three minutes later NOC engineers found "water intrusion through the roof on the north side open data area through a smoke detector ceiling fitting."

The leak was immediately plugged. A critical incident management process was enacted and customers were informed with a request to remotely check their systems.

Power outage

About an hour later, the water leakage caused a short circuit of the automatic static transfer switch that fed power distribution board number seven.

"Customer equipment that was only serviced by one electricity supply was affected for up to 12 hours while the circuit board was replaced or workarounds put in place," the PIR said.

Spare power circuits were installed for the Telstra media converter and switch and Optus racks, the PIR said.

The failed circuit breaker was disconnected, removed and replaced to restore electricity to affected cages.

The replacement took slightly longer than first thought because Datacom was "overly cautious with restoration of service focusing on safety and potential flow on effects."

A staged power-up of distribution board seven started at 3.40AM and was completed 25 minutes later.

From today, Datacom account managers were approaching customers to "review redundant electricity and network supply options."