Datacom calls in structural engineers for data centre

By on
Datacom calls in structural engineers for data centre

Post incident report on flood damaged data centre.

Datacom brought in structural engineers yesterday to assess the integrity of its data centre roof after a freak storm caused flooding and a power surge in part of the facility last weekend.

Engineers were also expected to "review the effectiveness of roof drainage", according to the official post-incident report (PIR) sighted by iTnews.

The PIR shed new light on the carrier issues caused by the water leak and outage.

Telstra network connectivity was lost for "approximately 45 minutes", the report said.

Optus connectivity was lost "for approximately nine hours" and was restored at 1.20AM the following day.

Nextgen was out for almost 11 hours with services restored after the operator replaced switches.

The network operations centre (NOC) chronology showed that emergency power remediation was implemented for Telstra, Optus, Pipe and Nextgen within 1.5 hours of the water leakage alert at 3.12PM.

"Optus and Nextgen were notified to assess on-site equipment," the report said.

Leak source

The storm brought severe rain and hail to the Melbourne metropolitan area. Hail blocked drains on the data centre roof and caused "extensive external street-level flooding."

"The blockage, combined with the heavy rain, resulted in water leakage entering the facility," the PIR said.

"Customers were advised initially to not attend the site due to safety concerns whilst containment activities were completed."

Within 15 minutes of the storm, Datacom staff conducted a visual inspection of the roof, where they discovered hail had blocked gutters on the facility's north side.

The first alarm from the building automation system identifying water leakage was at 3.12PM.

Three minutes later NOC engineers found "water intrusion through the roof on the north side open data area through a smoke detector ceiling fitting."

The leak was immediately plugged. A critical incident management process was enacted and customers were informed with a request to remotely check their systems.

Power outage

About an hour later, the water leakage caused a short circuit of the automatic static transfer switch that fed power distribution board number seven.

"Customer equipment that was only serviced by one electricity supply was affected for up to 12 hours while the circuit board was replaced or workarounds put in place," the PIR said.

Spare power circuits were installed for the Telstra media converter and switch and Optus racks, the PIR said.

The failed circuit breaker was disconnected, removed and replaced to restore electricity to affected cages.

The replacement took slightly longer than first thought because Datacom was "overly cautious with restoration of service focusing on safety and potential flow on effects."

A staged power-up of distribution board seven started at 3.40AM and was completed 25 minutes later.

From today, Datacom account managers were approaching customers to "review redundant electricity and network supply options."

Copyright © iTnews.com.au . All rights reserved.
Tags:

Most Read Articles

You must be a registered member of iTnews to post a comment.
| Register

Poll

New Windows 10 users, are you upgrading from...
Windows 8
Windows 7
Windows XP
Another operating system
Windows Vista
How should the costs of Australia's piracy scheme be split?
Rights holders should foot the whole bill
50/50
ISPs should foot the whole bill
Government should chip in a bit
Other
View poll archive

Whitepapers from our sponsors

What will the stadium of the future look like?
What will the stadium of the future look like?
New technology adoption is pushing enterprise networks to breaking point
New technology adoption is pushing enterprise networks to breaking point
Gartner names IBM a 'Leader' for Disaster Recovery as a Service
Gartner names IBM a 'Leader' for Disaster Recovery as a Service
The next era of business continuity: Are you ready for an always-on world?
The next era of business continuity: Are you ready for an always-on world?

Log In

Username:
Password:
|  Forgot your password?