ISPs sweat on Primus data centre power failure report

Bevan Slattery, managing director of Pipe Networks, said yesterday that while "details are still being determined, there appears to have been a loss of mains power, combined with a failure of backup diesel generation, and therefore the depletion of the UPS batteries.

"This will have resulted in an outage to the Pipe Melbourne peering exchange, and to customers with Ethernet circuits that transit through or terminate at this site," Slattery said.

A service disruption notice posted by Westnet appears to confirm catastrophic failure of multiple redundancy layers.

"Fault identified as a failure of primary, secondary and redundant power systems," it said.

Hackett speculated that the Primus centre did not have the same plant-redundancy levels as the ISP's data centres in South Australia.

"[Our] main one actually uses two generator sets to back up mains power with automatic 'failover' should one of them not start up during a power outage," Hackett said.

"It appears Primus have a single generator set in their Melbourne facility which (obviously) didn't start up properly on this occasion."

Cheyne Jonstone, national sales manager at Cove, indicated in the Cove support forums that the centre's batteries had kicked in after the initial power failure.

"Battery power picked up the load but for some reason the generators were not adding charge to the batteries (from what I overheard)," Jonstone wrote.

"The only reason they would not be adding charge is if they had failed to start or there were some other issues when trying to bring that power into the mix."

While Jonstone said he had "no complaints about the Primus ground crew" that worked to restore power to the facility, he was disappointed that Cove wasn't warned that data loads were running off batteries.

"If I had known that, we could have shut down servers as a precautionary measure to ensure there was no data loss (not that there was any in the end)," he said.

A report detailing the cause of the failure is expected to be issued this afternoon.