Internode data centre struck by short power outage

A storm yesterday afternoon has cut the mains power to one of Internode’s main Adelaide data centres, affecting a subset of its customers for a period of up to sixty minutes.

Simon Hackett, Internode managing director, said the company was conducting an investigation into why its backup generators did not function correctly during the power outage.

"The generators have an output circuit breaker system to protect them, which appears to have been installed in the data centre with an incorrect configuration - in particular they were set to cut off the generators at a point substantially below the full rated power of the generator systems concerned," Hackett told iTnews in an email.

Hackett said the fault had remained undetected since the data centre generators were first comissioned.

"Since the last full generator test, a substantial increase in the number of installed customer racks has brought total power draw just over the (incorrectly) configured cutoff point," Hackett said.

He added that the fault would have been detected and corrected at the next scheduled full load generator test in November.

"Unfortunately the storms in Adelaide today shook this fault out of the woodwork first," Hackett said.

"That's the bad news. The good news is that the configuration fault was located and corrected during the on-site work today, and as a result, the data centre was in fact running at full load on generator power, with all customer racks powered up, prior to ETSA restoring mains power this afternoon," he said.

The data centre power failure followed a similar but longer lasting generator incident that affected Primus’ Melbourne data centre in August.

Hackett said that once UPS battery power was exhausted, some customers' co-located server racks had to be turned off. He said this was to bring total power demand within the capacity of what Internode's generator could handle.

"Internode apologises sincerely for the period of power loss to some customer racks today and does not anticipate a recurrence of the fault," Hackett said.

"This data centre has routinely provided full generator power flawlessly during Adelaide mains power outages in the past, and this is an isolated and surprising incident in that context," he said.

Service disruptions began at 3:31pm Adelaide time. All data centre services had been restored at 4:30pm.