QLD Govt services stalled by power failure

Two generators fail to sustain the data centre load.

Queensland public services have suffered a loss of IT services owing to a power outage in a CITEC-operated data centre in Brisbane's CBD late last week.

IT service provider CITEC lost power at its main data centre for nine hours last Thursday night. CITEC is a state-owned provider of IT services to the Queensland Government.

A spokesman for CITEC confirmed that the group "experienced a power interruption in the evening of Thursday 10 December."

The problem began with an equipment failure at an electrical substation operated by Energex, the spokesperson said, adding that further investigation is underway to determine the cause of the outage.

CITEC had plenty of redundant power options in place - but two generators designed to kick-in when the mains went down both suffered failures.

CITEC's UPS (uninterruptable power system) recognised "extreme instability condition" in the power supply (sustained over-voltage), the spokesman said, and had automatically switched to back-up power.

But problems set in once the UPS switched over to diesel-fuelled generators.

"Two of the generators supplying back-up power subsequently encountered unrelated mechanical faults that resulted in insufficient capacity being available from the remaining back-up power generation supply to sustain the data centre load," the spokesman said. 

"At this point, UPS battery back-up took up the data centre load.  This tertiary back-up supply sustained supply for a further 40 mins enabling emergency shutdown."

 Power was restored and most services were progressively returned to normal by early Friday morning, he said.

"After power was restored, CITEC followed standard data centre practices and procedures. There were no delays in restoring the power to the data centre."  

While infrastructure and network services were affected for most government departments, the CITEC spokesman claimed that nine hours to restore the number of systems involved is "not a long time given the size and nature of the incident."

"CITEC's disaster recovery plan was activated and operated throughout the night to ensure all services were restored as effectively and promptly as practical," he said.