Surges cut power to Centrelink data centre

Confirms cause of Easter weekend outage.

Centrelink has confirmed power surges at one its data centres in Canberra caused an emergency shutdown of the facility on the Easter long weekend.

The failure occurred just two months before the release of an Australian National Audit Office report, which exposed weaknesses in Centrelink's data centre and business continuity management procedures.

A spokesman for the agency told iTnews that IT services in the Canberra data centre were taken offline over the Easter weekend for "annual systems maintenance".

The centre is located at Tuggeranong in Canberra's south. Centrelink has a second facility in the Canberra suburb of Bruce and a separate data storage site.

The spokesman said maintenance work was completed and system health checks were being performed "when two power surges from the local grid damaged the uninterruptible power supply (UPS) infrastructure at the centre".

The result was that it did not failover to a back-up power source, causing an outage over the holiday weekend.

The surges led to an immediate shutdown of the centre, in line with the agency's emergency procedures, the spokesman said.

"Repairs were completed well before the system had to be back on-line for resumption of business Tuesday 14 April," he said.

"Subsequent maintenance has further strengthened the UPS arrays to avoid future issues. [But due to the timing], no customers, hardware, data or services were affected."

The Audit office [PDF] and Gershon [PDF] reports have cited Canberra's power grid as an ongoing risk to uptime in Federal Government data centre sites.

The Gershon report said "agencies [had] raised concerns over Canberra's reliance on a single power grid."

Canberra's grid has two feeds from NSW; one provides 85 percent of electricity to the Australian Capital Territory, and the second feed is only available in the outer suburb Fyshwick.

Centrelink hasn't been immune to power issues. The Audit office said it had previously experienced "a number of power outages to its data centres".

These included a December 2007 failure at the Bruce data centre that took services offline for 19 hours and a September 2006 event where primary and secondary systems at the same centre failed.

Centrelink also has other issues, the Audit office report found.

"There is no floor space available for future expansion, and relocation of the Tuggeranong data centre is considered urgent," the report stated.

"There are also air conditioning and other operational issues requiring urgent attention at both [Canberra data] centres."

The Audit office report found Centrelink's business continuity management framework and its implementation "lack the maturity that could be reasonably expected of Centrelink's size and complexity.

The framework is intended to limit or respond to disruptions to business disruptions that could impact the delivery of $70 billion in funds annually.

The Audit office called for improvements to high-level oversight and coordination of the framework and to business continuity planning and testing. Most planning activities still focused on crisis response, it said.

"Centrelink continues to work alongside other Government agencies to ensure the long-term integrity and security of its IT systems and data storage capability," a spokesman for the agency said.

Key points:
* Centrelink's Tuggeranong data centre loses power after twin surges on the Canberra electricity grid.
* The loss left customers and kit unaffected but underlines issues with the ACT grid network.
* UPS arrays have been upgraded to prevent a recurrence.