Diesel generators save the day at Brisbane data centre fire

Diesel generators and power protection systems have come to the rescue of AAPT and WebCentral after a Brisbane-based data centre facility the two companies share caught fire earlier this week.

The facility, in the inner-city enclave of Fortitude Valley, was only restored to mains power yesterday after a fire tore through its basement.

The fire, caused by a failed power transformer operated by utility Energex, began at 11:34am on Tuesday.

Glenn Gore, chief technology officer for Australia's largest web hosting company WebCentral said the power was cut while he was on a conference call.

"That was followed by a very strong smell of electrical burning," he said.

Gore's first instinct was to dash into WebCentral's data centre to see what had gone wrong.

Thankfully, the web hosting provider's UPS system had immediately kicked in, followed by its diesel generator 40 seconds later.

"The fire was not so big, but being that it was an electrical fire, it just gets so hot in a small, contained space," Gore said. "It can be a dangerous situation when you think about how much power the building can consume."

Elsewhere in the same building, AAPT technicians had noticed the blackout. Again, power protection and diesel generators had kicked in to prevent outages on their systems.

"It got pretty ugly for a while," AAPT CIO David Yuile said. "We ran on diesel for a good part of the day."

Disaster Recovery plans tested

By midday, the facility had been evacuated level by level, and the fire brigade had arrived to try and contain the fire.

"We went into Lights Out Managed Mode," said Gore. "Our DR practices kicked in - crucial staff were sent to another office in East Brisbane, where they could remotely manage the data centre. A select number of call centre staff were also sent to that office. While we did have a reduced capacity to take calls, it was better than nothing."

Gore said WebCentral's generator ran from 11:34am to 3:00am Wednesday morning. At that stage, Energex arrived with its own generators to carry the power load through Wednesday until the early hours of Thursday morning. For two further hours WebCentral switched back to its own generator while Energex prepared to switch the mains back on at 2:00am.

To Gore's knowledge, no services were affected.

"Nothing," he said proudly. "Not a glitch. It was a great example of the policies you put in place and hope not to use."

AAPT's Yuile was equally as relieved.

"We got a little bit hot in one of the rooms, but we didn't lose anything - not a customer nor a connection," he said.

"These things can happen any day," he said. "It's all about having the right controls and processes in place. It can be as simple as making sure you have diesel in the tank."

These processes, Yuile said, have to be known, documented and repeatable.

"You have to have well-understood and embedded processes - you can't be making this stuff up as you go along."

Yuile said it wasn't fair for hosting companies or telcos to blame their utility partner every time there was an outage.

"At the end of the day, customers buy a service from us, and we are the ones made accountable" he said.