QLD data centres operating despite floods

Updated: AAPT confirms flood damage, Pipe threat eases.

Update 5.10PM, 13/1: AAPT status updated.

Queensland's major data centres operators indicated today that they had so far been spared the impact of rising flood waters in the state with only AAPT suffering water ingress.

Nextgen Networks, Polaris, Pegasus/Dedicated Servers, iSeek, Digital Sense, Brennan, Melbourne IT, Host Networks and Over The Wire all indicated they were operating.

AAPT had water enter one of its Brisbane points of presence and the basement of a co-location facility.

In addition, Pipe Networks' Creek Street facility had been under threat of losing mains power, but the latest issue advisories indicated the facility was now likely to escape the flood threat.

Most facilities, however, were unconcerned about flood damage as they had taken steps to ameliorate any potential threat, either in disaster recovery plans or when the facilities were built.

Nextgen data centre

"Nextgen's Brisbane data centre which includes a major node and our customer co-lo is still operating on mains power and above the flood level," a spokesman for the company said a short time ago.

"Indications are that this data centre is approximately nine meters above the expected flood peak. In fact it is above the level of a Brisbane official evacuation centre.

"Power is not yet cut. We have more than three days generator fuel on site and are arranging for a fuel trailer to be put on site to keep the building fuel tanks topped up as a contingency. As this is a N+1 redundancy site, we also have a spare generator."

Pegasus

Statements attributed to Dedicated Servers indicated that the Pegasus data centre could also be spared.

"We wish to assure you that our data centre is not only on one of Brisbane CBD's geographically elevated areas, but also a number of storeys up into the building," the statement, pasted in part to the Whirlpool broadband forums, read.

"Contrary to other data centres, this also includes our power infrastructure, which is located above our carpark, which incidentally remains dry.

"Equipment is safe from the water at the moment, and we have no concerns of that changing."

Polaris data centre

Strategic Directions Group director Mike Andrea told iTnews that the Polaris data centre at Springfield was fully operational.

"The Polaris site and its underground services are well above the 1 in 100 flood level for the area, and it is not located in the vicinity of any rivers or general flooding," Andrea said.

"David Goss [Polaris' facilities manager] confirmed there is no immediate threat to the premises or services from either flooding or land subsidence."

Andrea said that Polaris had enough diesel and water stored on site "to sustain full operations for a period up to 96 hours (4 days) should mains power or water be affected".

"Additional fuel supply delivery plans are in place should they be required," he said.

"While Energex has informed the region that areas at risk around Brisbane and Ipswich will have mains power shut-off for safety reasons, the affected areas and services will not impact power supply to [us]."

Andrea said there were some reports of "small localised flooding" in neighbouring suburbs but that road access to Polaris was good.

He said that Polaris' facilities management team had "invoked a staffing contingency plan should road access be limited, in order to continue delivery of critical services 24x7 during this weather event."

Andrea also indicated that dark fibre links to and from Polaris were operating as usual.

iSeek

Facilities manager Brett Cain told iTnews that iSeek's data centre facilities and communications network were under "no immediate threat".

"The physical location of our facilities were chosen with the knowledge of historical flood records and are in locations where flooding had not been previously recorded," Cain said.

"We have also been watching the constantly updated maps released by local authorities and we are in no immediate threat."

Cain said that iSeek's facilities had bund (brick) walls around key infrastructure and that the company had "taken additional action to protect core equipment."

Staff remained onsite at core points of presence and facilities and iSeek had set up an alternate phone number for customer support - 1300 139 161 - which was operating in addition to the regular support numbers.

There were still no reported issues as of 4pm.

Over The Wire

A spokesman for Over The Wire told iTnews its data centres 1 and 2 in Spring Hill were "well above the expected flood areas".

"In addition, power is fed from Spring Hill based substations, again well above the CBD areas," the spokesman said.

"There is no current expected mains power outages. In addition, there is [about] 48 hours of diesel [fuel] onsite [for the generators]."

The spokesman said that hosted voice and SiP trunking services were online. He also said the company had seen a "number of customers re-locate equipment this morning from their offices to our facilities so that staff can continue to work remotely."

Pipe Networks

Excerpts of an issue advisory from Pipe Networks were pasted to various forums, indicating that the company's DC1 and DC2 centres in Creek Street "will be one of the last buildings to be disconnected from the grid by Energex as it houses a critical QLD Government department co-ordinating the SES response to the flood."

"Energex have advised that if the flooding reaches 15-20cm of standing water in the lower basement car park of 127 Creek St, all power (including the generators) will be cut to the building," the advisory reportedly read.

"This will result in DC1 & 2 going dark until such time as Energex advise us that power can be safely restored to the building. Worst-case scenarios puts this time at 48 hours."

An earlier threat to the facility's access to mains power appeared to have subsided late tonight with Pipe Networks advising that today's high tide peak in flood waters at 4.6 metres had left flood waters about 350 metres from DC1 and DC2.

Tomorrow's flood peak has also been revised down 300 mm to 5.2 metres -

Earlier, the company said it had kept two employees on site.

"If we have a total power failure we will endeavour to provide an urgent update to allow you time to access your equipment and gracefully shut it down before the UPS batteries are drained."

AAPT

Telco AAPT had water enter one of its Brisbane points of presence and the basement of a co-location facility, but the latter had remained online courtesy of back-up power, according to chief executive Paul Broad.

Earlier, rumours had circulated that two - possibly three - AAPT facilities in Brisbane had been downed by the flood crisis.

One source told iTnews that the carrier's 344 Queen Street facility had been running on generator power.

A separate issue advisory posted to the Whirlpool forums indicated the carrier's Riverside colocation facility in Eagle Street had lost both mains and battery back-up power.

"Due to the flooding of the Brisbane region, mains power to the facility at the Riverside Centre site has been shut down as a safety requirement," the advisory stated.

"The site is currently running on battery backup. Back-up battery power to the facility has now failed and network infrastructure devices have powered down," the carrier said in a revised advisory.

A report by news site Delimiter claimed that the remaining AAPT facility, also in Eagle Street, was powered down this morning.

Host Networks

A Host Networks spokesman told iTnews that its carrier-neutral data centre in Eight Mile Plains was “at this point unaffected by the floods in Brisbane”.

The spokesman said the company was “not aware of any reasons why our electricity supply will be impacted” by the floods.

“Our distribution substation is also on the top of the hill in the area, which is above our height,” he said.

The spokesman said that the facility had no issues with mains water; that it had “at least 12 hours of diesel [fuel] on-site and [that] all roads in and out of the facility were clear and accessible.”

Digital Sense

Digital Sense's data centre at Kenmore near Brisbane was "fully operational", according to a company spokesman.

"We're expecting it to be completely operational during these testing times," he said.

"While we have lost some carrier links into Ipswich, both our geographical and physical diversity for communications still remains intact."

The company had already implemented a disaster plan called 'Ready For Action', which the spokesman said was designed to allow customers of other affected facilities to relocate to Digital Sense.

"Part of our action plan for major events such as flooding includes items such as restocking of fuel, food and water (for our operations personal) to see us entirely through the floods," he said.

"We ordered enough diesel fuel ahead of the disaster to ensure a full 110 hours (or 4.5 days) of operating capacity could be acheived without further refueling.

"This would allow time for flooding to recede and time for delivery queues to clear should the stocks run low (as they do for these kind of events) and we require a top up."

Melbourne IT

A spokesman for Melbourne IT told iTnews that the company's Wickham Street data centre had "not experienced any power cuts so far" and that the facility was continuing to operate as normal.

"We have generators standing by should the power be cut to the Wickham Street data centre as part of Energex's safety precautions around the CBD," the spokesman said.

"As the data centre is located at the higher end of Wickham Street we are not at this stage expecting flooding around our building", he said, adding that Brisbane City Council flood prediction maps appeared to confirm this.

The spokesman said that most staff were managing systems remotely but the company was keeping some critical staff "on-site and close by should we need to bring them in".

"We'll keep monitoring the situation but as it stands right now we are hopeful that we can get through this flood situation without incident," he said.

Brennan

Brennan managing director Dave Stevens said that the company's data centre and point of presence in Brisbane were expected to escape the flood threat.

"Our data centre is not located within the areas where flood warnings have been issued and are not under immediate threat. However we are monitoring the situation closely," he said.

"We are not on the CBD power grid where issues are expected - we are on the Royal Brisbane Hospital power grid.

"We also have the added security of having in place diverse paths over multiple carriers."

Stevens said the company had back-up generators with a fuel load capable of operating the data centre for 36 hours.

Further information was being sought from other data centre operators.

Stay tuned to iTnews for the latest on the Queensland flood crisis.