Macquarie data centre loses water supply

By Ry Crozier on Jun 29, 2009 3:39 PM
Filed under Telco/ISP

Hosting customers offline briefly early on Saturday.

Emergency water repairs near Central Station may have led to a short outage at the Macquarie Data Centre in Sydney early on Saturday morning, affecting its hosting customers.

A spokesman for the data centre said it lost water supply late the night before, affecting its air conditioner.

It switched to secondary and tertiary backup cooling systems and auxiliary power, the spokesman said.

Equipment in the managed services area was turned off after 7am Saturday AEST.

"The managed hardware area was restored and operational by 9am," the spokesman said.

Indications are that a leaking nearby fire hydrant may have cut water services to buildings, although its timing is in dispute.

A spokesman from Sydney Water's maintenance division confirmed emergency-repair crews attended a hydrant but he said it happened on Sunday.

"We had no reactive jobs and no presence in the area [on Friday or Saturday]," a spokesman for the utility said.

Macquarie Telecom said its operations manager informed Sydney Water of the issue early Saturday morning.

The company cited internal operations and its own Sydney Water sources.

Macquarie told its customers, including iTnews, using text messages on Saturday morning when it was clear that the servers would need to be shut down.

Customers on the Whirlpool forums reported shutdown of some servers and indicated they had sent techies onsite to manage other equipment.

The core network was unaffected.

The data centre spokesman told iTnews that Macquarie Telecom was "reviewing [its] engagement processes" with Sydney Water to determine why it wasn't informed of the emergency maintenance sooner.

A post-incident report will be issued shortly.

As a result of the issue, iTnews and sister sites Computer Reseller News, Secure Computing, PC Authority and Atomic went offline briefly Saturday morning. All were restored by 9am.

Macquarie data centre loses water supply

7 comments in this discussion

"I'm kind of assuming that the water was used in water cooled condensers for the air-conditioning. if the main system uses water cooled condensing then it's a fair bet that the backup system uses ..."

By Slatts

Tags

macquarie, telecom, data, centre, sydney, managed, server, technology, business, water

Sydney Water turned off wrong pipe

Defence to kit out 'Building 195' data centre

Sydney Cricket Ground strikes payments data deal

Researchers tap phone data to establish mobility patterns

Breaking Stories

Telstra opposes 700 MHz for emergency 4G network

AusPost, NAB combine on Agile project

Aussie IT workers threatened by skilled migrants: survey

Transfield wins new NBN contracts

G8 countries sign up to Open Data Charter

Comments: 7

Cardy Jun 29, 2009 8:52 PM	I doesn't sound like a big deal when you read this article, but I tell you what, we had servers down all over the place because of this! Not acceptable for a colo of this size! Apparently it was like an oven inside, and I heard that servers cooked to the point of no return.
Sams Jun 30, 2009 9:21 AM	I don't know the first thing about data centre air conditioning, but it seems a bit dodgy to rely on a continuous water supply without some sort of redundant system.
jonsey32 Jun 30, 2009 8:50 PM	"I don't know the first thing about data centre air conditioning, but it seems a bit dodgy to rely on a continuous water supply without some sort of redundant system." You did read this part of the article, didn't you? "It switched to secondary and tertiary backup cooling systems and auxiliary power, the spokesman said." I find it odd that these backup systems couldn't cope with the data centre's heat load. I would have thought that is the whole point of having these systems in place.
Slatts Jun 30, 2009 9:04 PM	Sams wrote: It switched to secondary and tertiary backup cooling systems and auxiliary power, the spokesman said. Sort of like... iTnews wrote: It switched to secondary and tertiary backup cooling systems and auxiliary power, the spokesman said.
Slatts Jun 30, 2009 9:14 PM	Sorry jonsey, didn't mean to step on your post. 100% redundancy is very expensive. I'd imagine they'd go for something that could carry the load for a few hours, Which it sounds like it mostly did.
Sams Jul 1, 2009 12:23 AM	"I find it odd that these backup systems couldn't cope with the data centre's heat load." Well that was my point, although I admit it didn't come out right. It ain't a redundant system if it doesn't actually work.
Slatts Jul 2, 2009 8:54 PM	I'm kind of assuming that the water was used in water cooled condensers for the air-conditioning. if the main system uses water cooled condensing then it's a fair bet that the backup system uses the same method of condensing. i can only assume that the designers of the building didn't foresee the council turning off their tap for several hours without warning. After 3 decades in the trade I can assure you that making a system fool proof is very difficult. Making it idiot proof is bloody impossible.