Macquarie data centre loses water supply

 

Hosting customers offline briefly early on Saturday.

Emergency water repairs near Central Station may have led to a short outage at the Macquarie Data Centre in Sydney early on Saturday morning, affecting its hosting customers.

A spokesman for the data centre said it lost water supply late the night before, affecting its air conditioner.

It switched to secondary and tertiary backup cooling systems and auxiliary power, the spokesman said.

Equipment in the managed services area was turned off after 7am Saturday AEST.

"The managed hardware area was restored and operational by 9am," the spokesman said.

Indications are that a leaking nearby fire hydrant may have cut water services to buildings, although its timing is in dispute.

A spokesman from Sydney Water's maintenance division confirmed emergency-repair crews attended a hydrant but he said it happened on Sunday.

"We had no reactive jobs and no presence in the area [on Friday or Saturday]," a spokesman for the utility said.

Macquarie Telecom said its operations manager informed Sydney Water of the issue early Saturday morning.

The company cited internal operations and its own Sydney Water sources.

Macquarie told its customers, including iTnews, using text messages on Saturday morning when it was clear that the servers would need to be shut down.

Customers on the Whirlpool forums reported shutdown of some servers and indicated they had sent techies onsite to manage other equipment.

The core network was unaffected.

The data centre spokesman told iTnews that Macquarie Telecom was "reviewing [its] engagement processes" with Sydney Water to determine why it wasn't informed of the emergency maintenance sooner.

A post-incident report will be issued shortly.

As a result of the issue, iTnews and sister sites Computer Reseller News, Secure Computing, PC Authority and Atomic went offline briefly Saturday morning. All were restored by 9am.


Macquarie data centre loses water supply
"I'm kind of assuming that the water was used in water cooled condensers for the air-conditioning. if the main system uses water cooled condensing then it's a fair bet that the backup system uses ..."
By Slatts
 
 
 
Comments: 7
Cardy
Jun 29, 2009 8:52 PM
I doesn't sound like a big deal when you read this article, but I tell you what, we had servers down all over the place because of this!

Not acceptable for a colo of this size!

Apparently it was like an oven inside, and I heard that servers cooked to the point of no return.
Sams
Jun 30, 2009 9:21 AM
I don't know the first thing about data centre air conditioning, but it seems a bit dodgy to rely on a continuous water supply without some sort of redundant system.
jonsey32
Jun 30, 2009 8:50 PM
"I don't know the first thing about data centre air conditioning, but it seems a bit dodgy to rely on a continuous water supply without some sort of redundant system."

You did read this part of the article, didn't you?

"It switched to secondary and tertiary backup cooling systems and auxiliary power, the spokesman said."

I find it odd that these backup systems couldn't cope with the data centre's heat load. I would have thought that is the whole point of having these systems in place.
Slatts
Jun 30, 2009 9:04 PM
Sams wrote:
It switched to secondary and tertiary backup cooling systems and auxiliary power, the spokesman said.


Sort of like...

iTnews wrote:
It switched to secondary and tertiary backup cooling systems and auxiliary power, the spokesman said.

Slatts
Jun 30, 2009 9:14 PM
Sorry jonsey, didn't mean to step on your post.
100% redundancy is very expensive. I'd imagine they'd go for something that could carry the load for a few hours, Which it sounds like it mostly did.
Sams
Jul 1, 2009 12:23 AM
"I find it odd that these backup systems couldn't cope with the data centre's heat load."

Well that was my point, although I admit it didn't come out right. It ain't a redundant system if it doesn't actually work.
Slatts
Jul 2, 2009 8:54 PM
I'm kind of assuming that the water was used in water cooled condensers for the air-conditioning.
if the main system uses water cooled condensing then it's a fair bet that the backup system uses the same method of condensing. i can only assume that the designers of the building didn't foresee the council turning off their tap for several hours without warning.
After 3 decades in the trade I can assure you that making a system fool proof is very difficult.
Making it idiot proof is bloody impossible.
Comments have been disabled for this article.
 
 
 
Top Stories
Telstra shifts BigPond email to Windows Live
All data to be migrated to Microsoft cloud.
 
Windows 8: Under the hood
Part One of iTnews' enterprise guide to Windows 8.
 
iTnews on tour: The Executive Summit Series
Join us in Sydney and Melbourne to meet Australia's tech leaders.
 
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

Latest Comments
Polls
Would you be concerned about your business' email data being hosted offshore?

   |   View results
Yes
  90%
 
No
  10%
TOTAL VOTES: 62

Vote