Datacom CEO speaks about flood damage

 

Affected customers can expect compensation.

Datacom chief executive Michael Browne has confirmed that a flood in the company's Globalcenter data centre in Melbourne on Saturday was caused by hail blocking drains in the roof.

Browne told iTnews today that one of five suites within Globalcenter was damaged by the storm. Services have since been restored, he said.

Affected customers should expect some sort of compensation from the service provider, "depending on the nature of the relationship with Datacom," he told iTnews.

"Needless to say we do provide an uptime guarantee," he said. "I fully expect some clients will want to sit down and chat through this. It is highly probable that uptime availability requirements would have been breached."

Browne said that compensation would not necessarily be in the form of cash and could be awarded via service credits.

Browne wasn't able to confirm network operations centre (NOC) reports that customers experienced a surge in the data centre's power distribution system that knocked out equipment, nor whether overload switches on some older PDUs failed to trigger.

"We cannot confirm on behalf of customers if this is accurate, considering the equipment housed is not owned by Datacom," he said. "We can confirm that any overload current was contained by the switchboard affected."

But after viewing the incident report, Browne said it became clear that affected customers were relying on a single power source for their equipment. "If clients had dual power, the story could have been different."

Browne said Datacom "always recommends" the use of dual power supplies but "ultimately it's the customers prerogative."

Browne said that Datacom staff were first made aware of a potential problem at approximately 2:45pm on the Saturday. Reports from Datacom customers suggest they were first informed within 45 minutes (at 3:30pm).

"I am pleased with the speed and efficiency of the response from Datacom's staff and supporting third party contractors," Browne told iTnews. "We have placed a strong emphasis on communication as part of our critical incident procedure and that has been received very positively by the majority of our clients.

"In many respects we have been one of the more fortunate businesses affected by the storm, noting that it has been acknowledged as an unpredicted and catastrophic weather event."

He said that "by and large the feedback so far has been supportive", with some customers "pleased with the response to date".


Datacom CEO speaks about flood damage
"There are a large number of ISP's, HSP's and Corporates in Datacom and the majority were not affected, We were one of them that had zero interruption to power or crossconnects. For us, Datacom ..."
By sparkymelb
 
 
 
Comments: 7
DJ
Mar 10, 2010 7:36 AM
Who runs a hosting business dependent on a single power rail ??????????

Were there any customers who had equipment damage from water ?
EMwyres
Mar 10, 2010 8:38 AM
Mr Browne - can you explain how you believe customers have been "pleased with the response to date", when initially, you yourself denied there was even a problem/incident occurring? That is seriously poor customer service.
TNS_Inc
Mar 10, 2010 11:06 AM
speaking directly with Datacom Mgmt (Browne) and visiting our kit at the site yesterday, they’ve never denied the incident. They deny flooding (and I didn’t see any evidence of it). Our on-call support received a txt msg from Datacom where the word “flooding” was used, certainly got out attention! We figure their operator was under plenty of pressure to keep people informed during the incident and didn’t chose his words wisely.
TheEvilMuppet
Mar 10, 2010 5:51 PM
@DJ - Good question! I'm not sure how much can be said of whether this is true or not, but some customers in the impacted area apparently lost power dual-powered equipment because of one distribution board feeding both rails. This isn't confirmed in anything that's been issued publicly, but I haven't seen it denied either. As for water, I have lost some minor equipment due to water damage.

@EMwyres - Could be a case of confusion of terms here. The term "outage" was what was denied, so perhaps there's some specific meaning assigned to that term (one that I don't think anyone will agree with). Regardless, a major incident did occur that impacted quite a number of people.

drawkcab
Mar 10, 2010 5:59 PM
@DJ there is some confusion there - but you'd expect that from company management commenting in the media. The area most affected by the power outage has a lot of racks which have dual power rails, but they are fed from the same distribution board. Not entirely sure on the reason for this, probably due geography/distance to the next available distribution board.
I had a rack that was in this situation, both PDU's fed from the same distribution board. So it was out for 12 hours. Protects you from a circuit overload, but not a source failure such as this.

Its a really fubar situation for those that lost kit due to water/power surge damage - but for the lucky ones (and believe me, some of us used up two cat lives with this) that came through still operating - the facility will increase in resiliency and be better off for it.

Its the painfully ironic thing with DC outages, they are so public that the facilities put everything into NEVER letting that-thing-that-happened happen again.

Oh yeah, except for that one place over the river and some shaky generators...
TheEvilMuppet
Mar 10, 2010 7:16 PM
@drawkcab You're right there - the typical operator response to problems like these tends to be very good.

The only examples I can think of where a lot of additional engineering wasn't done in response to an outage/incident are Equinix with a fairly recent power issue (given they're already N+1 and cited a rare failure) and PacNet with a recent, but more total power issue (I don't the facility has the scope to be upgraded).
sparkymelb
Mar 12, 2010 3:28 PM
There are a large number of ISP's, HSP's and Corporates in Datacom and the majority were not affected, We were one of them that had zero interruption to power or crossconnects. For us, Datacom handled this well. We have only a relatively small budget for Datacenter and Connectivity, but even we plan well enough to spread it accross multiple facilities with modest IDC connectivity for failover redundancy. As good as a facility may be... just remember, the worst that could happen may happen anywhere one day, its up to network managers to manage the geographic resiliancy appropriately.
Comments have been disabled for this article.
 
 
 
Top Stories
iTnews on tour: The Executive Summit Series
Join us in Sydney and Melbourne to meet Australia's tech leaders.
 
NBN Co braces for secrecy probe
Opinion: Is commercial sensitivity a catch-all?
 
Exclusive: NBN Co withholds fibre upgrade price
Fears trial details could breach contractor confidentiality.
 
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

Latest Comments
Polls
Was your 2012 IT budget...




   |   View results
Cut by less than ten percent?
  17%
 
Cut by more than ten percent?
  34%
 
Flat
  26%
 
Increased by less than ten percent?
  7%
 
Increased by more than ten percent?
  15%
TOTAL VOTES: 350

Vote
Will you still use DropBox and other cloud storage in the wake of the Megauploads saga?

   |   View results
Yes
  63%
 
No
  37%
TOTAL VOTES: 230

Vote