NYC data centres under Sandy storm cloud

 

Updated: Multiple outages.

Post-tropical cyclone Sandy hit US East Coast data centres and networks hard, causing extensive and lasting damage.

Several providers are reporting that their facilities are running on diesel generators currently, with some running out of fuel.

InterNAP has confirmed that its data centre at 75 Broad Street (known as LGA11) lost power after flooding cut fuel supplies being pumped to generators.

"At the peak there was three feet of flood water in the lobby at the site, causing both basement floors to be inaccessible," the firm said in a blog post.

"As a result of the flooding, both our redundant fuel pumps and our generator fuel tank were compromised and shut down.

"We are working as quickly as possible to implement a workaround for the fuel system that will allow us to bring the generator farm back into operation".

Earlier, the text of a customer note from InterNAP's network operations centre was uploaded to Pastebin and various online forums. 

iTnews also sighted a series of separate customer notifications (including this Pastebin) for a "catastrophic power failure" at InterNAP's LGA8 facility at 111 8th. Generator power was restored after two-and-a-half hours.

Twitter reports have suggested that the issue had occurred during "backup generator switchover", but this is yet to be confirmed.

"Our 111 8th Ave. data centre continues to run on generator power successfully," InterNAP said.

"We have enough fuel to last several days and expect fuel delivery to the site will be possible prior to depleting on-site reserves."

The facility at 111 8th is a key internet hub and home to a number of large data centre players.

IP backbone operator Init7 reported that power outages could cause "possible routing issues from / to the United States".

Most data centre problems did not appear to be related to flooding but widespread cuts to mains power in parts of New York.

Some issues are being attributed to a substation explosion in Manhattan, though reports since have blamed flooding of electricity assets for mains power cuts.

Steadfast Networks reported a facility at 121 Varick Street (known as nyc16"went down" and that no ETA was available for its restoration.

"The nyc16 datacenter currently has no electricity service, including supply to the elevator and interior lighting," the company said.

"The Fire Department of New York is currently working to clear the area of water and debris, at which point work will begin to restore utility electricity service and generator service."

Equinix has confirmed it faced issues at its NY9 internet business exchange (IBX), after a network operations centre notification was published in various web forums.

"NY9 experienced a failed generator that impacted service to several customers," chief global operations officer Sam Kapoor confirmed in a statement to iTnews.

"We made repairs and service was returned this morning. The site remains on generator power."

Kapoor said that "all sites in New York/New Jersey, and several in Philadelphia and Washington, DC, experienced power outages and customer loads were transferred to generator power."

This confirmed overnight reports that at least three other Equinix facilities - NY1NY2 and NY4 - had switched to generator power, without issue.

"All of these sites have at least 48 hours of fuel onsite with fuel vendors on standby to deliver more as needed," Kapoor said.

"As of this morning, some of these sites are back on utility power, while some remain on generators."

Renesys said its Portsmouth, New Hampshire data centre experienced "multiple power outages in the last twelve hours". 

Fibre optic network provider Zayo is reported to have advised that workloads in two facilities in which it has a presence — 60 Hudson (NY) and 165 Halsey (NJ) — are on generator power, and that refueling was being arranged. The company has not made an official statement.

NetAccess Corporation posted an outage summary to the North American Network Operators' Group outage mailing list, noting it had facilities on a mix of mains and generator power, and had not had any downtime.

Another facility owned by Datagram, which hosts several media websites, has been out of action since yesterday.

Internet forum Webhostingtalk.com is maintaining a list and extensive thread on data centre outages.

Connectivity battle

New York City is a major international telecommunications and data transit node.

The AC-2 trans-Atlantic cable which lands in Bellport, New York and Tuckerton, New Jersey, reportedly went down as both those areas saw heavy storm damage.

Connectivity issues were last night being reported for a Savvis data centre property in Boston. The company was this morning directing customers to log into a site to view updates.

Web host InterServer reported issues with fibre optic links in the New York area.

Renesys published a graph showing the impact of connectivity cuts on internet traffic by US state:

Source: Renesys

 

Copyright © iTnews.com.au . All rights reserved.


NYC data centres under Sandy storm cloud
 
 
 
Top Stories
Frugality as a service: the Amazon story
Behind the scenes, Amazon Web Services is one lean machine.
 
Negotiating with the cloud email megavendors
[Blog post] Lessons from Woolworths’ mammoth migration.
 
Qld govt to move up to 149k staff onto Office 365
Australia's largest deployment, outside of the universities.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

The great data centre opportunity on Australia's doorstep
The great data centre opportunity on Australia's doorstep
Scott Noteboom, CEO of LitBit speaking at The Australian Data Centre Strategy Summit 2014 in the Gold Coast, Queensland, Australia. http://bit.ly/1qpxVfV Scott Noteboom is a data centre engineer who led builds for Apple and Yahoo in the earliest days of the cloud, and who now eyes Asia as the next big opportunity. Read more: http://www.itnews.com.au/News/372482,how-do-we-serve-three-billion-new-internet-users.aspx#ixzz2yNLmMG5C
Interview: Karl Maftoum, CIO, ACMA
Interview: Karl Maftoum, CIO, ACMA
To COTS or not to COTS? iTnews asks Karl Maftoum, CIO of the ACMA, at the CIO Strategy Summit.
Susan Sly: What is the Role of the CIO?
Susan Sly: What is the Role of the CIO?
AEMO chief information officer Susan Sly calls for more collaboration among Australia's technology leaders at the CIO Strategy Summit.
Meet the 2014 Finance CIO of the Year
Meet the 2014 Finance CIO of the Year
Credit Union Australia's David Gee awarded Finance CIO of the Year at the iTnews Benchmark Awards.
Meet the 2014 Retail CIO of the Year
Meet the 2014 Retail CIO of the Year
Damon Rees named Retail CIO of the Year at the iTnews Benchmark Awards for his work at Woolworths.
Robyn Elliott named the 2014 Utilities CIO of the Year
Robyn Elliott named the 2014 Utilities CIO of the Year
Acting Foxtel CIO David Marks accepts an iTnews Benchmark Award on behalf of Robyn Elliott.
Meet the 2014 Industrial CIO of the Year
Meet the 2014 Industrial CIO of the Year
Sanjay Mehta named Industrial CIO of the Year at the iTnews Benchmark Awards for his work at ConocoPhillips.
Meet the 2014 Healthcare CIO of the Year
Meet the 2014 Healthcare CIO of the Year
Greg Wells named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at NSW Health.
Meet the 2014 Education CIO of the Year
Meet the 2014 Education CIO of the Year
William Confalonieri named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at Deakin University.
Meet the 2014 Government CIO of the Year
Meet the 2014 Government CIO of the Year
David Johnson named Government CIO of the Year at the iTnews Benchmark Awards for his work at the Queensland Police Service.
Q and A: Coalition Broadband Policy
Q and A: Coalition Broadband Policy
Malcolm Turnbull and Tony Abbott discuss the Coalition's broadband policy with the press.
AFP scalps hacker 'leader' inside Australia's IT ranks.
AFP scalps hacker 'leader' inside Australia's IT ranks.
The Australian Federal Police have arrested a Sydney-based IT security professional for hacking a government website.
NBN Petition Delivered To Turnbull's Office
NBN Petition Delivered To Turnbull's Office
UTS CIO: IT teams of the future
UTS CIO: IT teams of the future
UTS CIO Chrissy Burns talks data.
New UTS Building: the IT within
New UTS Building: the IT within
The IT behind tomorrow's universities.
iTnews' NBN Panel
iTnews' NBN Panel
Is your enterprise NBN-ready?
Introducing iTnews Labs
Introducing iTnews Labs
See a timelapse of the iTnews labs being unboxed, set up and switched on! iTnews will produce independent testing of the latest enterprise software to hit the market after installing a purpose-built test lab in Sydney. Watch the installation of two DL380p servers, two HP StoreVirtual 4330 storage arrays and two HP ProCurve 2920 switches.
The True Cost of BYOD
The True Cost of BYOD
iTnews' Brett Winterford gives attendees of the first 'Touch Tomorrow' event in Brisbane a brief look at his research into enterprise mobility. What are the use cases and how can they be quantified? What price should you expect to pay for securing mobile access to corporate applications? What's coming around the corner?
Ghost clouds
Ghost clouds
ACMA chair Chris Chapman says there is uncertainty over whether certain classes of cloud service providers are caught by regulations.
Was the Snowden leak inevitable?
Was the Snowden leak inevitable?
Privacy experts David Vaile (UNSW Cyberspace Law and Policy Centre) and Craig Scroggie (CEO, NextDC) claim they were not surprised by the Snowden leaks about the NSA's PRISM program.
Latest Comments
Polls
Which bank is most likely to suffer an RBS-style meltdown?





   |   View results
ANZ
  20%
 
Bankwest
  9%
 
CommBank
  11%
 
National Australia Bank
  17%
 
Suncorp
  24%
 
Westpac
  19%
TOTAL VOTES: 1493

Vote