NYC data centres under Sandy storm cloud

Updated: Multiple outages.

Post-tropical cyclone Sandy hit US East Coast data centres and networks hard, causing extensive and lasting damage.

Several providers are reporting that their facilities are running on diesel generators currently, with some running out of fuel.

InterNAP has confirmed that its data centre at 75 Broad Street (known as LGA11) lost power after flooding cut fuel supplies being pumped to generators.

"At the peak there was three feet of flood water in the lobby at the site, causing both basement floors to be inaccessible," the firm said in a blog post.

"As a result of the flooding, both our redundant fuel pumps and our generator fuel tank were compromised and shut down.

"We are working as quickly as possible to implement a workaround for the fuel system that will allow us to bring the generator farm back into operation".

Earlier, the text of a customer note from InterNAP's network operations centre was uploaded to Pastebin and various online forums. 

iTnews also sighted a series of separate customer notifications (including this Pastebin) for a "catastrophic power failure" at InterNAP's LGA8 facility at 111 8th. Generator power was restored after two-and-a-half hours.

Twitter reports have suggested that the issue had occurred during "backup generator switchover", but this is yet to be confirmed.

"Our 111 8th Ave. data centre continues to run on generator power successfully," InterNAP said.

"We have enough fuel to last several days and expect fuel delivery to the site will be possible prior to depleting on-site reserves."

The facility at 111 8th is a key internet hub and home to a number of large data centre players.

IP backbone operator Init7 reported that power outages could cause "possible routing issues from / to the United States".

Most data centre problems did not appear to be related to flooding but widespread cuts to mains power in parts of New York.

Some issues are being attributed to a substation explosion in Manhattan, though reports since have blamed flooding of electricity assets for mains power cuts.

Steadfast Networks reported a facility at 121 Varick Street (known as nyc16) "went down" and that no ETA was available for its restoration.

"The nyc16 datacenter currently has no electricity service, including supply to the elevator and interior lighting," the company said.

"The Fire Department of New York is currently working to clear the area of water and debris, at which point work will begin to restore utility electricity service and generator service."

Equinix has confirmed it faced issues at its NY9 internet business exchange (IBX), after a network operations centre notification was published in various web forums.

"NY9 experienced a failed generator that impacted service to several customers," chief global operations officer Sam Kapoor confirmed in a statement to iTnews.

"We made repairs and service was returned this morning. The site remains on generator power."

Kapoor said that "all sites in New York/New Jersey, and several in Philadelphia and Washington, DC, experienced power outages and customer loads were transferred to generator power."

This confirmed overnight reports that at least three other Equinix facilities - NY1, NY2 and NY4 - had switched to generator power, without issue.

"All of these sites have at least 48 hours of fuel onsite with fuel vendors on standby to deliver more as needed," Kapoor said.

"As of this morning, some of these sites are back on utility power, while some remain on generators."

Renesys said its Portsmouth, New Hampshire data centre experienced "multiple power outages in the last twelve hours". 

Fibre optic network provider Zayo is reported to have advised that workloads in two facilities in which it has a presence — 60 Hudson (NY) and 165 Halsey (NJ) — are on generator power, and that refueling was being arranged. The company has not made an official statement.

NetAccess Corporation posted an outage summary to the North American Network Operators' Group outage mailing list, noting it had facilities on a mix of mains and generator power, and had not had any downtime.

Another facility owned by Datagram, which hosts several media websites, has been out of action since yesterday.

Internet forum Webhostingtalk.com is maintaining a list and extensive thread on data centre outages.

Connectivity battle

New York City is a major international telecommunications and data transit node.

The AC-2 trans-Atlantic cable which lands in Bellport, New York and Tuckerton, New Jersey, reportedly went down as both those areas saw heavy storm damage.

Connectivity issues were last night being reported for a Savvis data centre property in Boston. The company was this morning directing customers to log into a site to view updates.

Web host InterServer reported issues with fibre optic links in the New York area.

Renesys published a graph showing the impact of connectivity cuts on internet traffic by US state:

Source: Renesys