Data centres still shaky in Sandy's aftermath

Parts of the 'carrier hotel' at 111 8th Avenue in New York experienced issues as lack of mains power continued to test data centre facilities in the aftermath of superstorm Sandy, which has now claimed 93 lives.

Equinix and fibre optic network operator Zayo, which host points-of-presence in suite 734, separately reported generator-related problems this morning Australian time.

A status update by Zayo that temperatures in the suite rose above 100 degrees fahrenheit (37.7 degrees Celsius) was widely reported, though the operator noted it had since been able to stabilise temperatures "at 95-100 degrees" and had not seen IT equipment fail.

The company switched equipment over to battery power while the generator powering the suite was taken offline. The battery array is not shared with the suite's other tenant, Equinix.

"The root cause of the problem with the generator was a faulty isolation valve, allowing air into the fuel supplied to the generator, resulting in sub-optimal operation," the firm noted.

"The generator was no longer able to support the load of both the networking equipment and the site air conditioning systems.

"A temporary repair of the isolation valve was attempted, which marginally improved fuel flow. The generator will need to be taken offline to implement permanent repairs and Zayo is coordinating with the landlord to perform permanent repairs."

In the interim, Zayo said it was trucking in a 2 megawatt generator that would be parked on the street. Electrical engineers were preparing a way to tie the generator into electrical infrastructure. The suite is on the 7th floor.

Equinix, whose NY9 site is hosted in suite 734, had not fared so well with the generator problems.

"Over the last two days our NY9 site, which is housed in a multi-tenant building with shared generator infrastructure and fuel service, has experienced three outages due to generator failures that impacted service to several customers," Equinix's regional operating chief for the Americas, Raouf Abdel, noted.

"Repairs have been made and service was restored. The site remains on generator power and we are monitoring the situation closely to minimise impact to customers."

Other network and colocation operators with a presence in 111 8th Avenue have indicated they have escaped the impact of the malfunctioning generators.

"Internap would also like to confirm that we are NOT experiencing any issues with our generators or pumps onsite at this time," the firm noted in a blog post.

"We cannot speak for the other tenants in the building."

Internap noted it was operating on "stable generator power" and said that utility power could be restored to the building within two to three days.

Buckets to Broad

A team of more than a dozen people carried five-gallon (19-litre) buckets of diesel fuel up 17 flights of stairs to keep a rooftop generator powering equipment in Peer 1 Hosting's data centre at 75 Broad Street, according to a report by the Huffington Post.

The so-called "bucket brigade" operated for a number of hours, restarting several times to keep generator power on.

A customer who helped in the brigade indicated an hour ago that the host had managed to get a fuel pump working, replacing the "man-power" effort.

Peer 1 did not reference the operation in hourly status updates, which noted only that the facility was still under generator power.

Internap —which also has a presence at 75 Broad — noted the building was still off-limits for its customers, though equipment was operating on "stable generator power" and the company had access to five days worth of fuel.

Though it indicated mains power could return to 75 Broad within three days, it was unsure. "We are still gathering details [on restoration]," it noted.

Varick Street returns

Steadfast Networks reported a short time ago that it had secured a temporary generator and cooling system for its nyc16 facility at 121 Varick Street.

"Critical systems are beginning to come back online," the firm noted.

Sandy knocked out mains power to the building and damaged generators beyond repair. Customer systems were expected to come online again shortly.

Though data centres were impacted, internet systems proved largely resilient to the storm.

Read initial reports of Sandy's impact on data centre infrastructure here.