Snafus forced Twitter data centre move

An ambitious plan to prevent "Fail Whales," the cartoon icon that greets frustrated Twitter users during network outages, turned into a fail whale of its own.

A new, custom-built facility in Utah meant to house computers that power the popular messaging service by the end of 2010 has been plagued with everything from leaky roofs to insufficient power capacity, people familiar with the plans told Reuters.

The botched move threatened new product development and forced Twitter -- whose user accounts have burgeoned to 200 million in just five years -- to seek another location despite committing significant investment to the facility.

The data centre move reflects growing pains faced by red hot tech start-ups as they aim to justify soaring valuations and transform into reliable channels for advertising and commerce.

Twitter said last month its data centre had moved to a new home at an undisclosed location -- a feat that industry insiders say is impressive by any measure.

People familiar with the matter said the move was to an existing Sacramento facility more than 600 miles away, owned by co-location company Raging Wire, rather than into the custom-built data centre in Salt Lake City, Utah.

Twitter had signed a US$24 million, 4-year minimum commitment lease with C7 Data Centres which was building the Utah centre. It was not immediately clear how much, if any, of the fees would be returned or if it planned to pay out the full term of the contract.

Those plans "spiralled out of control," one source said. "They were in over their heads and they did not recognise that until far too late."

Twitter declined to comment about its data centres or finances. Twitter Vice President of Engineering Michael Abbott said the company has done more to upgrade its infrastructure in the last six months than it did in the previous 4.5 years.

"Twitter now has the team and infrastructure in place to capitalize on the tremendous interest in Twitter and continue our record growth," Abbott said in an email.

Large Internet companies such as Google and Yahoo spend anywhere from US$50 million to US$2.5 billion to construct and equip such data centres, according to Forrester Research analyst James Staten.

Water hazards

In December, Twitter was valued at US$3.7 billion in a US$200 million funding round led by venture capital firm Kleiner Perkins Caufield & Byers.

An auction of Twitter shares on the secondary market last month suggested investors were valuing the company at more than US$7 billion.

Last month, Twitter hired eBay's Mazen Rawashdeh as vice president of operations -- a job that sources say didn't exist at Twitter in July 2010 when it announced plans to move its operations infrastructure.

For Twitter, which had been renting servers from hosting provider NTT America for the past several years, moving into its own data centre marked a significant milestone.

Within a couple of months of Twitter's July announcement though, it was quietly moving many of its servers out of Salt Lake City and instead shipping all new gear to Sacramento, where server capacity and space was initially very tight.

Twitter concluded that the Utah facility failed to meet its needs, these sources said. The centre initially lacked key features such as a second fibre network connection, and less than half of the electricity was actually available.

The roof leaked water onto the top of Twitter's server cabinets with every rain, forcing staffers to move equipment out of harm's way, sources said.

C7 President Wes Swenson declined to confirm Twitter as a customer but defended his company's work at the centre. He said 5 megawatts of power were available from Day One and the building had a brand new roof once construction was over in October.

"We experience less than 1 percent churn in our customer base," Swenson said in an interview. "But sometimes we deal with companies that are not very sophisticated; often times a lot of customers that have never owned their own equipment may go through a learning curve."

Twitter's change of plans to move its data centre to Sacramento affected product plans as management enforced a several month moratorium on launching new major features.

"Minor stuff was OK," said the person familiar with the matter, but "if you developed a new feature which needed 20 servers, you just weren't going to get the 20 servers."

For now, Twitter is using the remaining servers for limited roles, such as providing analytics functions for its service.

"Nobody wants to put any part of the site that's mission critical in Salt Lake City," said another source familiar with the matter. "That would be a huge battle royale inside the company if they were told they had to."

(Reporting by Alexei Oreskovic; Editing by Kenneth Li and Richard Chang)