Salesforce shuns virtualisation for multi-tenancy

Salesforce.com has passed up traditional virtualisation software for custom technology that puts up to 10,000 customers on what is essentially a single virtual machine.

The company has a total of five production data centres in the US, Japan and Singapore to support more than 100,000 customers and their 400,000 applications.

According to Salesforce’s head of infrastructure Steve Fisher, the facilities share a common architecture, built on commodity servers, storage and switches and custom software.

“We use Dell app servers, Dell servers for our databases – all x86 stuff. We use F5 load balancers, Cisco SAN switches, storage from EMC and Hitachi,” Fisher told iTnews.

“It’s your typical [hardware] stuff. It’s really our software that’s highly differentiated.”

Virtualisation products from the likes of VMWare and Microsoft typically improve IT efficiency by allowing organisations to pool physical server resources and present them as multiple isolated environments.

Fisher said Salesforce was even more efficient by treating all customer data as part of a single application server and database, while presenting customers with the “illusion” of their own environments.

“When you’re a customer, you think you’ve created your own database table, but … that never actually shows up as a table in our database. It is all done through metadata,” Fisher said.

“The only thing that’s customer-specific anywhere in our infrastructure is just records in a database.”

“What virtualisation is great for is taking the traditional, single-tenant architecture and making it more efficient,” he said.

“We’ve basically built our platform for multi-tenancy from the ground up. From the [Salesforce] operations team’s point of view, it’s really just one application.”

Fisher said Salesforce had “well under 10,000” servers across all its production data centres worldwide – “orders of magnitude” less than if it had used traditional virtualisation.

The platform allowed Salesforce to aggregate and provide for the peak demand of tens of thousands of applications, rather than having to treat the peak demand of each application separately.

“The efficiencies are kind of unbelievable. We’re running 400,000 applications, I don’t know how many millions of users ... [and] we do that in with just a few thousand servers,” he said.

“Even in a full virtualised model, every application would [require] a couple of app servers for redundancy, a couple of database servers, and then we would have to replicate that.

“So we’re talking about eight virtual machines times 400,000. That is not going to run on a few thousand servers.”

Limiting risk

Infrastructure sharing may have economic benefits, but security experts have warned that it could expose organisations to the risk of collateral damage from cyber attacks.

A single platform could also mean that any disruptions – whether due to hardware, software, or personnel – are felt service-wide.

Salesforce has attempted to limit the effect of any disruptions by splitting its capacity into “pods” of 5000 to 10,000 customers.

The company has 15 pods in North America, two in the Asia Pacific region and three in Europe, the Middle East and Africa.

“A pod is a unit of multi-tenant capacity,” Fisher explained. “Any one customer lives on an individual pod and that pod is replicated to another data centre.

“We’re big believers in multi-tenancy … but it is nice that if something catastrophic should go wrong, it would only affect customers on that one pod.”

Fisher pointed to a storage failure in its North American ‘NA2’ pod that disrupted services for seven hours in late June.

“We had an incident on NA2 which was a failure in our storage subsystem and it did exactly what you would expect. There was a disruption on NA2 but it didn’t disrupt anybody else,” he said.

“Clearly, that was not good for customers on NA2 but it was certainly good that we had that architecture as opposed to having datacentre-wide outages or service-wide outages.”

Read on to page two to find out how Salesforce manages capacity and what it might look for in an Australian data centre.

Salesforce adopted the pod model in 2006, to assist with capacity planning.

Fisher described pods as “self-contained units”, each consisting of application servers, database servers, search servers, storage, networking and load balancers.

Each pod houses a mix of large, medium-sized and small customers to balance demand and is replicated in another geography.

“If you don’t create pods and you’re just adding capacity then you’re assuming that every part of your stack can scale infinitely,” he explained.

“You’re assuming that you can always add more app servers, always add more database servers … the world of technology doesn’t really work that way.”

Salesforce deploys new pods in accordance with its predictive models and allocates new customers to regional pods automatically.

Pods may also be split should existing customers demand more capacity. This July, Salesforce split the San Jose-based NA1 into two, moving half its tenants to a new NA13 pod in Chicago.

Salesforce’s own instance – used by Fisher and other internal staff – remained on NA1. But customers who moved to Chicago “just didn’t care” about being moved, he said.

“They shouldn’t care – that’s our goal,” he said. “It’s the cloud; you should just access it and use it and you shouldn’t really care where it’s located. And for the most part, I think that’s really the experience that our customers have.”

Salesforce’s data centre locations have been a sticking point for Australian organisations in previous years, due to concerns over the legal jurisdiction of data stored offshore.

Communications minister Stephen Conroy and finance sector regulator APRA have separately called for caution over cloud adoption.

Just two years ago, APRA stepped in to apply pressure on Perpetual Private Wealth after the firm revealed that it had implemented a Salesforce customer relationship management system hosted in Singapore. ANZ Bank removed some of its Salesforce applications due to the same pressures.

But the sector appears to have overcome its initial misgivings, with the Commonwealth Bank, Bank of Queensland and CGU Insurance outing themselves as users of Salesforce’s customer relationship management system this year.

Data centre wish list

Fisher said that Salesforce chief executive officer Marc Benioff planned to build out an Australian data centre but “we don’t know when”.

In the nearer term, he said the company would expand its US footprint to meet capacity demands and look to establish its next data centre in Europe.

Any new data centres would use Salesforce’s common data centre architecture and be built in partnership with “wholesale data centre providers”.

Salesforce colocates much of its infrastructure in Equinix data centres overseas, but Fisher said the company has moved to a wholesale model under which it managed its IT infrastructure at lower cost.

“It’s a much better price if you want to manage it all yourself. We pretty much do all the data centre stuff ourselves [but] we wouldn’t look to buy land and build our own thing – certainly not in Australia,” he said.

Fisher said the company would consider internet connectivity, power costs, seismic activity and the risk of natural disasters,  plus the proximity of its hardware suppliers when seeking out a new site.

Salesforce claims its data centres run at a power usage effectiveness (PUE) ratio of 1.53.