How Facebook ships software at warp speed

Australian-raised engineer Joel Pobar is part of a team that "performs heart surgery" on the software infrastructure that runs Facebook.com every two weeks.

Pobar's job is to incrementally improve the HipHop Virtual Machine (HHVM) - a programming language virtual machine built by Facebook which is akin to Java or .Net.

Crudely speaking, HHVM was built to ensure Facebook's servers wouldn't melt under the load of millions of simultaneous users. It took two years and three competing teams to build, and even today is never entirely 'stable' in the sense that the project to improve it never ends.

Pobar's team updates the HHVM at least once every two weeks, and the HHVM is more or less overhauled completely every 12 months.

"Imagine," Pobar described for iTnews' IT manager readers, "building a new version of Java every two weeks, and deploying that to every one of your servers."

Pobar - who had previously developed software for both Sun Microsystems and Microsoft in what was arguably their halcyon days - had never experienced the demands and agility of Facebook's dev and test process. This week he returns to Australia to share his insights into how local organisations might learn from the experience.

His presentation, "Move fast and ship things", will be heard for the first time today during the Melbourne leg of the YOW developer conference.

"I'm here to bring home what I've learned about shipping software at lightning speed," he told iTnews in the weeks prior to the event.

Speed culture

HHVM is but one tool of many that helps Facebook's software developers ship code up to twice a day to tens of thousands of servers across several data centres.

Changes to Facebook.com tend to be small and incremental, so much so that at times thousands of users are piloting them unawares. But always, Facebook's team is observing and measuring to determine which features should be rolled out to its billion-plus users and which should quietly disappear.

Facebook boasts one of the best resourced software teams in the world, and some of the dev and test tools it has developed in-house could arguably be successful products in their own right. Indeed, several, like HHVM, have been released on GitHub as open source projects. But in Pobar's view, the speed at which Facebook improves its online services is as much a question of culture as resources.

First, he said, Facebook management don't micro-manage the development process or prescribe how new features should be built, but instead give consistent, clear messages to the team about where they would like the product to move.

"They are accountable for moving the needle in the right direction for the company," he said. "It is always clearly articulated how our infrastructure goals translate into Facebook's vision."

Facebook also tolerates a degree of failure. Goals are set high enough that - as often as 50 percent of the time - developers might miss them.

Further, project teams are at times set up to attempt diverging solutions to a stated problem, all in the knowledge that the slowest of them at any stage will be asked to peel off to work on something more impactful. The problem HHVM was built to solve was also attacked by two competing projects - the authors of which would feel no shame about conceding defeat in the face of a superior option.

To reinforce this culture, every two to three months developers down tools for a hackathon and are asked to build a prototype for a new idea from the ground up. Many of these ideas make it into production. Software engineers might also be plucked out of one project and placed in another for short bursts of activity, simply to keep an open mind to different ways of approaching a problem.

Joel Pobar addresses his fellow developers

"We embrace experimentation," Pobar said. "In order to do experiments, you need fast, effective feedback loops. You need to not only tolerate but embrace failure.

"I remember being scared of that concept at first - by the second day of boot camp I was told I will ship code this week to 600 million users. That is a frightening thing to hear."

What tools does Pobar's team use to achieve this speed? Read on to find out...