Dev like Netflix: Top tips from the world's savviest engineers

Automate everything

Glover’s top tip is to turn everything into an automated deployment pipeline.

“Anytime a task is done more than once, consider that an opportunity to make it a pipeline,” he said.

It is easy to overlook one-off infrastructure management tasks or occasional updates which seem easier to just complete, than to codify in a repeatable process, he said. 

But unless you try to catch them all you risk creating “towers of knowledge” limited to one or two people who have expertise in a particular area - and no-one to turn to when those individuals are unavailable.

“By codifying this automation in the pipelines, anyone can run them with the benefit of consistency," Glover said.

Be ready for bad code

Despite all of its toolsets and best practice standards, the Netflix team acknowledges that sometimes it will deploy bad code into production.

This code will even get past its Automated Canary Analysis (ACA) - Netflix’s “last line of defence” that channels a small amount of traffic to the last deployed version of software and the soon-to-be-deployed new version simultaneously. It compares their performance and gives the new code a score out of 100 in terms of how it behaves.

Engineers can build gates into Netflix’s deployment platform - called Spinnaker - that automate a go or no-go threshold based on these ACA scores.

And even if this goes sideways, Spinnaker features a one-click rollback button that instantly reverts to the old version of software until the new version passes health checks.

“As much customer testing as you can do, occasionally bad code still gets out. When that happens we want to make sure it is as easy to shift back to the good code as fast as possible so we don’t ruin the customer experience,” Hahn said.

"Bad things will happen, you can’t stop them. We try to limit how long it takes to fix them," Glover said.

Blameless post-mortems

Nearly all of the wisdom in Netflix’s deployment practice comes from deep-dive sessions that analyse failures while eschewing finger-pointing.

“We conduct blameless post-mortems so we can learn from mistakes and not repeat them," Glover said.

“Many of [our] best practices ... have come right out of these post-mortems." 

Paris Cowan travelled to AWS Re:Invent as a guest of Amazon Web Services