Domain does DevOps

The IT team at the Domain Group has dramatically reduced the time it takes to deploy production changes to its real estate listings site, after splitting its IT operations out of the broader Fairfax Media Group and embracing DevOps.

The Domain Train lets developers know when changes are deployed to production servers.

DevOps is an agile model that groups IT development and operations staff according to task, breaking down the walls that might otherwise have sat between these functions in large organisations. It aims to automate repetitive processes, particularly in testing and deployment.

The agile model tends to be in favour for the maintenance of online, customer-facing applications that demand constant change - which makes the likes of REA Group and Domain Group prime candidates. It doesn’t tend to be as warmly welcomed for development and deployment of more static systems of record or in those organisations burdened with legacy IT.

This week Domain Group technology director Paul McManus revealed the company has embraced DevOps to such a degree that it was able to push 150 small changes into production in a single month, up from eight to ten under its former model.

What’s more remarkable is that the Domain team has managed to move at this speed while developing on Microsoft’s .NET platform, which has been used in large enterprise for the better part of 12 years. Most organisations embracing DevOps have chosen to take the Linux path, thanks largely to the breadth of orchestration tools made available by the open source community.

Domain isn’t using Microsoft’s source code management tooling (Team Foundation Server), which is geared towards deploying an application to the Microsoft Azure cloud, and is yet to be made available to the general public in Australia.

Instead, the team has strung together an automated application development, deployment and monitoring pipeline that pushes changes to its .NET apps onto the cloud services of Microsoft rival Amazon Web Services, using a suite of tools largely developed by startups.

Embracing change

McManus sold the DevOps model into the business by clearly identifying the goal: “to be able to deploy new features multiple times a day,” he told iTnews.

That required making a conscious choice to untether Domain Group from its reliance on Fairfax’s broader technology operations as part of a broader corporate reshuffle 18 months ago. More crucially, it required the hire of operations staff that understood software development and software developers in the context of DevOps. Both are rare commodities.

McManus found Jason Brown, an infrastructure engineer who had worked in web development earlier in his career, and paired him with development manager Fabien Ruffin, to build a continuous deployment pipeline.

The operations function - beginning with Brown and now expanded to a team of three - were very deliberately placed inside the development team.

In traditional approaches to IT, Brown told iTnews, code was “thrown over the walls” between isolated teams responsible for development, testing and infrastructure. DevOps, by contrast, demands that the entire team “takes ownership of the application from development right through to hosting”.

“DevOps only works if it is embedded into the development team,” McManus said.

McManus made a case to the business that Domain Group would reap the rewards of DevOps if it first gave Brown and Ruffin some leeway to finesse a continuous deployment pipeline that could be proven to deliver changes to production systems safely.

"I explained that to ship features at speed, we needed to invest in people like Jason and Fabien and provide them the time required to implement this,” he said.

Read on for Domain's step-by-step deployment pipeline...

The CD pipeline

Brown and Ruffin’s project took roughly two months, and has been explained in great detail on Domain’s technology blog.

At the ‘build’ end of the cycle, the team uses a set of products developed in Australia - Atlassian’s Bamboo for build orchestration and Octopus Deploy for deployment orchestration.

Domain Group's continuous deployment pipeline

Microsoft’s scripting language, Powershell, remains a critical part of the build process - which makes use of existing skills hardened in many IT teams over the past decade.

The engineers ensure they are spinning up repeatable templates in Amazon Web Services using both Powershell Desired State Configuration (DSC), AWS’ CloudFormation config management tool and a second DSC tool provided again by Brisbane-based Octopus Deploy.

Brown and Ruffin have attempted to automate the hand-off between as many of these tools as possible, so much so that Brown jokes he has “literally automated myself out of a job.”

“What that means is that I can now focus on much more interesting things,” he said.

What's the best way to celebrate a successful deployment to production? Introducing The Domain Train.

McManus said that typically, operations engineers that do automate their own function are provided opportunities to move further up the stack - from production and deployment into implementing instrumentation that monitors the health of those applications undergoing routine, incremental change. Domain Group uses New Relic and Pingdom for health monitoring, among others.

With software testing integrated much earlier in the process, developers also have to shift their mindset, he said. Their changes can be pushed to production in minutes - which demands that they code under the assumption that nobody will be there to fix any sloppy work or lax attention to security or system dependencies once they hand over.

Domain Group might not be a .NET shop forever - but the team’s work is proof that traditional IT shops can achieve a DevOps-like state by augmenting some of the tools they are likely to already familiar with.

Brown recommends that IT operations staff start learning to code to complement their existing skill sets. Searching for tools that can be integrated together to move application changes through to production at speed might seem a little daunting at first, but that’s fast becoming what’s expected of IT operations. He simply accepts that a DevOps role demands he piece together solutions that remove complexity from the total process.

“I don’t think we’ll ever see a single, generic pipeline to handle all the different clouds
and programming languages,” he said.

McManus said that while he understood why those managing legacy IT environments were reluctant to consider the DevOps model, invariably they won’t have a choice.

“My advice is to move forward or you’ll be overtaken,” he said.