DevOps and hard decisions

Australia’s most progressive organisations have embraced the agile methodology to push out new digital products to customers at ever-narrowing intervals.

Who gets to push the button?

Their goal - as we discussed in Pt 1 of this feature series - is to automate both the delivery of infrastructure and the testing of applications to the point that production is only a click of a button away.

The automation of both software testing and the infrastructure builds required to push new features into production creates an opportunity to re-think myriad roles within the IT organisation.

Where should testing sit, once a testing process is automated and available to developers at the push of a button? How should an application be supported when developed in an environment of constant iteration and change?

Consider the impact of one agile practice: test-driven development. Under TDD, a test is written for the business requirement and the code is tested against it continually. It’s a radical departure to regression tests against completed code at the ‘end’ of the project.

What does that mean for the testing discipline? The least dramatic impact of TDD is when software testers become embedded within agile development teams, rather than sitting in their own silos.

More dramatic is when an organisation decides that if a business analyst is clever enough to write a great functional spec, their skills might apply to testing. And the most dramatic impact is when software developers write automated test harnesses that come close to replacing the testing discipline altogether.

The majority of the organisations we spoke to for this feature series had chosen one of these three approaches, and nearly all report tense discussions between those in charge of software delivery and those in charge of the testing function.

The old rules of software development - correlations between the numbers of testers per head of developer, for example, are being rewritten.

Software testing managers argue that the independence of the testing function is paramount to software quality; that the discipline requires specialisation; or that it is often more expensive to continually concern your most productive software developers with testing rather than using a relatively cheaper resource to write a test at the back of the process

The heads of software delivery, by contrast, are tasked with seeking to eliminate any manual process that can be automated. That includes the build of software infrastructure to support an application.

Everything is up for discussion in a world where he who automates most wins.

IT Operations and DevOps

On that basis, any discussion of continuous delivery of agile software development is incomplete without diving into DevOps.

The politically sensitive definition of DevOps speaks to greater collaboration between software development and IT operations to deliver better quality software.

While a commendable aim, DevOps is unquestionably also driven by a desire to automate repeatable operations tasks, rendering them as more applicable to the software delivery function rather than IT Operations.

Careers are on the line, and tensions are high.

“The best coders are those that really get infrastructure applications in code,” says Pete Steel, CIO of retail and business banking at the Commonwealth Bank.

“The best engineers are obsessed with continuous improvement, and for every problem their first question is, how can we automate this? The goal is to take as many humans as possible out of the equation.

“These engineering functions are the crack teams that enable scaling out. They lift the waterline for everyone else. The dividend you get from these engineers is that it dumbs problems and processes down for everyone else."

Craig Rees, head of technology and product architect at Vodafone Australia, has a DevOps team in place that achieves these outcomes for the telco. Their focus, he said, is not on supporting applications but on building automation capabilities.

“They write our Chef frameworks, they build our monitoring infrastructure, and they build them as a platform,” he said. “They create those things, and it is up to the software delivery teams to build up their own environments within that framework and push out their own deployments.”

This wouldn’t be possible, Rees said, without deploying new applications to public clouds for Vodafone’s infrastructure needs. This leaves a marked impact on IT Operations.

Consuming infrastructure-as-a-service offers the automated creation of environments, he said.

“If I want to build a new test environment, I push a button and it’s there,” Rees said. “We do single click deployments without Ops needing to do anything.”

The cloud also offers auto-scaling, which prevents Operations from having to step in and provide increased capacity during periods of extraordinary demand. It also offers auto-healing features, he said, “so if servers go down I don’t need people monitoring CPU to see if we should shut off daemons or anything like that.”

Rees joked that running digital services in the cloud had “made us boring”.

"The job of what would have been L2 support has decreased by 80 percent,” he said.

The traditional response from IT Operations has been to argue that reliability will suffer as soon as its role is marginalised.

But Rees argues that - done right - DevOps can reduce operational risks.

“I’ve worked in some places where you have to change parameters in your code to compile it before you put it into production,” he said. 

“The way I look at that is that you are putting something into production that is not tested - which is ridiculous. Using that approach, people spend six months testing something but the organisation puts something into production that wasn’t tested.”

Under the DevOps model - assuming the right tools are in place - nothing goes live unless the test environment and production environments are validated as identical.

“That way, if it works in dev, we know damn right it will work in production,” Rees said. “That actually de-risks it.”

Tim Batten, head of strategy and architecture for the Woolworths Group, recommended agile advocates adjust their taxonomy to help win over those in the business unfamiliar with agile practices.

“I find having someone on the agile team with a strong understanding of ITIL is useful in bridging the gap,” he said. “The language we have been using is making any release a “standard change” - which means it is a pre-approved change. Speaking like that helped Ops people understand, if ‘we’ can make a release something I’ll accept as a standard release, I’m on board.”

Accountability for code

Steel, in his experience with the Commonwealth Bank, agrees the first step to DevOps success is “making the development and release process safe” by using automation to ensure consistency and compatibility with production systems.

The second step is far more challenging. If embedding IT Operations roles into software delivery teams sounds like a tough job, try telling a software developer that they can’t rely on operations to test and fix their code. Try telling them it’s now a case of “you break it, you fix it”.

“That is a cultural journey - a harder one for us - which requires accountability for the operations of a system to remain with its developers long after it has gone into production,” Steel said.

Steel is inspired by organisations like Netflix, which is structured is such a way “that the person that writes the code has ownership when putting it into production”.

“So if I as a developer write some bad code, I will be held to account. If I write code that goes into production that night, I’m the one likely to be getting the phone call if there are issues. I need to have that sense of responsibility.”

Michael Moseley, digital portfolio manager at Suncorp argued that DevOps sees the issue through the following lens:

“Our Ops people are usually held accountable for systems to be up, and our developers are there to generate business value,” he said.

“What if, just for a thought experiment, we swapped them around? What if your Ops people are accountable for delivering value faster, and your developers are told not to break stuff?”

Andrew Mitchell, IT development manager at Allianz noted the most successful systems the insurer has deployed - with the fewest bugs and page outs - were those occasions when developers of the code set up an error alert system off their own bat.

“If there is a single error occurs in the log file, the developer gets an SMS alert,” he said. “The developer fixes the bugs because they feel they own the software.”

What skills will organisations need to embrace in the future? Read on for more...