DevOps and hard decisions

Australia’s most progressive organisations have embraced the agile methodology to push out new digital products to customers at ever-narrowing intervals.

Who gets to push the button?

Their goal - as we discussed in Pt 1 of this feature series - is to automate both the delivery of infrastructure and the testing of applications to the point that production is only a click of a button away.

The automation of both software testing and the infrastructure builds required to push new features into production creates an opportunity to re-think myriad roles within the IT organisation.

Where should testing sit, once a testing process is automated and available to developers at the push of a button? How should an application be supported when developed in an environment of constant iteration and change?

Consider the impact of one agile practice: test-driven development. Under TDD, a test is written for the business requirement and the code is tested against it continually. It’s a radical departure to regression tests against completed code at the ‘end’ of the project.

What does that mean for the testing discipline? The least dramatic impact of TDD is when software testers become embedded within agile development teams, rather than sitting in their own silos.

More dramatic is when an organisation decides that if a business analyst is clever enough to write a great functional spec, their skills might apply to testing. And the most dramatic impact is when software developers write automated test harnesses that come close to replacing the testing discipline altogether.

The majority of the organisations we spoke to for this feature series had chosen one of these three approaches, and nearly all report tense discussions between those in charge of software delivery and those in charge of the testing function.

The old rules of software development - correlations between the numbers of testers per head of developer, for example, are being rewritten.

Software testing managers argue that the independence of the testing function is paramount to software quality; that the discipline requires specialisation; or that it is often more expensive to continually concern your most productive software developers with testing rather than using a relatively cheaper resource to write a test at the back of the process

The heads of software delivery, by contrast, are tasked with seeking to eliminate any manual process that can be automated. That includes the build of software infrastructure to support an application.

Everything is up for discussion in a world where he who automates most wins.

IT Operations and DevOps

On that basis, any discussion of continuous delivery of agile software development is incomplete without diving into DevOps.

The politically sensitive definition of DevOps speaks to greater collaboration between software development and IT operations to deliver better quality software.

While a commendable aim, DevOps is unquestionably also driven by a desire to automate repeatable operations tasks, rendering them as more applicable to the software delivery function rather than IT Operations.

Careers are on the line, and tensions are high.

“The best coders are those that really get infrastructure applications in code,” says Pete Steel, CIO of retail and business banking at the Commonwealth Bank.

“The best engineers are obsessed with continuous improvement, and for every problem their first question is, how can we automate this? The goal is to take as many humans as possible out of the equation.

“These engineering functions are the crack teams that enable scaling out. They lift the waterline for everyone else. The dividend you get from these engineers is that it dumbs problems and processes down for everyone else."

Craig Rees, head of technology and product architect at Vodafone Australia, has a DevOps team in place that achieves these outcomes for the telco. Their focus, he said, is not on supporting applications but on building automation capabilities.

“They write our Chef frameworks, they build our monitoring infrastructure, and they build them as a platform,” he said. “They create those things, and it is up to the software delivery teams to build up their own environments within that framework and push out their own deployments.”

This wouldn’t be possible, Rees said, without deploying new applications to public clouds for Vodafone’s infrastructure needs. This leaves a marked impact on IT Operations.

Consuming infrastructure-as-a-service offers the automated creation of environments, he said.

“If I want to build a new test environment, I push a button and it’s there,” Rees said. “We do single click deployments without Ops needing to do anything.”

The cloud also offers auto-scaling, which prevents Operations from having to step in and provide increased capacity during periods of extraordinary demand. It also offers auto-healing features, he said, “so if servers go down I don’t need people monitoring CPU to see if we should shut off daemons or anything like that.”

Rees joked that running digital services in the cloud had “made us boring”.

"The job of what would have been L2 support has decreased by 80 percent,” he said.

The traditional response from IT Operations has been to argue that reliability will suffer as soon as its role is marginalised.

But Rees argues that - done right - DevOps can reduce operational risks.

“I’ve worked in some places where you have to change parameters in your code to compile it before you put it into production,” he said. 

“The way I look at that is that you are putting something into production that is not tested - which is ridiculous. Using that approach, people spend six months testing something but the organisation puts something into production that wasn’t tested.”

Under the DevOps model - assuming the right tools are in place - nothing goes live unless the test environment and production environments are validated as identical.

“That way, if it works in dev, we know damn right it will work in production,” Rees said. “That actually de-risks it.”

Tim Batten, head of strategy and architecture for the Woolworths Group, recommended agile advocates adjust their taxonomy to help win over those in the business unfamiliar with agile practices.

“I find having someone on the agile team with a strong understanding of ITIL is useful in bridging the gap,” he said. “The language we have been using is making any release a “standard change” - which means it is a pre-approved change. Speaking like that helped Ops people understand, if ‘we’ can make a release something I’ll accept as a standard release, I’m on board.”

Accountability for code

Steel, in his experience with the Commonwealth Bank, agrees the first step to DevOps success is “making the development and release process safe” by using automation to ensure consistency and compatibility with production systems.

The second step is far more challenging. If embedding IT Operations roles into software delivery teams sounds like a tough job, try telling a software developer that they can’t rely on operations to test and fix their code. Try telling them it’s now a case of “you break it, you fix it”.

“That is a cultural journey - a harder one for us - which requires accountability for the operations of a system to remain with its developers long after it has gone into production,” Steel said.

Steel is inspired by organisations like Netflix, which is structured is such a way “that the person that writes the code has ownership when putting it into production”.

“So if I as a developer write some bad code, I will be held to account. If I write code that goes into production that night, I’m the one likely to be getting the phone call if there are issues. I need to have that sense of responsibility.”

Michael Moseley, digital portfolio manager at Suncorp argued that DevOps sees the issue through the following lens:

“Our Ops people are usually held accountable for systems to be up, and our developers are there to generate business value,” he said.

“What if, just for a thought experiment, we swapped them around? What if your Ops people are accountable for delivering value faster, and your developers are told not to break stuff?”

Andrew Mitchell, IT development manager at Allianz noted the most successful systems the insurer has deployed - with the fewest bugs and page outs - were those occasions when developers of the code set up an error alert system off their own bat.

“If there is a single error occurs in the log file, the developer gets an SMS alert,” he said. “The developer fixes the bugs because they feel they own the software.”

What skills will organisations need to embrace in the future? Read on for more...

Gaining trust

Mitchell nonetheless understands the objections of IT Operations staff to the DevOps concept.

“For me, DevOps boils down to one word: Trust. Does your Ops team trust you?” he asked.

 “With a waterfall approach, they only ever got to see the software every six months. And they’ve got used to thinking, 'if it’s anything like the last six monthly release, I’m going to make you jump through a lot of hoops'.”

Successful practitioners of the DevOps model recommend that the Ops team be invited into stand-ups and “letting them define some requirements on the developers before the coding even begins.”

“That’s a key way to gain the trust - by inviting them to the table,” Mitchell advised.

Equally, software developers can begin to win trust by asking Ops what repetitive functions they most want to be automated in software.

It is inevitable, however, that Ops staff will respond in one of two predictable ways.

“Some on the operational side will embrace it and say ‘thank God, you’re one of the few guys I’ve ever worked with in Dev that has automated a process for me. I hated that process,” Mitchell said.

Others will try and put up a barrier.

“Once you truly go down the agile path, you realise there has been a lot of damage done in the relationship between software development and IT operations in the past,” Claudia Lajeunesse, head of platform solutions at FIIG Securities said.

Successful DevOps programs require tech leaders to play the part of a counsellor, she said.

“Part of [Ops] not wanting to come to the stand-ups might be because they remember the years of blame – one side blames ‘your crappy code’, the other blames ‘your shitty tin.’ You sit back and realise, this has been really, really painful for these guys. They are sick of being blamed.

“So you need to say to them, ‘How about we all start over? Because we’re not getting the right outcome to the end customer. We’ve had a lot of years of blaming each other for everything. Can’t we just pretend we’re a start-up, can’t we let that go? Neither of us will be successful if we continue this’," she said.

“You need to revisit that damage, rather than just say: ‘We’re doing agile now, get with the program’.”

New skills

As traditional IT roles are automated and Operations responsibilities blur with the software delivery domain, workplace theorists have highlighted the need for staff to develop ‘T-Shaped skills’.

‘T-Shaped skills’ refers to having almost as much adjacent knowledge about other areas of the business (the horizontal line) as depth in your own area of expertise (the vertical line).

Lajeunesse recommended that every effort be made to show staff stuck in siloed Operations roles there are opportunities for learning adjacent skills and future-proofing their career.

Mitchell said IT Operations staff should view DevOps as an opportunity to “focus on the things that add more value,” such as capacity planning, or improving infrastructure monitoring.

“They’re not necessarily out of a job, it’s just a different job,” he said.

Batten said the opportunity for staff to cross-skill is the “sell” that should win all comers to the DevOps idea.

“While some people don’t accept that, others say, ‘yes, I am interested in adjacent expertise’,” he said.

The move to DevOps is an opportunity, said Lajeunesse, to look at the skill sets your organisation might want in the future rather than what has made sense in the past.

“There are people that might be a great technical fit in the organisation today that don’t fit the future organisation, and we have to be comfortable with helping them move on,” she said.

Perversely, in the same breath software delivery executives will tell you of their struggle to help siloed staff reskill or move on, they also report difficulties in recruiting new staff that have the right breadth of skills.

Many have gone to outsourced suppliers, pairing the agile specialists from these organisations with their own teams, only to find that these outsourced suppliers often don’t practice what they preach. DevOps and continuous delivery is simply too new for there to be adequate human resources on the market.

Each cited different approaches to solving this dilemma.

Suncorp found success by recruiting university graduates that weren’t predisposed to a pre-agile way of working.

“We ran an intern program with a university - and the students got to the point where they were doing daily iterations,” Moseley said. “They were pivoting and changing the way they work every day.

“They have come from a culture where it is expected they don’t know anything, where they have to constantly learn and apply their knowledge.”

For Lajeunesse, hiring the right staff for the agile workplace requires the buy-in of the HR department.

“We invited HR into the scrums,” she said. “I wanted them to come and see how we worked so that when they are helping me recruit and do the initial screening, they know what happens daily. I needed HR to have physical knowledge of actually standing with us.”

This assumes, of course, that the organisation has an appetite or budget to hire staff trained in agile. Even if they can be found, the challenge for many a software delivery lead is how to free up budget to afford them. How, if you have no budget to hire, might you bring in an agile coach or an iteration manager?

Ed Cortis, head of software delivery for Bankwest, has found that some of required skills might be available in your own team, just not in a form ready to sprint from the get-go. With a bit of training, somebody with a BA, testing or development background might be ready for the challenge. But you have to create a culture that devours learning.

Bankwest flies in motivational speakers and runs various workshops to encourage that culture of learning and keep existing staff on the right path.

“The thing that I’m most passionate about is the culture of organisations,” Cortis said. “If you haven’t got the people engaged, you’re wasting your time.

“We talk about the tools but culture is the piece we never talk about. Selling the cultural change is enormously hard and very important. There are no tools and vendors to help you with cultural change.

“At Bankwest we have been very deliberate in creating a learning organisation. If we are to embrace digital disruption, we must encourage learning. It’s delighted me to see how much we have changed this place. We’ve found our mojo.”