ANZ Bank brings DevOps to data

ANZ Bank is bringing DevOps to its enterprise-wide data environment, creating an “enablement” team to stand up common tools and drive adoption into lines of business and across geographies.

Head of data engineering David Hall is leading the push, which he believes is being made necessary by a “business environment of rapid change, explosion of data and the need for greater insights."

“All of these things are going to hit us harder and harder,” Hall told IBM’s recent World of Watson conference in Las Vegas.

“At the same time the technology we use [as IT and developers of data solutions] to deliver these business solutions is changing faster than ever.

“To keep up with that is a really hard thing to do and we’re not really set up to do it.”

Hall said ANZ was not alone in looking to apply the DevOps methodology to the development and evolution of data systems.

However, he said he wanted “to start a groundswell around this in the data industry”, and is seeking partners to aid both an industry-wide drive to DevOps as well as to help document how to make it work.

“There’s a massive amount of material available on the internet about DevOps and pretty much all of it will talk in a Java, mobile or web development context,” Hall said.

“You have to ‘lens’ through it and work out how you can take the concepts and apply it in a data warehousing or data analytics type space.

“There’s very little published material on how to do this in data.”

Hall – who this year assumed responsibility for “all of the technical build and platform evolution across all of the data assets across ANZ across all lines of business and geographies” – has created a small DevOps enablement team of four to bring the methodology to ANZ’s data operations.

“Their job is to be the evangelists, the catalysts, the assistants, and the pushers to try to get this driving forward, but then they also take care of setting up some of our common infrastructure -  systems like Jira and Bamboo,” he said.

“You don’t want every [data] team [out in the business] to have to do that from scratch themselves, but you also need some people [centrally] who are thinking, living and breathing this.

“So my enablement team will set up the build automation server, which is the container you put your scripts in, but it’s up to each of the development teams in the legacy [data] platforms to then write scripts or capabilities using those enterprise toolsets or their own.”

Until Hall reclassified them as his DevOps enablement team, two of the four were mainframe DB/2 and COBOL developers, one was an IBM InfoSphere DataStage developer, and the other focused on Microsoft Reporting Services.

He said although his enablement team was driving DevOps for data, other teams at ANZ were now “coming to the party”.

Data 'PaaS'

Hall assumed the role of head of data engineering for ANZ back in February this year, effectively taking ownership of many of ANZ’s key data platforms including IBM, Teradata, Oracle and the bank’s Hadoop clusters, as well as front end tools like SAS, Tableau and Qlikview.

“We’ve started an enterprise technology data function for ANZ so we’re responsible for data capabilities across all lines of business, geographies and technologies,” Hall said.

Under this structure, data systems are being redefined as “enterprise assets for the bank”.

However, Hall said some aren't enterprise assets yet but "at least I’ve got a mandate to take them to there if and when it makes sense".

Hall has also assumed line of business delivery of data analytics capabilities.

However, he said more and more of what his team is trying to do is "push away from ‘come to us and we’ll build it’ to effectively trying to set myself up as a ‘platform as a service’ provider” to allow business end-users to self-service.

“I don’t want a team of report writers, for example,” he said.

“I want to give a reporting toolkit to the businesses to say, ‘Hey, you create your own reports, dashboards, analysis, whatever else it might be. I’ll supply you with some really great tools that have got production-level support infrastructure and industrialisation, and some ready-to-use data with known data quality, and then go knock yourselves out’.

“Then, if you need help, talk to me. If you need it industrialised – so this might be something that’s business critical and has to run real-time, 24x7 then we will take what you build as a prototype and turn it into production-ready code.”

Ry Crozier attended IBM’s World of Watson as a guest of IBM.