ATO restructures cloud ops, embraces 'approval by default'

The Australian Taxation Office (ATO) has halved the size of its cloud team after expanding the use of self-service capabilities on its Amazon Web Services platform, redeploying affected staff into application development roles.

Sam Thomas, ATO

The restructure forms part of the ATO’s broader push to accelerate digital service delivery by removing internal bottlenecks and embedding components of AWS’ shared responsibility model across its platform and engineering teams.

The ATO brought cloud services into its developer experience branch last year and, since then, has shifted much of the approval burden away from the platform team by streamlining processes and expanding self-serve tools.

ATO assistant commissioner for developer and application shared services Hosh Elavia said the change has helped the cloud team “add value” while also reducing "the need for our cloud team to be as big as it was”.

“We’ve reduced it by half,” he told an audience at the AWS Public Sector Symposium in Canberra in August. “And what we’ve been able to do with that is actually give those people back into the app team.

“They've now got some cloud knowledge native in [the app] team, so they can also grow their capability. They can move faster, and they've got someone who's seen the other end of it as well.”

Elavia described the ATO as “a pretty early adopter of cloud” but noted that what “worked for us back in 2016 no longer works for us today”.

The previous set-up became a “barrier to delivery, rather than enhancing the delivery of application teams”, he said.

Despite increasing headcount to develop “richer cloud services across the organisation,” the ATO hit a limit on cloud scalability with its existing internal structure and processes.

“We had a model where everything was approved for teams in a particular use case to using cloud, but that meant the next team that came along didn't know what was already there, didn't know what they could or couldn’t use and had to go through that whole process as well,” Elavia explained.

“Internally, we had a joke that you needed six gods to approve something. It was very much seen as a great platform, but getting onto it was quite challenging for our teams.”

Other issues included a process-heavy approval system with long service-level agreements, limited transparency about available resources for developers and poor communication around platform changes and improvements.

To address this, the ATO began reworking its internal responsibility model, using AWS’ shared responsibility framework, something the agency alluded to at last year's public sector conference in the context of optimisation and cost control.

ATO program director of developer and application shared services Sam Thomas explained how the framework is now helping support a governance approach of “approval by default”.

“This kind of model allows people to have a really easy common understanding,” he said.

“It might be a little bit more blurry, but it means that we can have a real conversation without having to go down to right at the bottom and that just helps clarify everybody's accountability.”

The ATO also stood up a new platform for applying open-source-style collaboration internally, which enables engineers to “play with things, break things that don't affect the users”, Thomas said.

“We need it to make it easy for our platform engineers to do what they need to do because when we're making a change, it's our responsibility to provide that place for them to do their testing to do their development,"

The ATO also introduced controls around versioning changes within the AWS platform in an effort to improve standardisation.

Rather than “blasting out” updates organisation-wide, it now assigns version numbers to changes, tests it in one division, checks it, and only rolls them out once verified.

Divisions are selected based on their risk appetite, with changes trialled in low-risk or “bleeding edge” teams before reaching units with more mission-critical or security-sensitive workloads.

This approach, Thomas said, prevents major downstram impacts such as “hundreds of people sitting on their hands waiting for us to come back up again” when an environment goes down.

Natural evolution

Alongside these structural issues, the ATO’s AWS environment had become increasingly fragmented since its implementation nine years ago.

The platform’s “natural evolution” saw each new developer project treated as a “special case”, according to Thomas, resulting in knowledge fragmentation across hundreds of accounts.

“We ended up building all these internal silos, and our ability to scale is limited to the number of accounts that we own,” he explained.

To remedy this, the ATO has standardised this structure into five categories, improving metadata visibility so the platform knows who owns what and how everything fits in the organisation.

From a security standpoint, ATO is making several changes, including re-implementing its data perimeter using a more standard and consistent framework.

Still a “work in progress”, the redesigned perimeter will combine its own architectural goals with reference implementation shared by AWS on GitHub.

In parallel, the ATO is rebuilding its cyber model around testable controls and continuous validation, adopting a ‘continuous authority to operate (CATO) approach.

The model includes integrating automatic testing into deployment pipelines, cleaning up legacy exceptions within its firewall and proactively simulating incidents within its infrastructure.

Thomas said changes aim to embed a culture of continuous improvement while providing “next level of validation”.

“Even just today, we found out that a control that we thought was effective was not -- based on the testing we've done,” Thomas added.

“But, we can produce a report that says: ‘Here’s all my controls. Here’s my proof that they’re effective. It's something that gives a lot of value and we want to continue building out.”

Eleanor Dickinson attended the AWS Public Sector Symposium in Canberra as a guest of AWS.