Home Affairs reveals core systems cloud push

Home Affairs has laid out an ambitious plan to re-architect parts of its core systems using smaller - possibly cloud-hosted - components built around standard patterns and practices.

Assistant secretary for the architecture and innovation branch Matt Jones - who is effectively the super-agency’s chief architect - told a recent AWS public sector summit that Home Affairs’ IT “can’t move fast enough” for many internal business areas.

It was revealed that Home Affairs has spent the best part of three years laying the groundwork to move faster - with AWS virtual private cloud (VPC) and DevOps practices central to that proposed shift.

But the groundwork has occurred at the department’s periphery, where the blast radius is lower should anything go wrong.

“So far, everything ... has been at the edge of the organisation,” says Chris Gough, delivery practice manager at Canberra-based consultancy GoSource, which has been heavily involved.

“This is because of the initial posture with risk and keeping the initial experiments away from the main business systems. 

“[But] because of these projects, the organisation has gained experience with doing DevOps and infrastructure-as-code securely. 

“We've developed templates and patterns that satisfies the organisation's defensive security posture, and we've made quite a lot of progress since the early days.”

Home Affairs’ IT now believes it has seen enough to apply the new ways of working more deeply. 

This is likely to involve opening up monolithic and mainframe-based systems with APIs, refactoring some existing applications to run in the cloud, and work on more cloud-native systems.

Building before laws come

One of the main drivers for Home Affairs to move faster is to shorten the cycle to code changes to visa processing conditions that are passed by parliament.

“Every media report that creates a new headline about the latest visa loophole that some nefarious body has exploited, for us creates new visa rules, new legislation, more software, and multiplies the myriad of different test cases that we have to run [on] each [software] release,” Home Affairs program manager Paul Morrison said.

“It's not untypical for a [code] release inside the department to touch at least eight or 10 different platforms for a typical visa change, and that happens fairly regularly as the politicians or the Parliament pass legislation and change the rules around visa processing,” Jones concurred.

Current long lead-times and complexity mean Home Affairs is coding changes in anticipation of law changes.

“We have to run projects before the legislation passes,” Morrison said.

“With recent governments having such slim majorities, at times the legislation doesn't pass, but we've already run the project and coded up all the changes, because if the legislation does pass, it has to be live a couple of weeks later. 

“So we have to hurriedly unpick the code for that project a few weeks before go-live, and that can leave us with pockets of technical debt.

“[In addition], each system is a monolith. So there's a high fixed overhead cost for even small incremental changes.”

Jones said the challenge for Home Affairs was to “make the environment much more agile so that we can react to what the business wants”.

“That's what the team's been working on,” he said.

Why Home Affairs moves slowly

The department’s IT environment is the result of 30 years’ of “mergers and MOGs” or machinery of government changes, Jones said.

“Consequently most IT platforms that have ever been invented in the last 30 years exist in one form or another inside our organisation.

“The challenge that we've had is how to take this large monolithic beast that's grown up over the last 30 years and break it down into a number of much smaller, agile chunks, effectively trying to convert an elephant into a gazelle, which is very challenging.”

The department has “over 450” business systems it considers “essential to the functioning of the agency.”

“Then there's probably a long tail of maybe another 600-700 IT systems that have grown up over time that fulfill a specific need for one or more areas in the business,” Jones continued.

“Most of these systems have relationships with one another and so managing the ripple effect of making one change in one system, and managing that through into another, is actually very, very complex.”

For the past 15 years, Home Affairs pursued an enterprise service bus (ESB) architecture, in the hope of creating a bank of standard, reusable IT services. But the benefits had largely failed to materialise.

“One of the reasons the enterprise service bus isn't working for us is because we find we get very little reuse from the services we put on it,” Morrison said.

“Our funding models are tied to project work, rather than an outside-in view of how we should best design and expose services, so we have over 300 separate services, and we're only seeing about 10 percent of reuse with any one service. 

“[It’s] not the reuse we were all hoping for when we embarked on this paradigm 15 years ago.”

The ESB also created a landscape “where integrating systems becomes custom code,” Morrison noted.

“For example, system A has to send a different request packet to three different systems - B, C and D - and they in turn respond with three distinct response packets,” he said.

“Individually, each custom request response only represents a very small coupling of the systems but added up over time you create an ecosystem of quite tight coupling.”

That coupling required multiple internal teams to work together to implement any changes.

“[Because] we went down the big ESB path ... that requires projects to choreograph several teams in unison to ensure the ESB custom request and responses always work between the different systems,” Morrison said.

“We've been thinking about whether there is a way to simplify our integration landscape, so that systems A, B, C, and D, can avoid needing specific interchange contracts with each other, and instead implement a new pattern.”

Another reason the department is slow is that change occurs on a six-month release cycle.

The problems were immediately apparent to Gough as he started to work with Home Affairs.

“As an outside observer coming into Home Affairs, the key problems as I saw them were [firstly], it was too difficult to respond to user insight because change is too slow, with alf-yearly release cycles plus significant planning lag times,” Gough said.

“[Second], small changes are difficult to make cost effectively. Communication and coordination overheads across many vertical silos imposes a high fixed cost on any change.”

Home Affairs had spent 15 years building a “complex web of interconnected monoliths tied together with an ESB,” Gough noted.

The department then mapped out what Gough called an “idealised integration surface that would allow the same business domains to operate more autonomously.”

“It's more of a loosely coupled network of comparatively simpler services that only communicate through their defined interfaces,” he said.

“We express this as API microservices, but in reality, it's about refactoring, architecture and governance. This isn't possible without enabling technology [for] effective automation.”

Creating the patterns

Morrison said there was no “magical” way for Home Affairs to rid itself of complexity.

“What we're thinking about is how we work within this same complexity, but by breaking that complexity down into more manageable pieces, so each piece needs to act in a similar way, become less custom and more patterned or predictable,” he said.

The department saw a path to repeatability by running code releases in a more Agile way, underpinned by DevOps and a continuous improvement and continuous delivery (CI/CD) pipeline.

CI/CD automation is handled by Jenkins. Infrastructure to support new applications is spun up and down in AWS cloud.

“Cloud was the easiest way for us to achieve infrastructure-as-code so that's where we started,” Gough said.

One of the key outcomes of this work is an “AWS cloud pattern” that internal Home Affairs teams can use to more quickly spin up infrastructure in support of their projects.

Developing the pattern involved “a lengthy process with plenty of engagement” with Home Affairs’ IT security.

“It's a general purpose, high availability infrastructure pattern, and it's available to any new cloud project in the department,” Gough said.

“No matter what the project is, if they can start with this pattern, they can begin development in days, not weeks or months.”

The pattern covered “basic network components like DNS, firewall, load balancer, queue service and an API gateway,” Gough explained. 

“It has two subnets across two availability zones with an autoscale group. There are some other optional components that we can add in.

“It was a lot of hard work to get this pattern approved, developed, deployed live and connected back into the department's systems with our first solution, but now that the bridge is built, other projects can quickly and simply reuse it.”

First forays into cloud

Home Affairs’ first foray into cloud-native applications was an online booking system for citizenship appointments, first flagged in 2016.

The apps appear to run in AWS VPC, a walled-off segment of AWS resources. This model is favoured by other risk-averse large corporates and government agencies.

“The original prototype was developed in partnership with the Digital Transformation Agency, who used their service design methodology to focus on user needs,” Gough said.

“The business problem here is that there's about 140,000 new citizenships conferred every year. 

“As part of the process of becoming an Australian citizen, you need to sit an interview and pass a test. These tests are conducted in offices around the country by appointment. 

“That's a lot of appointments, and they need to be scheduled at a time when both the applicant and the facility are available.”

Previously, appointments had to be scheduled or changed by calling Home Affairs.

“What we have now is a new system that provides a low cost self-service model for applicants to change the time of their appointments online,” Gough said. 

“It's significantly faster and more convenient than going through a call centre. By reducing the load on call centres, it reduces costs and also shortens the wait times for other kinds of queries.

“It's a public-facing system that integrates back into Home Affairs through the enterprise service bus. The system security planning and approvals process took about nine months, which is a lot longer than it took us to build it.”

GoSource remained involved in cloud-native projects but Home Affairs increasingly handled more work internally as it built capability.

“One place where the department likes to experiment is with improving the experience of travelers arriving to the country,” Gough said.

One experiment saw a team within Home Affairs create “a pilot application for the inbound passenger card” filled in by arriving travellers. 

Gough said that the application “has not proceeded past pilot for a range of business-related issues that will be explored as part of ongoing business process improvement”.

But he said the experiment had “validated two important ideas”.

“Unlike earlier projects delivered by GoSource, this time we provide a DevOps support, but the product was [built] entirely in-house,” he said.

“Second, this experiment was quicker and cheaper than it would have been using self-hosted systems, but the quality didn't suffer.”

This model was replicated on a subsequent project for an application that enables multiple countries in the Pacific to validate passports, record vessel movements in their territorial waters and share information with each other.” 

“It used entirely cloud-hosted backends and was developed by an internal Home Affairs team with Gosource providing DevOps again,” Gough said, noting the project landed an Australian Border Force commendation.

As time has progressed, Home Affairs has picked up speed in running cloud-native projects.

The department built a visa expiry notification system that sends SMS messages to certain types of visa holders, reminding them of an upcoming visa expiry. 

“There's two interesting things to note about this,” Gough said.

“It was very cheap to build and operate because AWS PaaS does all the heavy lifting, and the planning and approvals went very quickly. 

“And because we'd used all of those services before, it reused infrastructure-as-code from other projects.”

Marks of maturity

Home Affairs has “more than one internal team building and supporting cloud-native solutions using a mature delivery pipeline in a consistent way”, according to Gough.

“This has taken a couple of years and we now feel more confident to increase the scope of cloud adoption.

“Going forward, the risk profile is different - it's lower.”

What this means in practice is that cloud will move from ‘edge’ use cases to impact some of Home Affairs more critical systems.

No specific timeline was put on this work, though officials indicated that scoping work is underway to understand “Home Affairs' business domains and resources in each domain”. 

Domains range from those with live systems in the cloud, to those that are involved in experiments, and more still that have yet to embrace cloud and DevOps at all.

“This is the beginning of a process to envelop complex legacy mainframes with modern developer-friendly APIs,” Gough says.

“It's a kind of roadmap for the first stage of decoupling the components of ‘the elephant’ so that we can refactor it into clusters of microservices inside autonomous domains.”

Despite the past three years, there is still significant work to do.

“If we intend to move from our interconnected 'elephant' to our herd of fast 'gazelles', we're going to need to do a few things differently,” Morrison said. 

“We must redesign our monoliths over time to be much more granular applications, remove the custom integrations between applications, and then let those applications self-service their integration needs via APIs. 

“We need to wrap all of our new systems in comprehensive automated tests and use standard patterns to deploy applications into. 

“We need to have a functioning continuous testing and continuous deployment pipeline, and we want to do lightweight prototyping with our business and then iterate quickly towards the intended solution, rather than asking or expecting them to detail all their requirements several months in advance. 

“To achieve this, we will rely on the cloud since we don't have this very high level of automation on-premises at low cost. 

“If the overhead of creating and integrating a new 'gazelle' is low, then we will have moved a long way towards addressing some of our initial problems.”