Westpac brings automation and AIOps to life, chasing CPU and memory alerts

Westpac is progressing its adoption of event-driven automation and AIOps in the IT infrastructure space, using it to add data and context to support tickets raised in ServiceNow.

The bank indicated at a Red Hat Ansible event in March last year that both event-driven Ansible (EDA) and AIOps were priority areas, after laying the foundations through an implementation of Ansible Automation Platform or AAP.

A year on, service owner of patterns automation Sean Dudding said the bank had succeeded in building out some EDA and AIOps use cases, raising CPU and memory alerts as one such area of focus.

“The key here is you have to place the automation between observability and incident management. We placed AAP between our observability tooling and ServiceNow,” Dudding said.

“We built some event filtering into ServiceNow, so events come from the observability tool … we then send that event via EDA and then Ansible does what it does in the background.

“We’ve also built in a way that we can query AI via a number of models that we’ve distributed on our OpenShift AI platform to also interact with that workflow which Ansible’s doing.”

Using the example of CPU and memory alerts, Dudding said that the EDA and AIOps-augmented workflow meant that the alerts could be actioned “straight away”.

“Previously, you’d raise a ticket, [and] somebody would look at it half an hour or an hour later, [when] it’s probably resolved itself already,” he said.

“So [the alert] comes from observability, it goes to EDA, we run a workflow we’d already written a number of years ago which does a very comprehensive health check on the endpoint, gathering data like what processes are using the CPU, what’s causing the memory load, look at the storage, and also go look in the event log, see what’s happening there.

“We take all that information within seconds of the alert coming in and we generate this comprehensive artifact and we attach it to the ServiceNow ticket. 

“What we’ve done recently is we’ve now taken that health check and we send it to our AI platform - not for it to make a decision but for it to provide advice on the data we’ve just given it - and we now also attach that AI artifact to the ticket, and we send it on to the current process of L1/L2 support.”

The artifacts not only have a point-in-time utility, but also are being collectively analysed after the fact.

“Where we’re finding additional benefit is these artifacts, because they’re taken at a point in time and they’re all consistent, they give us some data that we can analyse later down the line and see which services are causing the biggest problem, [or if] there’s anything else we can automate,” Dudding said.

“[We can] analyse [it] afterwards and get further benefit from automation.”

Dudding urged caution generally in the embrace of AI in infrastructure domains.

“With AI and AIops, be suspicious, be cautious,” he said.
“Each of the LLMs out there has [its]own personality. Get to know how it’s going to respond, avoid hallucination, and then once you’re happy then go for it.”

It was also revealed during the presentation that Westpac has “tested” Lightspeed, a coding assistant in Ansible that aims to help deliver automations.

Asked whether AI tools generally could lower barriers in terms of allowing less development-proficient users to craft automations, Dudding was doubtful.

“Do I think it allows anyone to write automation? No,” he said.

“It does enable people with little coding experience to deliver small pieces of well-defined automation, but you have to make sure you have the right guidelines and framework in place to allow them to do that.”

Dudding said that coding assistants were more useful in making skilled developers “more efficient with their time and being able to deliver more, rather than lowering barriers to entry.”