Skills SA to treat multi-agent workflow as a reusable pattern

Skills SA, supported by the Department of State Development, has built a multi-agent AI workflow to help it review how well vocational education and training providers meet a set of student support requirements each year.

The setup, running largely on a Databricks stack, has the makings of a pattern that would be applicable to a broad range of other government enterprise processes, and reuse inside of Skills SA is already on the cards, according to AI, data and integration lead Jarrad Taylor.

Speaking at the Databricks Data + AI Summit in San Francisco, Taylor said the initial use case is to help assessors review responses to an annual compliance process for VET training providers, known as the Organisational Self Assessment or OSA.

The OSA is effectively a 40-to-50 question form that providers must complete annually to show they meet six Skills SA ‘student support standards’.

Taylor said that reviewing the completed self-assessments, and analysing them for trends, was manual and time-consuming.

It was also challenging to ensure that they were assessed consistently.

“OSA is a compliance process used to assess training providers against a defined set of standards each year,” he said.

“We receive [hundreds] of these submissions, which are quite large and contain both structured and unstructured information and large volumes of free text within the responses. 

“Assessors review these submissions, evaluate them against the known standards and determine whether further investigation or follow-up may be required. 

“The process is highly manual, requires significant assessor effort, and more importantly it relies on people reviewing large volumes of information and applying standards consistently across every submission.”

Taylor said the assessment process was considered a good candidate for AI, “to support the assessors to help improve consistency and reduce effort.”

“The challenge really wasn't in any individual step in the process. It was the cumulative effort,” he said.

Work on the multi-agent system designed to aid assessors started about 18 months ago.

A supervisor agent orchestrates the work of three other agents - an operational agent, an analysis agent and a decision support agent - that assess the completed OSA forms.

Claudia Kralik, lead business analyst, demonstrated the system in action at the Databricks event.

Taylor described three layers of benefit to come from AI augmentation of the OSA workflow.

“For our assessors, the goal is to reduce the time spent on routine activities and allow them to focus on the submissions that genuinely required further human judgment, investigation, and/or follow-up,” Taylor said.

“For our operational staff, it provides some real-time visibility into workflow statuses and finding bottlenecks and trends across the assessment process.

“And for Skills SA, it helps apply standards more consistently while maintaining a complete audit trail of how recommendations and decisions were reached.”

He suggested that what had been built for OSA represented a reusable pattern that could be applied to other enterprise processes.

“While we've used OSA as the example, we see it as a reusable pattern for operational AI in regulated environments,” he said.

“The same characteristics that made OSA a good candidate for AI also exist across many other government enterprise processes. 

“Anywhere you have a high volume of information, clearly defined standards, human decision-makers, and a requirement for governance and auditability, you'll often encounter the same challenges.

“What we've shown … is one way of addressing those challenges through a combination of specialised agents, governed operational state, and human oversight.

“The next step is expanding this pattern into additional domains and workflows. The broader opportunity is much larger than this single use case.”

Ry Crozier attended Databricks Data + AI Summit in San Francisco as a guest of Databricks.