Revenue NSW directed to document its AI risk

Revenue NSW is lagging in its effort to keep risk assessments and documentation for its use of AI and automation in its fines and collections operations current, according to the state’s audit office.

The Audit Office of NSW reached the conclusion [pdf] after identifying “gaps” in the agency’s risk assessments for the technology and documentation intended to outline its accuracy expectations for it, in a recent audit of its hardship assistance administration.

The agency was directed to have a current risk assessment for every automation and AI tool in use, and to document "key expectations for accuracy and performance" of those systems, by October next year.

The direction came after the auditor found that NSW Revenue risk assessments for the use of garnishee order systems had not been revisited since 2023, in spite of work to develop an organisation wide risk and opportunity assessment for automaton and artificial intelligence that was completed in July this year.

Revenue NSW uses artificial intelligence and machine learning tools across several of its fines and debts functions, including identifying and shielding vulnerable individuals from enforcement action and generating debt profiles.

These efforts previously attracted public scrutiny in 2021 when the NSW Ombudsman found the agency unlawfully used machine technology to issue garnishee orders without human oversight.

In some cases, the automated systems wiped out the bank balances of vulnerable welfare recipients whose only source of revenue was Centrelink payments.

The auditor yesterday also suggested that the agency reassess "its use of automation and artificial intelligence tools against the NSW Artificial Intelligence Assurance Framework principles."

"While there is no mandatory requirement to reassess the tools against the framework, key considerations under the NSW government’s artificial intelligence ethics policy outline that ‘it is imperative that AI solutions are designed with and monitored against explicit standards for performance, reliability, robustness and auditability, and that they align with the ethical AI principles'," the auditor noted.

Revenue NSW collected $43 billion in 2023-2024 accounting for 35 per cent of the state’s total income for the period.