Revenue NSW unlawfully used an automated system to recover unpaid debts and in some cases emptied the bank accounts of vulnerable people between 2016 and 2019, before making technical changes.
NSW Ombudsman Paul Miller exposed the conduct in a report calling for greater visibility of the use of artificial intelligence and machine learning technology in government decision-making.
Agencies are not currently required to report their use of 'machine technology' to parliament, and do not routinely notify people when decision have been made with the assistance of technology.
The report, released on Monday, found Revenue NSW had used "machine technology" to issue garnishee orders without human oversight following a series of complaints to the Ombudsman.
Garnishee orders are used to recover outstanding fines by requiring financial institutions to transfer funds to Revenue NSW from an account holder with an outstanding fine without their prior knowledge.
But under the Fines Act, orders are only to be issued when a person has not engaged with Revenue NSW after several notifications.
The Ombudsman found a number of complainants, who were not aware that "machine technology" was in use, had “been left with a zero balance in their account” following garnishee orders.
It said “some of the complainants were welfare-recipients, whose bank accounts had held the funds they were receiving from Centrelink as their only source of income”.
Between the 2010-11 and 2018-19 financial years, the number of garnishee orders issued by Revenue NSW climbed from 6905 to more than 1.6 million.
Following the inquiries, Revenue NSW made a number of changes to the automated system, including taking steps to “exclude ‘vulnerable persons’ from the making of garnishee orders”.
The change involved “implementing a new machine learning model... with the intention of identifying and excluding persons identified as vulnerable” in September 2018.
“The model was developed using machine learning algorithms that compared all customer records with the 60,000 people already identified as vulnerable in the system,” the Ombudsman said.
“Overall, the model was trained to identify if a person was vulnerable using 250,000 customer files, and having regard to a list of potential variables.”
Revenue NSW also introduce a manual step in the process of issuing garnishee orders in March 2019 that required formal sign-off from a staff member.
“This is [done] by way of a traffic light system that applies criteria developed from the Fines Act and business rules to a bulk number of files selected by the technology systems for a garnishee order,” the Ombudsman said.
“Where all lights are green, a Revenue NSW staff member approves the garnishee orders and the electronic file is transmitted to the banks.”
The Ombudsman also recommended that Revenue NSW “seek expert legal advice on the legality and design of its automated system”, as it held doubts about the continued use of the system.
But almost a year later it found that “no advice had been sought, either externally or from the legal branch of the Department of Customer Service, of which Revenue NSW is part”.
Revenue NSW told the Ombudsman that “it did not consider it necessary to seek such advice, as it considered the changes it had made to its process... had addressed any potential legal concerns”.
“Following our intervention, and to its credit, Revenue NSW took a number of steps to address our concerns about unfairness,” Miller said.
“However, what it did not do... is seek expert legal advice on whether the use of the automation process was lawful and in accordance with the powers under the Fines Act.”
As a result, the Ombudsman sought its own legal advice, which concluded that Revenue NSW’s processes “were not lawful” between early 2016 and March 2019.
“Counsel advised that Revenue NSW’s use of 'machine technology' for the making of garnishee orders... was unlawful because no authorised person engaged in a mental process of reasoning to reach the state of satisfaction required to issue a garnishee order,” it said.
The advice also indicates that changes introduced in March 2019 “do not appear to have fully addressed the legal concerns”, with the manual step inserted into the process “not sufficient for the decision-maker to approve the issuing of... orders”.
“Counsel advised that problems described with the lawfulness of the process could be addressed by modification of the process or legislative amendment,” the Ombudsman said.
Revenue NSW is now “considering further changes to its garnishee system”, according to the Ombudsman.
Miller also cautioned other agencies using "machine technology", particularly in the areas of law policing, child protection and drivers' licence suspensions, to ensure they are using it lawfully.
“We are concerned that other agencies may also be designing and implementing 'machine technologies' without appreciating all the risks, without transparency, and without getting appropriate legal advice,” he said.
The report has suggested the agencies consider seeking parliamentary approval through legislation before adopting 'machine technology' for important administrative functions.
“Seeking express legislative authorisation not only reduces the risks for agencies,” Miller said, adding that “it also gives parliament and the public visibility of what is being proposed, and an opportunity to consider what other regulation of the technology may be required”.
“That might include a statutory right to ask for a person to review any machine decision, or a requirement that the machine’s algorithms be externally validated, and then audited at regular intervals, with those reports to be made publicly available.”