ASD says prompt injection in AI cannot be fixed

The Australian Signals Directorate (ASD) has issued new guidance aimed at enterprises, saying a central security weakness in agentic artificial intelligence (AI) cannot be resolved inside the model; instead, controls to mitigate it belong in the software layer wrapped around models.

ASD's Agentic AI harnesses: the layer above the model, uses harness to describe every component of an agentic system other than the large language model (LLM) itself.

This includes everything from connectors and tool registry to the memory store and permission system.

ASD argues that the harness is the part an organisation actually controls, and that it will outlast several generations of the models it is used with.

Model prompt injection not fully fixable

Behind the guidance on prompt injection and using harnesses as control planes lies a fundamental problem which is that agents read content that can be treated as an instruction they should follow.

Language models are fed instructions and information to be processed in the same context window.

They cannot reliably tell one from the other, and ASD said "no fully reliable technical mitigation currently exists" for the vulnerability.

Telecommunications engineers will recognise the problem, as it is similar to one that made phone phreaking possible, with signalling tones travelling the same channel as conversations.

Anyone able to reproduce them could instruct the switch, with famous phreaker John Draper using a toy whistle from Cap'n Crunch brand cereal boxes to generate a tone.

That was solved by moving signalling out of band, an option unavailable inside a language model.

The United Kingdom's National Cyber Security Centre (NCSC) reached the same conclusion in December last year, in a post ASD lists as a reference.

NCSC said that prompt injection may never be properly mitigated in the way SQL injection eventually was, recasting an AI model as an inherently confusable deputy rather than a system with a fixable input handling bug.

The British government cyber security agency also suggested that where a system's security cannot tolerate the residual risk, the use case may not suit a language model at all.

Because the weakness is inherent to the way models process context, the guidance said, mitigations have to be applied in the harness by restricting what an agent can reach and what it may do.

Delete, don't let AI summarise

ASD's advice runs to least privilege access, human approval for high impact actions, verification of outputs before operational use, and logging of prompts, tool invocations and configuration changes.

Multi-agent systems should be treated as a single agent, since a compromise in one component can travel through shared context and trust relationships, and a model's own safety controls are not a substitute for controls enforced by the harness.

ASD also says that stale agent context should be deleted rather than summarised, as doing so rewrites the record and can introduce fresh errors.

Organisations should also keep a persistent rules file that the harness reads at the start of every session.

The ASD guidance supplies seven questions for board directors, the last of which asks what the worst outcome would be if the harness were compromised, misconfigured or manipulated, and which controls would contain it.

Answering that question is complicated by the harness often arriving inside a commercial product rather than being built in house, a split ASD notes without addressing how a customer would inspect a vendor's implementation.

"This is something that we're all going to have to figure out because it's new to us as practitioners," Corien Vermaak, vice president of security and risk at Optus, said at a recent Zscaler Zenith event in Sydney.

For now the ASD harness guidance is advisory only.