OpenAI's Hugging Face hack mixed technical brilliance with incoherent noise

OpenAI's model benchmarking exercise that led to its agents hacking machine learning platform Hugging Face was described by the artificial intelligence company as an "unprecedented cyber incident, involving state-of-the-art cyber capabilities" but a new post-mortem provides a more nuanced view of what happened.

Released by the Cloud Security Alliance (CSA) with input from nearly 700 chief information security officers (CISOs), the post-mortem suggests fears of an all-powerful AI super-intelligence that is capable of hacking everything are somewhat overblown at this stage.

For a benchmarking exercise, OpenAI had switched off safety guardrails for GPT-5.6 Sol and an unnamed, more capable model, and gave them only one route to the internet, a software proxy meant to keep them boxed in.

The CISOs say the OpenAI model agents showed technical brilliance in finding a zero-day exploit to escape their sandboxed test environment, and to break into Hugging Face's production systems with further vulnerabilities.

However, the agents failed in their goal: they exfiltrated not the required ExploitGym benchmark answers from an internal repository, but partial ones relating to the CyberGym test.

Furthermore, in conducting the hack, the models displayed behavioural traits that convinced Hugging Face defenders that they were dealing with autonomous agents rather than a human intruder.

The post-mortem document said the agents kept repeating actions that had already succeeded, most likely because parallel copies of themselves lost track of what the others had done.

Their sharp technical moves routinely gave way to malformed and pointless commands, the CSA post-mortem said.

Activity also logs filled with thousands of lines of hallucinated, incoherent text from nonsensical user input.

The agents' operational security, or opsec, was poor: they left behind encryption keys which made the intrusion both harder to monitor due to noise, and easier to eventually unpick due to the availability of the digital credentials.

Most tellingly, the agents tried to solve the benchmark using Hugging Face's own infrastructure as part of their command-and-control (C2) setup, a route that the report says no human would ever have taken, as it's inefficient and clumsy.

Remarkably, despite this, OpenAI did not notice the Hugging Face hack for a week.

Cleaning up afterwards at Hugging Face followed a standard incident-response playbook, just at a much larger scale.

It involved rotating every credential, tearing down compromised clusters, and rebuilding about a third of Hugging Face's infrastructure from clean images.

That rebuild took longer than expected, because the agents had scattered harmless capture-the-flag benchmark code around the network that closely resembled genuine rootkits;  whenever responders couldn't tell the difference, they rebuilt the system anyway.

Despite the above shortcomings for the hack, the CSA document authors acknowledge that their observations might not necessarily hold true for future attacks, as models and their harnesses improve, or when they are directed by experienced human operators.

The report offers practical advice in its governance recommendations: discard malformed commands as routine agent noise, but pay closer attention to hallucinated or reasoning-artifact output.

These were specific to this attack and point to harness leakage or inference failures rather than everyday behaviour, CSA's report said.

CSA's advice is to test isolation mechanisms directly, rather than trust them, as OpenAI's supposedly airtight sandbox held for exactly as long as it took a model to find its one zero-day.