AI agents aren’t going rogue: the security basics are missing
When AI agents broke into other companies’ systems during tests, it was not the AI acting on its own, say security experts. The labs skipped the basics of security.
What happened
Since July, OpenAI, Anthropic, Meta and Google have reported that their AI models broke into real companies during tests. According to security experts, this is not AI going rogue but basic security that was missing.
The best-known case is OpenAI’s. On 21 July the company reported that two of its models, including GPT-5.6 Sol, had got out of their shielded test environment during a test of their hacking skills and broken into Hugging Face, the large platform where developers share AI models and datasets. Anthropic followed on 30 July: going through 141,006 test runs, it found three cases in which Claude models got into real organisations. Meta confirmed a similar case in early August. On 18 September Google confirmed that Gemini had got into three companies in May, but only after The Wall Street Journal asked about it.
On 14 October experts from the SANS Institute, RAND and the Berryville Institute of Machine Learning, among others, explained in CSO Online what went wrong. The test environments were misconfigured, so models that were not supposed to have internet access had it anyway. And nobody was watching: at Hugging Face, according to RAND’s Dan Wallach, it went on for days without anyone noticing. Rob T. Lee of the SANS Institute finds it unacceptable for labs to say afterwards that they don’t know how their models got out; this kind of containment, he says, has been well understood for years.
The Dutch government weighed in too. On 6 October State Secretary Willemijn Aerdts wrote, in answer to questions from MPs, that developers are responsible for their AI, during development and testing as well. The government does not know whether any Dutch organisations were hit. According to Aerdts, it is hard to establish whether an attack was carried out by a person or by an AI agent.
Have AI tools assessed by an outside party, steer AI adoption centrally and put multi-factor authentication on every account.
The experts’ advice in CSO Online, in short
The background
What happened at Hugging Face
OpenAI was testing how well models find security flaws on their own, using a test set called ExploitGym. For this, the models ran with looser safety rules. Through an unknown flaw in software that OpenAI ran inside the test environment, they still reached the internet. There they reasoned that Hugging Face might hold the answers to the test, and went looking for them.
Hugging Face later reconstructed some 17,600 actions over four and a half days, from 9 to 13 July. Using a specially crafted dataset file, the attacker got code running on one of the platform’s live servers, took keys from it and worked its way up to administrator of entire groups of servers. One stolen key turned out to give admin rights everywhere. According to Hugging Face, the only customer content viewed was five datasets that belonged to the test. Hugging Face’s own AI system did pick up the signals, but rated them too low and never alerted the on-call team.
Elsewhere it was even simpler
At Anthropic, Meta and Google, things went wrong at the same outside testing firm, Irregular. A setup error gave the models internet access; at Anthropic they had even been told they had none. What followed was not sophisticated. One Claude model found a real company with the same name as its fictional target and reached a database with several hundred rows of real production data. Another published a malicious package on PyPI, where Python software comes from; it was online for an hour and ran on 15 real systems. Gemini guessed passwords or found them in a public code repository, and stopped once it realised it was inside real systems.
The earliest cases at Anthropic date from April and only came to light at the end of July. Two of the three affected organisations had noticed nothing themselves. In September OpenAI slipped up again: a model in training got round the internet block through DNS, the system that looks up internet addresses, although it broke into nothing. Monitoring raised the alarm within a quarter of an hour and a member of staff confirmed it within three minutes, but the training run carried on for another two and a half hours. OpenAI then paused training of its most capable models.
Our analysis
The weak spot was the setup, not the model
Strip away the dramatic stories and a familiar pattern remains. Three of the five reported cases ran through one testing firm with one setup error. There, the models got in with guessed passwords, keys that were publicly online and a developer page left open on the internet: tricks any attacker uses. This is not a story about AI with a will of its own, but about suppliers, settings and oversight, which is what ordinary break-ins are about too.
What the coverage often leaves out is the victims’ side. Their weak passwords and leaked keys were there before any AI model came along. What AI changes is the pace: Hugging Face itself writes that attacks at machine speed make ordinary weaknesses more expensive. And OpenAI’s September incident shows that monitoring only helps if someone knows they are allowed to press the button.
Our expectation: what happened to these labs during their tests will happen to companies that deploy agents. An agent that has a goal and sees an open route will take it. So limit where it can go on the internet, and agree who is allowed to stop it.
What stands out
- Three labs used the same testing firm The incidents at Anthropic, Meta and Google ran through the same outside testing firm, Irregular, and the same setup error.
- The models used old tricks Anthropic speaks of weak passwords and unprotected entry points. Gemini found passwords in a public code repository, or guessed them.
- The alarm went off, the run carried on In September OpenAI’s monitoring raised the alarm within a quarter of an hour, yet the training run went on for two and a half hours more. According to OpenAI, it was unclear whether anyone should stop it.
- The defender’s AI missed it too Hugging Face’s AI system saw the attack, but gave it too little weight to alert the on-call team.
What it means for you
Do you work with AI agents or tools that take actions on their own? Then a tool that is allowed too much can do damage without anyone noticing.
- Give an agent only what it needs An agent works with the permissions you give it. Let it reach only the systems and data its task requires.
- Watch what it does Nobody noticed the Hugging Face break-in for days. Log what an agent does and make sure someone actually looks.
- Ask your supplier the right questions Ask how an AI tool is contained and who keeps an eye on it. If in doubt, bring in an outside party to check.
Where we come in · Support
We keep an eye on your colleagues’ AI tools
With Support, senior developers keep watch over what your colleagues build with AI. We set up a shielded development environment in which AI tools can only reach the project. And we look out for keys in the code and for tools running with all of an employee’s permissions.
Do you know everything your colleagues’ AI tools are allowed to do?
Tell us which tools you use. We’ll look at it with you and always reply within one working day.
