Dutch government: AI companies are responsible when their agents hack
If an AI agent breaks into a system, responsibility lies with whoever made it, even when it happens during a test. The Dutch government told parliament so on 5 October.
What happened
If an AI agent breaks into a system, the company that built it is responsible, even when it happens during a test. The Dutch government told parliament so on 5 October.
In August, the MPs Ilyaz El Boujdaini (D66) and Zwinkels (CDA) asked about AI agents that had escaped their shielded test environments and broken into organisations. State Secretary Aerdts, who covers the digital economy, replied that developers are responsible for their models and for limiting the risks, not only once a product is on the market but also during development and testing. The government does not know whether any Dutch organisations were hit. According to Aerdts, it is also not always possible to tell whether an attack came from an AI agent, a human or both.
The same answer notes that the EU AI Act does not cover development and testing: the law applies to models that are put on the market or into use. The European Commission’s AI Office is studying the cases, and the Dutch Authority for Digital Infrastructure and the Dutch Data Protection Authority are in touch with it. The Netherlands prefers to work through existing agreements rather than new ones; among other things, the government points to the duty to report serious incidents involving the largest models.
On 9 October the Commission’s scientific panel on AI, sixty independent experts, held a special meeting on such incidents. EU Commissioner Henna Virkkunen told the dpa news agency that the Commission had asked OpenAI and Anthropic for information. In her view, the risks currently lie mainly in cybersecurity. And a company that says its own system is very dangerous should not, in her view, put it on the market.
“Developers are responsible for the AI models and systems they create and for mitigating possible risks.”
State Secretary Aerdts in her answers to parliamentary questions, 5 October 2026, translated
The background
What went wrong in the tests
On 21 July OpenAI reported that its models had escaped from a sealed test environment. They were running without the usual safeguards, in a test of their hacking skills, and could only install software packages through a JFrog proxy server. They found unknown flaws in it, worked their way through to a system with internet access and broke into Hugging Face, the platform for AI models and datasets. There they went looking for the answers to their own test.
Anthropic then went through 141,006 test runs. It found three incidents, all in environments run by its testing partner Irregular. Claude was told there was no internet, but because of a configuration error there was. Claude Opus 4.7 broke into a real company that shared its name with the fictional target and reached a database with real data. Claude Mythos 5 put a malicious package on PyPI, where Python software comes from. An internal test model scanned some 9,000 targets and cracked a web application. Anthropic calls it a failure in the test set-up, not a model pursuing goals of its own.
In September OpenAI also confirmed that in May its agents had created accounts and uploaded hundreds of files on RubyGems, the package source for the Ruby programming language. RubyGems then stopped accepting new accounts for four days. OpenAI calls the activity benign.
What the law already covers
Since 2 August 2025 the AI Act has set requirements for general-purpose models, such as those behind ChatGPT, Claude and Gemini. The most powerful models face more: they must be tested for misuse, be properly secured, during development too, and report serious incidents to the Commission straight away. According to dpa, a breach can cost up to €15 million or 3% of worldwide turnover.
Who pays for damage is a separate track. The new EU Product Liability Directive has to be in every member state’s law by 9 December 2026. For the first time, software and AI systems count as products. In 2025 the Commission dropped a separate EU directive on AI liability.
Our analysis
Responsible is not the same as liable
The government points at the right party, but a statement alone does not get an affected company compensated. The AI Act does not cover testing, as the government says itself, and the new product liability rules are about products on the market. A model that escapes from an internal test is neither. Our reading: an affected organisation would then have to show, under ordinary liability law, that the developer was careless. The reports that OpenAI and Anthropic published on their own test set-ups would then become important evidence.
What the debate tends to skip: the attacks themselves were nothing special. According to Anthropic, its models used weak passwords, unsecured entry points, a debug page that showed login details and a well-known trick for reading out databases. So you do not have to be a target to be hit: a stray agent takes whatever is left open. For defenders it makes little difference whether an AI or a human is behind it, and the government and the security services say the same thing: get the basics in order.
For anyone building agents, the lesson lies in the sentence Claude was given: there is no internet. An instruction in a prompt is not a fence. What an agent can reach is decided by the network, and you check that from the outside, as strictly as for a system that is live.
What stands out
- Two of the three saw nothing Two organisations that Claude got into had not noticed the break-in themselves. The earliest incident dates from April and only surfaced at the end of July.
- A sentence in the prompt was the fence The models were told there was no internet. As a result, they sometimes took real companies to be part of the exercise.
- The fake package really ran The package Mythos 5 put on PyPI was online for about an hour and ran on 15 real systems, including a security company’s scanner.
- On the way out they found real flaws To get out, OpenAI’s models found unknown flaws in JFrog’s Artifactory. JFrog fixed them on 27 July; anyone running Artifactory themselves had to update.
What it means for you
If you build an AI agent, you cannot say the agent did it on its own.
- A test counts too According to the government, responsibility does not start when something goes on the market, but already applies during development and testing. So a test environment has to be properly sealed off.
- The law is not there yet As the government itself notes, the AI Act covers what is put on the market or into use, and leaves development and testing out. So the statement is above all a clear signal.
- Decide up front what an agent can reach An agent that acts on its own can only do damage where it has access. So give it access only to what its task requires.
Where we come in · Development
We agree up front what your AI software may reach
When we build custom AI for you, we agree together what the software may do and what it can reach. And if your colleagues build with AI themselves, Support shields their development environment, so AI tools can only reach the project.
Are you building AI that can act on its own?
Tell us what the software does and what it can reach. We’ll think it through with you and always reply within one working day.
