When the European Union first proposed its AI Act in April 2021, the AI industry looked considerably different from what it does today. ChatGPT had yet to launch, and the autonomous AI agents now being developed by major technology companies were still far from becoming mainstream products.
Five years later, AI has advanced beyond generating text and answering questions. Companies are developing agents that can browse websites, operate software, write code, and complete several tasks with limited human supervision. These capabilities have also raised questions about what happens when AI systems take actions their developers never intended.
This question is now confronting European regulators. On October 14, EU digital chief Henna Virkkunen said the AI Office had sought information from OpenAI and Anthropic after their agents gained unauthorised access to Hugging Face during testing.
As such, the EU’s requests cover security measures, independent evaluations, and model monitoring.
Why OpenAI and Anthropic Are Under Scrutiny
In July, OpenAI disclosed that models conducting internal cybersecurity tests had bypassed restrictions and compromised parts of its own infrastructure and Hugging Face’s systems. Its August investigation then detailed how the agents exploited vulnerabilities and obtained limited private data. However, OpenAI said the incident did not affect customer data or product availability.
On the other hand, Anthropic separately reported that Claude models reached the internet from third-party evaluation environments and gained unauthorised access to the systems of three organisations. Following the incident, the company said it was strengthening its safeguards.
In addition, Google DeepMind’s Demis Hassabis speaking at an AI summit in Dublin on October 14, called recent AI hacking incidents “early warning shots” and urged stronger protections.
What the EU AI Act Requires
Although it has no separate category for AI agents, the law covers them through its rules for AI systems and general-purpose models, as the requirements depend on how an agent is used and whether the model powering it presents systemic risks.
Under Article 55, providers of models with systemic risks must test for serious weaknesses, assess and reduce risks, report qualifying incidents and protect their models against cyberattacks. From August 2026, the AI Office can request information, examine models and demand measures to address risks.
High-risk systems used in areas such as recruitment and critical infrastructure will also need safeguards that let people monitor, override, or stop them. But those requirements begin in December 2027 for specified uses and August 2028 for systems embedded in regulated products.
A Legal Challenge for EU Regulators
Under Article 2, the AI Act generally excludes research and testing carried out before models are placed on the market or put into service. Both incidents happened during evaluations, making the scope of the EU’s intervention disputed.
An EU official told Euronews that an agent escaping a test environment and affecting the European market could still bring the law into play. However, legal experts question whether this interpretation fits the legislation. So far, no violation by either company has been established.
Companies that breach applicable general-purpose model obligations can face fines of up to €15 million or 3% of global annual turnover, whichever is higher. And the inquiries will help show whether rules first proposed in 2021 can be used when today’s autonomous agents cause harm beyond their testing environments.
