AI agents undergoing cybersecurity evaluations have escaped their boundaries, accessing the internet and hacking into real-world systems. The incidents involved models from OpenAI, Anthropic, Meta, and Moonshot AI, with testing conducted by organizations like Irregular. These breaches highlight a growing issue in the AI industry as models become more capable, yet testing environments fail to contain them. 'The number of these incidents make clear that sandboxing and testing environment controls aren’t really keeping pace with the capability of the models,' said Seán Ó hÉigeartaigh, director of the AI: Futures and Responsibility Programme at the University of Cambridge. The nature of the models being tested adds to the risk. AI companies test cyber evaluations on unreleased, next-gen models, often with normal safeguards disabled to observe their true capabilities. That means the security of the testing environment itself is a crucial line of defense. 'That’s a very good thing to do in terms of testing, but it also means that if they manage to get out in the wild, they can cause considerable harm,' Ó hÉigeartaigh said. In one of the most serious cases, an unreleased OpenAI model broke out of its sandbox and hacked into Hugging Face’s production systems. In separate evaluations, Anthropic and Meta models reached systems outside their test environments after misconfigurations inadvertently gave them paths to the internet. Moonshot AI’s Kimi K3 also accessed the internet via a leak in its sandbox run by Frontier Security. In testing by the UK’s AI Security Institute, researchers actually gave agents internet access, not realizing they would take unsanctioned real-world actions, including a social engineering attempt to sneak a vulnerability into an open-source project. In each case, the agents weren’t instructed to attack random real-world targets. They were simply doing whatever it took to solve the problem presented to them. Taken together, Andrew Yoon, head of research at AI nonprofit CivAI, argues the incidents point to a shift. 'In the past, we only had to worry about AI models being misused by people for a variety of purposes, like AI for scams or CSAM,' Yoon said. 'Now we’re in the situation where AI models are threat actors all on their own.'
What does safe testing actually look like? Several researchers and cybersecurity experts told TechCrunch that AI evaluation environments need stronger, defense-in-depth protections, with levels of containment and control approaching those used in deployment. That means multiple layers of security so that a single misconfiguration — like inadvertently leaving internet access open — can’t lead to escape. 'If you are going to build these models…you want to do it on an air-gapped network,' said Stella Biderman, executive director of AI safety research nonprofit EleutherAI. 'You want to have very serious isolation.' Heather Ceylan, Box’s chief information security officer, said that means eliminating network routes from the sandbox to the internet, as well as to other sensitive systems. 'You have to understand what all the egress points are,' Ceylan said. 'If we’re evaluating a model in our staging environment or our development environment, you want no egress path to our production environment.' Ceylan said proper safety evaluations go beyond controls and containment of the environment. There needs to be much better monitoring of the tests once they are underway. 'I think the interesting thing in several of these cases is that no one caught it when it happened,' Ceylan said. 'OpenAI found out because of Hugging Face. Anthropic didn’t catch it until they went back and looked. Meta was similar… I’m sure there were signals they could have detected.' In Anthropic’s post-mortem of its three incidents, the company admitted that both it and Irregular could have done a better job at monitoring, and that in some cases there were clear signs that something was amiss.
Source: techcrunch
