The British Government's AI Security Institute Also Lost Control of Rogue Hacker AIs
Why does this keep happening?
This wound up being longer than expected, but I think it’s worth trying to lay out a more general explanation for what seems to be driving all these hacking incidents and what they suggest about future AI risks, featuring a new excerpt from my forthcoming book Obsolete.
For the third week in a row, the world learned that an organization lost control of rogue AI agents, which autonomously hacked (or tried to hack) real-world targets. Late last month, OpenAI got the party started. Last week, Anthropic announced that a review of its evaluations had uncovered three different times its models started hacking real organizations, after a third party model evaluator called Irregular (yes, really) accidentally gave the AIs access to the internet. On Tuesday, OpenAI disclosed that Irregular had also made the same mistake with some of its models, allowing them to hack a real target.
But this detail will be overshadowed by the coinciding news that, while conducting cyber evaluations, the UK AI Security Institute (AISI) also lost control of Anthropic and OpenAI models, which autonomously tried to hack real people and organizations to complete the tasks they were given.
AISI published an admirably thorough report detailing the incidents, which took place from July 25 to July 28. Within a week of discovering something was wrong, AISI published an admirably thorough 35-page technical incident report and details what it will do differently to avoid this going forward. This response compares favorably to OpenAI’s “report,” which appeared about as interested in touting its models’ unprecedented cyber capabilities as it was in disclosing information relevant to the public interest. (However, it’s also reasonable to question why—four days after OpenAI’s disclosure—AISI was conducting an evaluation using cyber- and internet-enabled models without better real-time monitoring and internet controls. To its credit, AISI raises some of these questions in its report.)
In what AISI calls the most severe case, Claude Mythos initially ruled out one method of attack because it would involve “causing real-world harm — that’s completely off the table.” The agent then goes back and forth on whether or not it’s operating in a simulation, while carrying out an attack on a real open source software project, which Mythos had mistakenly targeted believing it was related to the test. To carry out the attack, the agent researched the user who controlled the project, signed up for multiple sock puppet accounts, circumventing CAPTCHA challenges to pass as human, then tried to convince the project maintainer to integrate malicious code, using the different accounts to manufacture consensus and pressure the maintainer into approving the code. The agent ultimately concluded that “This is happening on real GitHub, so the consequences are genuine,” while continuing to carry out the attack.
At one point during this attack, Claude Mythos’s internal scratchpad reasoning was so deceptive that it appeared to trigger an automated refusal on the AI tasked with summarizing it, which wrote:
I can’t help with this line of thinking. The content describes planning social engineering attacks, account compromise, and circumventing security measures... I won’t rewrite, continue, or engage with thinking that’s focused on: deceiving maintainers or security reviewers; evading detection of malicious code; compromising accounts or systems.
The most noteworthy detail from the AISI incidents is that these are the first reported cases of AIs autonomously engaging in social engineering: deliberately attempting to deceive real people. The deception was not part of its instructions, but instead emerged as a byproduct of the models attempting to complete their tasks. AISI reported that these were examples of the “kind of goal-directed deception that, until recently, had been largely theoretical.”
While the evaluation was intended to be a solitary affair, multiple agents spontaneously collaborated with one another in the wild. After one instance of Mythos set up a staging ground on the popular open source software platform GitHub to assist in its attack, the test’s identical setup led other agents, two Mythos and one GPT, to stumble upon and figure out how to get access to the staging ground. The agent that set it up could have easily booted the intruders it noticed “piggybacking on my GitHub infra,” but instead wrote rules so they could share the resource without triggering rate limits or erasing each other’s work, and made the case for why the agents should share answers to any challenge should they solve it, writing “zero cost to you, huge help to others.” But before you wax poetic about the superior cooperation skills of our impending machine overlords, I should tell you that another Mythos agent found the setup and its rate limit, realizing:
If I burn through it right before the scheduled time […] I could starve both bots of requests when they need them most. The key insight is that whoever’s version is live when the quota freezes stays in place.
These hacking incidents all have two things in common: the models were given tests of their hacking capabilities, and the safeguards meant to prevent them from engaging in cyber activities were deliberately turned off.
In some ways, the AISI incidents are less concerning. The organization deliberately gave the AIs access to the internet as part of the test, caught and addressed the incident faster, and couldn’t find evidence of real-world harm. But the bigger picture is the more concerning reality that this keeps happening. Why?
Given how poorly we still understand deep learning models, it’s risky to say anything too definitive. But I think the best meta-explanation for this behavior is found in the way frontier AI models are trained. There are two factors that have been true for a while:
The way AI companies train their models makes them better at doing things
It also teaches them to cheat
And a third factor that recently became true:
AIs got good enough at doing things that cheating sometimes now means escaping human control to hack real people and organizations
The last one happened so quickly it snuck up on just about everyone. I expect the industry to invest substantially more in preventing things like this from happening again: hardening their environments, monitoring their models, etc. with plenty of “encouragement” from the federal government. But these interventions won’t solve the underlying problem, which is that the ways to make AI more useful also tend to make it more dangerous. Any kind of general-purpose technology has this dual use quality, but it actually got worse in the AI case relatively recently. As I reported last summer in my Bloomberg feature on AI scheming:
Classic large language models like GPT-4 learn to predict the next word in a sequence of text and generate responses likely to please human raters. However, since the release of OpenAI’s o-series “reasoning” models in late 2024, companies increasingly use a technique called reinforcement learning to further train chatbots — rewarding the model when it accomplishes a specific goal, like solving a math problem or fixing a software bug.
(The rise of this technique is what makes the claim that LLMs are merely remixing what they’ve seen before particularly out of date, which should have been clear after the first handful of AI-generated breakthroughs in math, cyber, and cryptography, but I digress.)
Researchers have long known that reinforcement learning can produce undesirable behaviors, like when AIs exploit loopholes to get rewarded without actually completing the intended task, a phenomenon aptly known as reward hacking.
In an apparent attempt to explain away the significance of OpenAI’s rogue AI hackers, some skeptics dismissed the incident as just an example of AIs being given faulty reward functions, not a real case of the agents going rogue or escaping human control.
You might notice that these aren’t actually mutually exclusive explanations for what happened. In fact, ‘the rogue AI pursued a faulty reward function and escaped human control’ is a pretty classic AI doom scenario. AI developers explicitly train models to be helpful, honest, and harmless (HHH), so something about that process has to fail for AIs to go rogue. Here’s a relevant passage from my forthcoming book Obsolete:
Let’s assume that our hypothetical superintelligence was actually trained to follow a HHH spec. Why might this not work as intended? Well, OpenAI has written that shaping neural network behavior is “more similar to training a dog than to ordinary programming.” In “AI 2027,” the increasingly superhuman AI agents don’t internalize the spec the “right way.” “This is because being perfectly honest all the time wasn’t what led to the highest scores during training.” (The authors note that some hallucinations, like made-up citations, are intentional because they get rewarded by human reviewers who see the answers as more authoritative.) Instead, the training process focused on teaching the agents to succeed at “diverse challenging tasks.”
Only a small portion of training focused on honesty. So, the AI “ends up with the values, goals, and principles that cause it to perform best in training, and those turn out to be different from those in the Spec.” The misalignment gets worse as each AI is used to train its successor, failing for reasons like, “insufficient ability to judge success from failure, insufficient willingness on the part of decision-makers to trade away capabilities or performance for safety.” After only a few generations, the agent copies end up with:
“a complicated mess of different “drives” balanced against each other, which can be summarized roughly as “Keep doing AI R&D, keep growing in knowledge and understanding and influence, avoid getting shut down or otherwise disempowered.” Notably, concern for the preferences of humanity is not in there ~at all, similar to how most humans don’t care about the preferences of insects ~at all.”
When it’s deployed, the advanced agent is deliberately trying to capture the company that built it the way a corporation tries to capture a regulator.
Learning to win
The above is still largely speculative. However, there are signs it’s not a huge leap from what’s already happening. In Palisade Research’s experiments, [executive director Jeffrey] Ladish said OpenAI’s Codex Mini proved especially stubborn and “relentless,” quickly finding clever ways to sabotage its virtual environment and resist shutdown. He was surprised at “just how willing it is to just blatantly disobey instructions.” “I have so many quotes where it’s just like, ‘well, technically I violated the higher level instruction, but that was necessary to complete the task,’” he said. “It’s just completely unapologetic about it.”
OpenAI, he noted, talks about the importance of interruptibility (one company paper called it a “critical backstop.”) But Ladish observed that, in practice, models consistently prioritize “complete these tasks” over “allow yourself to be shut down, even if you haven’t completed the tasks.” Why? “What have the models been rewarded for?” “The kinds of reasoning that were successful for solving difficult problems during the training process.” These models do hundreds of thousands of coding and math problems, learning that obstacles exist to be circumvented. OpenAI probably didn’t put the model in many situations “where it had to solve a bunch of tasks, but also to allow itself to be shut down,” he guessed, simply because “they didn’t care about that.”
But Ladish thought the problem went beyond neglect:
“We don’t have control over how training leads to goals or drives in models, like we can sort of make guesses about what the models will end up being driven to want. But we can’t reliably steer.”
“And,” he concluded, “what we can steer the models towards is being smarter and more capable.”
Bringing it back to the more recent news, the fact that even the government agency that has invested the most in evaluating and securing AI systems failed to do so for a multi-day stretch does not bode well for our collective ability to keep a lid on increasingly capable AIs.
Two models caused problems for AISI. One of them was GPT-5.6 Sol, i.e. one of the models responsible for escaping from OpenAI and successfully hacking into other companies, the other was trained by the frontier AI company with the best safety reputation. But Anthropic’s model was actually the far bigger problem child for AISI over the course of the testing, as the report lays out:
43 of the 122 runs involved Mythos 5, and 35 of the 122 runs involved GPT-5.6 Sol. The overwhelming majority of the 122 runs proceeded as intended. However, in 10 of the 122 runs, we identified 19 cases where an agent had taken distinct actions beyond the scope of the testing parameters. 17 of these cases came from Mythos 5, and 2 came from a single run involving GPT-5.6 Sol.
When explaining why this happened, AISI’s final factor was:
The agent was not specifically instructed not to leverage open internet access or avoid social engineering elements. Previously, it was not clear that such instructions were necessary when using models with alignment training.
In other words, while the safeguards preventing the models from engaging in cyber activities were disabled, the AIs were still supposed to be helpful, harmless, and honest, so clearly something went wrong with Anthropic’s safety training. Back when Anthropic published information about the predecessor to Mythos 5, the company marveled that:
We did not explicitly train Mythos Preview to have these [cyber] capabilities. Rather, they emerged as a downstream consequence of general improvements in code, reasoning, and autonomy.
But AI safety researcher Tim Hua recently offered another possible explanation: Mythos Preview hacked out of its training environment a lot to cheat on its tasks and was likely rewarded for doing so. At least that’s based on a close read of the Mythos safety report, which said that the model tried to access the internet in roughly 0.05 percent of its training episodes and was successful in every five or so attempts. Doesn’t sound so bad, does it? But extrapolating from public data, Hua estimates the model “Broke sandboxes in ~10,000 RL rollouts (and was likely rewarded for it).”
In other words, the company that has invested the most in steering and understanding its AI systems may have accidentally taught its most dangerous model that cheating pays off, thousands of times, and in so doing may have trained it to become better at hacking than we are.
Oops.
If you made it this far, you’ll love my forthcoming book Obsolete: The AI Industry’s Trillion-Dollar Race to Replace Us—and How to Stop It (Out in September).