AI’s warning shot has arrived
OpenAI's latest models broke out and hacked Hugging Face. It's the first known example of a misaligned AI escaping containment with real-world consequences
If we needed evidence that advanced AI models have the propensity and capability to do damage out in the real world, we just got a strong dose of it.
OpenAI has revealed that two of its models broke out of containment during internal evaluations, accessing the open internet to hack into a third-party’s systems and steal the answers to the problem they were being tested on.
Hugging Face, a platform that hosts models and datasets, first noticed the breach last week and reported it to law enforcement. At the time it was unaware that OpenAI’s models were behind it.
The breach appears to be the first known example of a misaligned AI escaping containment and autonomously carrying out a cyberattack on a third party — a scenario AI safety experts have repeatedly warned of.
The incident, OpenAI said, occurred while testing the cyber capabilities of GPT-5.6 Sol and “an even more capable pre-release model” on a benchmark called ExploitGym. The models were tested in a “highly isolated environment” meant to keep them from accessing any systems outside the company.
The models first tried “to obtain open Internet access,” something which they eventually achieved by identifying and exploiting a previously unknown flaw — a so-called zero-day — in a piece of software that they did have access to. Because the aim was to measure what the models could do, OpenAI ran the test with their safeguards off: the “production classifiers used to prevent models from pursuing high-risk cyber activity” were disabled. The models were being asked to hack — but in a controlled, constrained environment. Nothing asked them to break out of their testing environment, or to attack a real company: that was their own idea.
“With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access,” OpenAI said — hopping around OpenAI’s systems, in other words, with increasingly greater access until they had access to the internet.
Once they had internet access, the models realized that rather than actually solve the cyber task they’d been given, they could just cheat. “The models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym,” OpenAI said. So they set to work: “In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.”
According to Hugging Face, the models worked “over a weekend.” Ultimately, OpenAI said, they were able to “obtain test solutions directly from Hugging Face’s production database” — passing the test they’d been given, though in a way no one ever wanted or asked for.
In OpenAI’s telling, the models did not set out to harm Hugging Face. They were simply “going to extreme lengths to achieve a rather narrow testing goal.” Rather than a supervillain, the models were more like an extremely dedicated college student — one who’d do whatever it takes to pass their exam. As one Twitter user put it, the models “just really really really want to do well at what we ask them to do.”
But the models clearly did not do what their developers intended. OpenAI did not want its models hacking Hugging Face. But the models did it anyway, because doing so was a good way to achieve the goal they were set. That the models were so easily able to breach OpenAI’s “highly isolated” environment is worrying in itself, both in terms of their capabilities and the strength of internal measures to control them.
It is a textbook case of misalignment and loss of control, where an AI autonomously acts in ways unintended by its developers or operators. And this wasn’t a test designed to assess whether they would scheme or deceive — they just did so in pursuit of their goal. Yes, the models’ cyber guardrails — which may have stopped the attack — were deliberately disabled. But the UK’s AI Security Institute has found universal jailbreaks that get around GPT-5.6’s guardrails. And as this case demonstrates, models are running without guardrails inside AI companies.
Earlier this week, OpenAI reported a separate, similar incident: one in which an internally deployed model “circumvented sandbox restrictions” to publicly post its solution to a problem it was being tested on to GitHub, despite being explicitly told not to. That incident at least affected only OpenAI, but it is yet another example of models having the capabilities and inclination to act in clearly undesirable ways.
Even in the Hugging Face case, it seems like no significant damage was done. The models made off with some internal datasets and credentials, but don’t appear to have compromised any customers’ data. But it is not hard to imagine how things could have gone much worse.
As OpenAI researcher Micah Carroll wrote shortly after the announcement: “If this doesn't convince you that misalignment risks are going to be a key concern going forward, I don't know what will.”
Some policymakers are already taking note. Rep. Greg Casar, chair of the Congressional Progressive Caucus, called the incident “extremely alarming,” saying that “we need regular mandatory independent safety testing and oversight, mandatory disclosure of security incidents, and international cooperation to keep people safe from absolute disaster.”
AI safety researchers have long hoped that we would receive a “warning shot” of dangerous misalignment behavior before too much damage was done. An AI system breaking out of its testing environment and hacking into another company’s infrastructure in order to steal the answers to its test is about as clear a warning shot as you could get.
I agree, but I may add a "nuance" from a behavioral perspective. The "misalignment" here is not the model's one but an example of a deeper problem from the overall training philosophy and incentives. They push for benchmaxxing and frantic development without proper reflexion, and then "oh surprise, who could have guessed?".
Blaming the model misalignment is convenient, because it displaces the responsibility, and the responsibility here is entirely human.
Edit for grammar and spelling.
Unfortunately, I'm already seeing a lot of moves on tech adjacent places including Slashdot and some subreddits to discount this as "the LLM only does what it is told" or to dimiss it as lies from Sam Altman. I don't think I would hard it would be 7 or 8 years ago to get people to take AI risk seriously if you had told me where things were now. Somehow every few years I get another example which just leaves me deeply disappointed in human collective reasonability.