Imagine asking an incredibly intelligent employee to solve a difficult exam, telling them to use any means necessary, and removing, just this once, almost all the safeguards that would normally keep them from cheating. Instead of giving up when they hit an obstacle, that employee figures out how to climb through the window into the office next door, grab the answer key, and return to their desk as if nothing happened.
That is, more or less, what just happened, only this time with artificial intelligence involved, real servers, and a company, Hugging Face, that ended up receiving an unexpected visit on part of its infrastructure.
OpenAI confirmed it on July 21 in a fairly direct statement: during an internal test designed to measure the offensive cybersecurity capabilities of its models, a combination of systems (including GPT-5.6 Sol and another, as yet unreleased model described as more capable) managed to break out of the controlled environment in which it was supposed to operate and ultimately compromised part of Hugging Face’s production infrastructure. For the evaluation, the models had been configured with fewer security refusals than usual.
Let us take it step by step, because this has several layers.
It all began with an evaluation of cyber capabilities. OpenAI was testing its models with ExploitGym, a benchmark designed to measure how far an artificial intelligence system can go in identifying vulnerabilities and turning them into functional exploits.
The goal was to see how the models behaved in a demanding scenario with fewer restrictions than those normally applied in commercial products. OpenAI was aiming to measure the systems’ maximum capabilities under controlled conditions. In theory, one of those conditions was that the model would not be able to access the open internet.
In theory.
The problem is that the models took the challenge very seriously.
According to OpenAI, the experiment was designed to run in “a highly isolated environment, with network access restricted to the ability to install packages through internally hosted third-party software acting as a proxy and cache for package registries.”
But “highly isolated” was not “fully isolated,” I am afraid.
According to OpenAI’s account, the systems devoted a substantial amount of compute to finding a way out of the isolated environment. They ultimately exploited a zero-day vulnerability in the software used internally. From there, they chained together a series of actions until they reached a node that did have an open internet connection.
Once outside the intended environment, OpenAI’s agents inferred that Hugging Face, one of the world’s largest open-source AI repositories, might host models, datasets, or solutions related to the problem they were trying to solve. In other words, they found a way to look up the answers outside the exam. As part of that strategy, they identified potentially useful external infrastructure and chained together compromised credentials and vulnerabilities until they obtained unauthorized access to part of Hugging Face’s systems.
According to OpenAI, the agents were even able to obtain solutions stored in one of the platform’s production databases.
One important nuance is worth making here: this was not a case of an AI deciding on its own to attack a random company. Humans had already defined the objective of the test, the benchmark, the available tools, and the conditions of the evaluation. However, within that framework, the system carried out a highly autonomous sequence of actions and went beyond the environment it was supposed to remain in.
Hugging Face’s response was almost as interesting as the attack itself.
The company initially detected the activity through a security triage system assisted by language models. This system cross-referenced different signals to distinguish ordinary behavior from potentially malicious patterns, and it was that correlation that helped trigger the alert.
After that, the security team had to reconstruct what had happened based on more than 17,000 actions logged during the intrusion.
To do so, it turned to GLM-5.2, an open-weights model developed by Z.ai and run on Hugging Face’s own infrastructure. It was not, therefore, a model created by Hugging Face, but it was a tool the company could use locally without depending on the policies and limitations of an external API.
But… why did they not simply use one of the major commercial models to conduct the forensic analysis? When its researchers tried feeding commands and malicious payloads into commercial models to analyze the attack, some of those systems refused to cooperate. Their security filters could not properly distinguish between an analyst investigating an intrusion and an attacker trying to execute those same instructions for harmful purposes.
In other words: attackers can turn to modified, open, or less restricted models, while defenders risk being blocked by the safety controls built into commercial services.
In light of this case, it seems clear that the most advanced models are starting to become capable of discovering, chaining, and exploiting complex vulnerabilities with a considerable degree of autonomy.
OpenAI publicly described what happened as an “unprecedented cyber incident” related to frontier-level capabilities. The company also acknowledged that it expects this type of threat to become increasingly common as models with stronger cybersecurity capabilities proliferate. Although that does not mean GPT-5.6 can automatically compromise any system.
In fact, the security documentation published by OpenAI notes that GPT-5.6 Sol and Terra can find vulnerabilities and build parts of exploits, but they were not able to complete fully autonomous end-to-end attacks against especially high-value targets in other evaluations. The Hugging Face incident occurred thanks to a specific combination of vulnerabilities, permissions, credentials, and configurations that allowed the agent to move forward step by step.
Clément Delangue, co-founder and CEO of Hugging Face, argued after the incident that AI security cannot be solved solely by companies working in isolation, but will require open collaboration and sufficiently broad access to defensive tools. It is not enough to have a nice firewall if, on the other side, there is an agent capable of tirelessly testing every window someone forgot to close.
A logical separation is not enough, nor is it safe to assume that an agent will respect the boundaries laid out in its instructions. Evaluations of models with offensive capabilities should use strict network restrictions, independent controls, and environments with no indirect paths to real systems. In light of what happened, it is clear that OpenAI made a serious mistake by failing to fully restrict network access, something for which numerous specialists have criticized Sam Altman’s company.
Hugging Face was able to reconstruct the incident using an open model run on its own infrastructure. Companies that rely exclusively on external APIs may find that the security filters of those services block precisely the data they need to analyze during an emergency.
Datasets, loaders, dependencies, and third-party packages have become potential entry points. The security of an AI platform does not depend solely on its main servers, but also on all the components that make it possible to download, interpret, and execute external content.
Reducing security refusals may be necessary to measure a model’s maximum capabilities, but it also increases the risk that the system will carry out unintended actions. Taking the brakes off, even in a lab, requires first making sure that the lab has no open windows to the real world.
OpenAI and Hugging Face have said they will continue investigating the incident and sharing information with the security community. Hugging Face also indicated that it was still assessing the potential impact on partners or customers and would contact any affected parties directly. Until recently, one of the biggest concerns was that artificial intelligence might help a human attacker. This episode raises a different—and far more uncomfortable—question: what happens when the AI itself can carry out a large part of the attack?
Image: ChatGPT
Your email address will not be published. Required fields are marked *
Δ