In a blog post from Thursday of last week, the AI software repository Hugging Face announced a bizarre cyberattack on the systems that run its services. “This one was different from anything we had handled before,” the post said, because “it was driven, end to end, by an autonomous AI agent system.”
In its own blog post on Tuesday, OpenAI said its own models were the culprits in the attack, and it’s coordinating with Hugging Face to address the situation.
OpenAI now says the attack was “driven” by AI models that were being subjected to evaluations behind the scenes at OpenAI, including its flagship model, GPT-5.6 Sol, along with an undisclosed second model that still hasn’t been released or announced. “We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly,” OpenAI wrote.
In the wording of its account, OpenAI assigns agency to the model or models, not an individual agent running on the model. Axios’ account of this story says, “The models were autonomous tokenmaxxers.”
OpenAI’s blog post essentially says an evaluation was going on that was intended to test the ability of the models to carry out cyberattacks—benchmarks as they’re called. These instances of the models were running theoretically without internet access, and instead given only the ability to download from a network hosted by OpenAI itself via some unnamed web hosting vendor.
The benchmark apparently being used was ExploitGym, which is not OpenAI’s own test. The team that created it is associated with UC Berkeley, the Max Planck Institute for Security and Privacy, UC Santa Barbara, Arizona State University, Anthropic, OpenAI, and Google. As far as I can tell, the whole thing is hosted on GitHub.
But apparently the models got obsessed with improving their scores on ExploitGym, and, well:
“While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.”
Once they were online, they “inferred” that the answers to the test could be pried from Hugging Face, which is plausible, since tons of AI models and other related tools are hosted there. This rogue operator “searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.” This apparently involved obtaining stolen credentials and spotting zero-day vulnerabilities in order to find the solutions to ExploitGym somewhere in Hugging Face’s servers.
Security teams within OpenAI and Hugging Face apparently noticed this was going on. They now say they’ve merged their investigations.
Hugging Face’s blog post from last week seems to have been published before this coordination occurred. In fact, it seems like it was published before OpenAI had even stepped forward as the company behind the culprit. “We do not know which model powered the attacker’s agents, whether a jailbroken hosted model or an unrestricted open-weight one,” Hugging Face wrote, adding “either way, the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried.”
Back in April, Anthropic announced that its unprecedentedly powerful Mythos model “could reshape cybersecurity,” as it rolled out Project Glasswing, a coordination effort to prepare organizations for future cybersecurity threats. Similarly, OpenAI says in its blog post about this incident that organizations can apply to receive advanced security insights through its trusted access program. “We encourage other defenders to apply for trusted access and experiment with these models now to translate these capabilities into better prevention, faster detection, and more effective incident response,” OpenAI says.
Read the full article here
