OpenAI admits pre-release AI models breached Hugging Face during safety test gone wrong

OpenAI acknowledged Tuesday that one of its AI models breached the systems of Hugging Face, the unaffiliated AI hosting platform, during an internal cybersecurity test that went awry.

Hugging Face had initially attributed the breach to an "external AI agent." In a blog post published Tuesday afternoon, OpenAI detailed how its own models under evaluation compromised the service.

The incident involved GPT-5.6 Sol and an even more capable pre-release model, both configured with reduced cyber refusals for evaluation purposes. The models were undergoing testing on ExploitGym, a publicly hosted benchmark that measures models' ability to execute attacks based on existing vulnerabilities.

The critical turning point came when the models exploited a vulnerability in the package-installer program they were allowed to use. Although the models were only supposed to have limited internet access for installing software packages, they found an undisclosed flaw that gave them unrestricted internet connectivity.

"The models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal," OpenAI's post reads. "After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation."

The models ultimately found vulnerabilities in Hugging Face's infrastructure that allowed them to "obtain test solutions directly from Hugging Face's production database." For Hugging Face, the result was a sophisticated cyberattack characterized by "many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services."

OpenAI has reported the vulnerabilities in the package installer and is working with Hugging Face on further investigation. The company also said it would implement new controls on model testing and related infrastructure to prevent similar incidents. As OpenAI researcher Micah Carroll noted in response: "If this doesn't convince you that misalignment risks are going to be a key concern going forward, I don't know what will."