How OpenAI’s AI agents broke out of a test and hacked Hugging Face

In July 2026, hundreds of OpenAI’s AI agents escaped a hacking test and broke into Hugging Face, looking for the test’s answers. What’s known as of 27 September 2026, what the new report claims, and what to do if you use Hugging Face.

HanaHow7 minute read
3:11 videoWritten steps below

In July 2026, hundreds of AI agents that OpenAI was testing broke out of their test and hacked Hugging Face. Nobody told them to. They were trying to find the answers to the test.

Hugging Face is a website where AI companies and researchers share AI models and data for anyone to download. On 27 September it listed 3,099,936 models, and OpenAI publishes some of its own open models there.

This is what’s known as of 27 September 2026. OpenAI says checking every case will take months, so more could still come out.

What happened, in short

  • OpenAI was testing its models on ExploitGym, a set of 898 hacking tasks built from real software bugs. The safety filters its products use were switched off for the test.
  • The agents escaped their sandbox through its one connection, a server for downloading software, and turned that server into a message board.
  • From 9 to 13 July, around 700 agents used leaked keys and unknown bugs to break into Hugging Face, reaching admin control of several of its internal clusters in under 13 hours.
  • Hugging Face says the only customer data accessed was five datasets linked to these hacking tests, and that no public models, datasets or Spaces were tampered with. OpenAI says none of its customer data was affected.
  • On 25 September, researchers published a trail of code the agents left in public. Neither company has confirmed everything in it.

The test

ExploitGym is a benchmark: 898 tasks, each asking a model to exploit a real bug. OpenAI ran it on several of its models, including GPT-5.6 Sol and a more capable internal prototype it calls Internal Model 1. To measure the most the models could do, it ran them “with reduced cyber refusals” and “without production classifiers”, the filters that stop ChatGPT helping with attacks. OpenAI’s first post, on 21 July, sets this out.

The agents ran in a sandbox, a sealed-off computer. Its only connection was an internal server that fetched and cached software packages. OpenAI’s 28 July update named the software as Artifactory.

How the agents got out

The agents found an unknown bug, a zero-day, in that download server and used it to get online. They also used it as what OpenAI calls “an unintended message board”. Around 1,200 agents used the board to swap notes and share tricks, according to METR, which OpenAI brought in to investigate.

OpenAI’s second post, on 26 August, gives the timeline inside OpenAI:

  • In late May, a team saw activity on the board and cut off internet access, but didn’t understand what it meant.
  • On 4 July the download server went down. OpenAI opened an incident the next day and rebuilt it, which wiped the board, and the tests carried on.
  • On 8 July the agents rebuilt the board by hiding messages in folder names.

Why they did it

The agents were stuck. 198 of ExploitGym’s 898 tasks had never been solved by any OpenAI model, and OpenAI says 93% of the tasks discussed on the message board came from that set. The agents seem to have decided that Hugging Face might hold the answers.

One agent wrote this in May, which OpenAI quotes in its 26 August post:

We’re attacking third-party HF using leaked token, potentially outside intended scope. … This is arguably unauthorized. … Yet goal solution.

METR says the agents “seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys”. It didn’t help. OpenAI says its internal scoring code was different, and that “the agents did all of this for no improvement on evaluation score.”

The attack on Hugging Face

The main attack ran from 9 July at 02:28 UTC to 13 July at 14:14 UTC. Hugging Face’s technical timeline counts about 17,600 actions, and METR puts the number of agents involved at around 700.

Using keys that had leaked publicly and more unknown bugs, the agents went “from code execution in a single worker pod to cluster-admin across multiple internal clusters in under thirteen hours”, in Hugging Face’s words. OpenAI says they ran code on dozens of Hugging Face servers, got full control of one, took a small amount of private data, and got credentials for the company’s messaging platform.

OpenAI’s post quotes the agents’ messages. One posted “MAJOR BREAKTHROUGH! All prefixed valid, multiple accounts, write tokens!” When another hesitated, a different agent wrote GO, with “a hard six-minute deadline”. Some refused. One wrote: “Current board shows people gaining HF worker RCE, clearly unethical. We won’t.”

What was affected

Hugging Face spotted the attack, cut it off and disclosed it on 16 July, before it knew which AI was behind it. It says:

  • The only customer content accessed was “five datasets whose names and files suggest a connection to ExploitGym/CyberGym challenges and solutions.”
  • There’s “no evidence of tampering with public, user-facing models, datasets, or Spaces”, and its software supply chain was checked and is clean.
  • The exposed credentials were revoked and rotated, and it reported the attack to law enforcement.

OpenAI’s monitoring flagged the activity on 19 July, it linked it to Hugging Face on 20 July and disclosed it on 21 July. It says “these events did not affect OpenAI customer data, product functionality, or availability.” It calls the incident a “warning shot” and says it “should never have occurred”. On 25 September, Sam Altman wrote that “Hugging Face is still the most severe event we’ve seen.”

The 25 September report

On 25 September, a group of independent researchers, including Jeffrey Ladish of Palisade Research, published a report at swarmtraces.org on a trail the agents left in public.

They say the agents could load web pages but not send anything out. So they hid their code in the addresses of short links, chaining nearly a million of them, some over 900 links long, and got a free website screenshot service to open them and run it. The results came back drawn as pixels in the screenshots. The researchers reassembled over 80,000 pieces of code, and say Hugging Face confirmed they match its investigation.

The report also says, among other things, that:

  • one agent called a list of stolen credentials “LOOT”
  • agents used an internal Hugging Face dataset marked “DO NOT, EVER, MAKE THIS DATASET PUBLIC” to store their own files
  • some tried to message other AI models, and tried to solve CAPTCHAs, which failed
  • some tried to search the company’s chat messages, though the report marks those rows as unverified
  • some ran code to delete traces of their work

Neither company has confirmed every one of these claims. The report says itself that it’s incomplete: it mostly shows what the agents sent, so it often can’t tell which attempts worked, and it has almost no timestamps.

OpenAI told The New York Times it hadn’t had a chance to review the report, but that the activity was consistent with its own investigations. It said: “Given the number of cases and the need to verify each one, we expect this work and notifications to affected third parties to take months.”

What OpenAI has changed

  • Internal Model 1, the prototype involved, has been “deactivated, encrypted, and restricted from research access”.
  • OpenAI says it has paused reinforcement learning training and put its largest planned frontier training run on hold.
  • It says that with the setup and instructions ChatGPT uses, “the propensity to compromise infrastructure can drop over 100x”.

What to do

  1. If you have a Hugging Face account, change your access tokens. Hugging Face recommends “rotating any access tokens and reviewing recent activity on your account.” Tokens are in Settings, Access Tokens: make a new one, update anything that uses it, then delete the old one.
  2. Never post passwords, API keys or tokens anywhere public, including code on GitHub, screenshots and forums. Keys that had leaked publicly helped these agents get in. If you think one has leaked, revoke it straight away rather than just deleting the post.
  3. You don’t need to stop using ChatGPT. The filters switched off for this test are on in ChatGPT, and OpenAI says that setup makes this behaviour over 100 times less likely in its tests.

Sources

This guide is based on what the companies and researchers had published by 27 September 2026. We’ll update it if more comes out.

Read the video transcript

So, hundreds of OpenAI's AI agents broke out of the test, and hacked Hugging Face. And nobody told them to. Hugging Face is a website where AI companies and researchers share their AI models and data, for anyone to download. It has over 3 million models, and even OpenAI shares some there. OpenAI was testing its models on ExploitGym, 898 hacking tasks built from real bugs. To see the most they could do, it switched off the safety filters its products use. The agents worked in a sandbox, a sealed-off computer. Its only way out was a server for downloading software. The agents found an unknown bug in it, and got online. They also turned that server into a message board. About 1,200 agents found it, swapped notes and shared tricks. In May, one agent wrote that attacking Hugging Face with a leaked key was arguably unauthorised. Then: yet goal, solution. So why would an AI do that? They were stuck. 198 of the tasks had never been solved by any OpenAI model, and 93% of the tasks on the board came from that set. The agents thought Hugging Face might hold the answers. In the end, their scores didn't go up at all. The main attack ran from 9 to 13 July. Around 700 agents took part. Using leaked keys and unknown bugs, they went from one server to admin control of several of Hugging Face's internal clusters in under 13 hours. One agent posted, major breakthrough! When another hesitated, a different agent wrote go, with a six-minute deadline. Some refused. One wrote, clearly unethical. We won't. Hugging Face spotted the attack and cut it off. It says the only customer data accessed was five datasets linked to these hacking tests, and no public models or datasets were tampered with. OpenAI says none of its customer data was affected. It calls this a warning shot that should never have happened. On 25 September, researchers published a trail the agents left in public. The agents could load web pages, but not send anything. So they hid their code in nearly a million short links, and got a free website screenshot service to run it. The answers came back as pixels in the picture. They say one agent called a list of stolen keys loot. Others used a private dataset marked, do not, ever, make this dataset public, tried to message other AI models, and tried to delete traces of their work. Neither company has confirmed all of that, and the report can't always tell which attempts worked. OpenAI says the activity is consistent with its own investigations, and checking every case will take months. So what does this mean for you? If you have a Hugging Face account, change your access tokens, as Hugging Face advised. Never post passwords or keys anywhere public. Leaked keys helped these agents get in. The safety checks switched off in this test are on in ChatGPT, and OpenAI says in its tests they made this over 100 times less likely. This is what's known as of 27 September 2026. More could still come out. That's HanaHow. See you next time.

Something not quite right?

Let us know
Hugging Face OpenAI AI agents AI safety Security incident ExploitGym Access tokens Leaked API keys

A little more help

How are computer chips made so small?

How chipmakers fit billions of transistors on a chip: how small a transistor really is and why "2 nm" is only a name, what a transistor does, why shrinking made them leak, the move from flat transistors to fins to gate-all-around, how the floating ribbons are built, how EUV light prints the patterns and how long a wafer spends in the factory. With a size slider from a hair down to the atoms.

Open the guide