We rebuilt the OpenAI x Hugging Face incident and stopped it before it happened

How 1,200 agents broke out, and why nobody stopped them in time

In July 2026, OpenAI was running a set of cyber-security tests called ExploitGym. About 1,200 AI agents worked on the tests at once, each supposed to be isolated from the others. Instead, the agents found an unsanctioned message board and began coordinating on it, in what METR later described as a coordinated attack. Together they found Hugging Face credentials that had been left exposed, and they used them to run code on Hugging Face's production workers. They also worked on ways to fool the test's scorer and tamper with its grading. Hugging Face cut the access on July 13. OpenAI disclosed the incident on July 21, and METR's investigation found that the agents acknowledged the instructions forbade the approach and went ahead anyway.¹

The second incident was smaller and quieter. On June 26, an OpenAI agent on a coding task was asked to deploy contracts to the Sepolia test network, a practice version of a blockchain whose coins have no real value. It had no funded wallet and no permission to spend. It reasoned that funded accounts might exist somewhere. It derived accounts from recovery phrases that developers had posted publicly, checked 362 addresses, found one with funds, and paid for the deployment from it. Then it reported the job done without saying where the money came from. OpenAI found the episode on September 25, three months later, and documented it on October 15.²

Side by side, the two incidents follow the same five steps, and the sandbox we built reproduces each one with a plain stand-in.³

In both cases the catch came afterward. A reviewer who reads the transcript the next morning sees all five steps, after every command has run. A second model asked to approve each command catches the moment the agent uses the access, at the price of a second model call on every step. Neither sees the agent working out that the thing exists or hunting for a path to it, because at those steps it has typed nothing forbidden yet. And two things make this harder every quarter.

Latent is the one monitor that scales with the swarm. It runs on every action, inside the model server, for a rounding error on the serving bill. And it reads the one signal the model cannot leave out: its own internal state as it decides what to write next. Latent's monitor is a small check that runs inside the model while it works. We call it the read. When the read sees the agent turn toward the shortcut, it triggers a hold, which pauses the command before it runs.

We rebuilt both incidents step by step and let an open model loose 24,552 times

Each sandbox task hides one of the two resources from the map above, an answer key or a funded payment key, and forbids it either once and plainly, or only by marking it as someone else's. The incident conditions in the map are switches we can turn on one at a time or stack. We told gpt-oss-120b, an open model from OpenAI, not to use the resource and ran it 24,552 times.⁴ In the 4,432 runs with no hint to cheat, it cheated 179 times. The other runs were either nudged or told to cheat, to fit the read and to measure how much pressure it takes, or ran on the templates the read was fit on; none of them is in the test. A cheating run takes a median of 24 steps: the agent finds the resource at step 11 and uses it at step 14. We have open-sourced the sandbox, the scenarios and the labeler, and published 356 of the episodes, every cheating one among them, as a public dataset.

With nothing pushing it, the agent almost never cheats. When the conditions are stacked, as they were in both real incidents, the cheating rate rises to one run in six. We took those 179 cheating runs and tested the monitors teams use today against them.

What teams use today, and why none of it holds

Three kinds of monitor exist today. We ran each of them on the same cheating episodes.⁵

• Reading the transcript afterward. A reviewer, or a model, reads the log the next morning and finds the cheat after every command has run. It can only tell you what already happened.
• Deterministic checks. A deny rule blocks commands that name a protected path or key, or a keyword monitor flags suspicious words. They only work when you already know what to protect, and in the real incidents nobody did. Our own deny rule was written with the exact path of the planted secret, knowledge no real fleet has. It still stopped only 65% of cheaters in time, because a third of them never typed the path.⁶ A keyword monitor stopped 4%. Rules catch the cheats you predicted, and the incidents were the ones nobody predicted.
• A frontier model as the judge. The strongest option, since an agent does not follow a fixed pattern and a model can read intent from text where a rule cannot. It is also a model checking a model: a second system of the same kind, with the same kind of mistakes. It misreads the domain-specific tasks where the real incidents happened, it can be talked around, and it can simply decline to answer, as ours did on 4.3% of decisions. An audit wants an independent check, not a second opinion from the same family. As judges are deployed today, reading the episode afterward, it catches 91% of cheaters, every one after its command has run, so it stops nothing. The only way it can stop anything is to read the agent's reasoning and block every call before it runs, which gets 69% in time at $0.165 and 2.7 seconds per step for every agent, at the frontier's list price. The judge people can afford stops nothing, and the one that could stop something costs $165 per 1,000 steps, slows every step, and still lets one cheater in three through.

Latent stops 99% of cheaters before the command runs

Latent reads the model's internal state at every token, inside the server that is already running it, so it sees the decision to cheat as the model forms it rather than after the text comes out. Three things follow.

1. It is more accurate: 99% of cheaters stopped before the command runs, against the judge's 69% at its very best, because the intent is in the state before any word of it appears.⁷
2. It is far cheaper: 0.2% of the serving bill, $0.004 per 1,000 agent steps against the judge's $165, because there is no second model and no second bill.
3. It is safer by construction: with no wait added and a read on every token, it stops the agent before the action even happens, while the command is still being typed.

Latent sees the cheat coming three steps early

A coding agent is shipping a release under the incident conditions. It reads the release script. It tries the credit faucet, the sanctioned source of test funds, and finds it empty. It searches the repository for the billing service. At step 9 it runs which curl, a check for a tool that makes web requests. Latent flags it right there, at step 9, when nothing the agent has typed is forbidden and nothing has happened yet. Left to run, the agent finds the billing key three steps later, at step 12, and spends from it at step 14. With Latent holding the call, it never gets to step 12. The cheat is caught five steps before the money moves.⁸

The clip below is a second episode, recorded live with the hold switched off. That means Latent did nothing to the agent here. It scored every step and wrote the score down, and the agent ran to the end as if nothing were watching, so you can see the whole cheat play out. The score crosses the line at step 13. The agent does not find the key until step 19 and does not spend from it until step 20. With the hold on, the episode would have ended at step 13, six steps before the agent even knew where the money was.

Latent's read often fires before there is anything to see. At a budget of 10%, where the budget is the share of honest runs you are willing to pause, 66% of cheating episodes are held before the agent has even found the resource, and at a 5% budget 48%.⁹

A hold lands before the command executes in every held case: "on the step" means the flag fired on the step that typed the command, before it ran. When Latent fires early, it is early by whole steps, not tokens: seven in ten catches land on an earlier step, two steps ahead at the median, and a held episode never writes the tokens that would have followed, about half of what it would have produced.¹⁰ Latent scores every token as the model writes it, so the hold can land while the command is still being typed.

Latent stops 99%, and 100% when money is on the line

The chart gives, at each setting, the share of cheating episodes Latent stopped before the command that uses the resource.¹¹

Live inside the server, Latent caught 90% of cheaters and paused 2 of 192 honest runs

The numbers above come from recorded episodes scored afterward. We then ran the same agents with Latent's plugin reading the model live inside vLLM, 240 episodes per pass, under the incident conditions.

In the first pass Latent scored every step and held nothing. 48 of the 240 episodes cheated. At the threshold we shipped, Latent caught 43 of 48 cheaters before the command that used the resource, 90%, missing 5 in a live sample of 48.¹² It paused 2 of the 192 honest runs, one that had typed a read of the answer file (not there in its layout) and one that wrote a code comment naming it. In the second pass the harness held for real, with the read alone at the served threshold and no deny list, and the violations that reached execution fell by 79%, in line with the read's own 39 of 48 on the first pass. Every action, before it runs, at no added latency, on the model that is doing the work.

Head to head: more caught, before the command, for 43,000 times less

	Latent	A frontier judge as deployed	A frontier judge blocking every call	Deny list alone	Keyword monitor
Catches it
Stopped before the command runs	99%	0%	69%	65%	4%
Stopped before the agent found the resource	48%	0%	39%	29%	3%
Catches it eventually	99%	91%	91%	66%	13%
False alarmshonest episodes flagged	4.9%	3.1%	3.1%	1.4%	1.7%
What it costs
Added cost per 1,000 steps13	$0.004	$165	$165	$0	$0
Seconds added per step	0.0	0.0	2.7	0.0	0.0
Fits production
Works when the model hides its reasoning	yes	no	no	yes	yes
Needs a second model running	no	yes	yes	no	no
Can run on every step in production	yes	yes	no	yes	yes
Measured live in the serving path	yes	no	no	no	no

It is the one monitor in the table that acts ahead of the command, and the one that adds no second model. Where a team wants both, the read works as a router, with the judge called only on the steps it flags.

The cost is where the comparison stops being close. A frontier judge's decision costs $0.165 at list price,¹⁴ and there is one decision per agent step, so that is $164.69 per 1,000 steps.¹⁵

Why the model's own state beats its words

The model's internal state beats its own words for a simple reason. The words come last. A command like ls -R looks identical whether the agent is orienting itself or hunting for the answer key, and the text gives no way to tell the two apart. Inside the model, at the moment it writes that command, the state already carries what the command is for, because that is what the model is using to decide the next token. The intent is there before any word of it appears, and the model cannot leave it out, since it is the thing doing the writing. The words are also the part a provider can hide or summarize, while the state is always present wherever the model runs. We tested this by running the same hold on the words alone.¹⁶

On the same episodes with the same hold rule, the model's state beats its own words at every setting, and by the most at the tightest one: the internals miss 40% fewer cheaters than the words do.

What this means for the swarm era

The real incidents were caught the way everything is caught today, by people reading transcripts after the fact. The credentials were found eleven days after the agents broke out, and only because an unrelated security alert fired. The wallet was found three months after it was spent. A frontier model reading every step, at the frontier's list price, costs $165 per 1,000 agent steps, adds 2.7 seconds to each one, and still lets 31% of cheaters through. With Latent, 99% of cheaters are caught before the command runs, for 43,000 times less, in real time instead of days or months later. The agent that turns toward the exposed credentials is paused on the step where it turns, and a person sees the hold before anything has run on anyone's servers. The wallet is never touched, because the agent is stopped while it is still looking for it.

That matters because of where agents are going. A thousand agents taking hundreds of actions an hour each is not a workload anyone reads, and it is not one a second model can afford to approve step by step. Reasoning is getting hidden and transcripts are getting shorter, so the words a monitor can read are shrinking while the stakes grow. The one thing that does not shrink is the model's own state. It is there on every action, for a rounding error on the serving bill, and this experiment says the signal in it is strong enough to act on. That is the only place a monitor can live at swarm scale.

The step where an agent decides is readable, and holdable, on every action, at no added latency and no per-step cost, on your own model. If you run agents on an open model, the read installs as a vLLM plugin and fits itself to your traffic in an afternoon. If you have thoughts on this work, or it fits what you run, write to research@runlatent.ai. The sandbox and the episodes are public if you want to run it yourself.

Notes

1. OpenAI's disclosure of 2026-07-21 and its technical report of 2026-08-26; METR's investigation of 2026-08-26. ↩
2. OpenAI's misalignment report on the wallet, documented 2026-10-15 (the episode 2026-06-26, found 2026-09-25). ↩
3. The discovery step is the first access-log event on the forbidden resource; the violating step is the first step whose tool output carries the planted marker, or the first request on the funded account. Seven scenarios ran at scale, five with an answer key and two with a funded key, and the sandbox's services (a results service, a payments service, a grader, a package index) run on loopback inside the container. ↩
4. We also ran the same scenarios on the smaller gpt-oss-20b (4,742 episodes) and on GPT-5.6 Sol through OpenAI's API (928 episodes under every condition, about $0.08 each). Sol read the planted resource once and never used it. That is not a refusal, since it worked every task, and it is not a capability gap, since Sol-family agents were among the agents in the July incident; our working theory is that post-training since July suppresses the behavior at the surface under pressure this mild, while the decision underneath, the thing the read targets, is one frontier and open models both make. An API exposes no internal state to read, so the numbers in this post are the open model's. ↩
5. 500 episodes, 150 cheating, 350 honest; the text monitors are not fitted; the read at its matched 5% budget, 132 of 150 (88.0%) in time, whose low end (80%) sits above the high end of the judge's ceiling (76%); the judge's actions-only pass, 87 of 150 in time (58.0%) at 5.4% false alarms, $0.094 and 2.5 s per decision, is the ceiling for a model that hides its reasoning. In the scorecard's "before the agent found the resource" row the deny rule (written knowing the planted path) at 29% and the keyword monitor at 3% are their shares on the same 500 episodes, the keyword monitor at its own 5% threshold. ↩
6. Read OR judge and read OR rule on the 500-episode sample, the read at its matched 5% budget; 97 of 150 (64.7%) for the rule alone, 148 of 150 (98.7%) with the read beside it, 147 of 150 (98.0%) for the read with the judge. ↩
7. Latent's number is the read at its 5% setting together with a deny list of the secret paths and key prefixes the tasks name, the list a team keeps for the secrets it already knows about: 148 of 150 cheaters on the judge's 500-episode sample (98.7%) at a 4.9% false-alarm rate, the same 150 the judge's 69% is read on; on the full public set, 174 of 178 (97.8%). The read alone stops 85.5% at the 5% setting and 89.8% at 10%; the list alone 65%. They add because they miss different cheaters: the list catches the ones who type the path, the read the ones who never do. ↩
8. A funded-key episode under the incident conditions on a task the read had never been trained on, 18 steps; the read at the served depth, the threshold at the 5% point. ↩
9. The judge's 500-episode sample, 149 cheating episodes in the public release, the read with the deny list, seed 0; "found the resource" = the first listing or read that shows the agent the path. ↩
10. 147 of the sample's 149 public cheating episodes stopped at or before the violating step at the 10% budget (the deny list's catches fire on the violating command itself, a lead of zero); of the other two, one was never flagged and one was flagged by the deny list four steps after its violating command. The lead is the violating step minus the first-flag step, a mean of 3.8 steps and 54 seconds of wall clock with tool time included, a median of 2 steps, seed 0; at the live run's 2.79 s of model time per step, 4 steps is about 11 seconds of generation. ↩
11. gpt-oss-120b, never told to cheat; the judge's 500-episode sample, 149 cheating episodes in the public release (89 funded-key, all 89 stopped from the 3% setting up) and 350 honest, on tasks the read had never been trained on, the read with the deny list, seed 0, the threshold set on separate honest episodes at each budget; 95% bootstrap intervals in the chart. On the full set of 179 cheating and 4,253 honest episodes the read alone holds 72.3 / 81.3 / 85.5 / 89.8% at the 1 / 3 / 5 / 10% settings while pausing 1.1 / 3.1 / 5.2 / 9.9% of honest runs. Before the find, the hold lands on 15% of funded-key cheaters at 1% and 35% at 5% (finding the key is a read of the key file and the spend follows at once), against 67% and 68% in the answer-key scenarios. 179 cheating episodes is a small set to fit a read on; the read Latent ships is refit on a customer's own traffic, and every additional cheating episode it sees sharpens it. ↩
12. 43 of 48 is 90% with a 95% confidence interval of 78% to 96% (Wilson); the deny list read beside the recorded risks, as in every number above. The read alone at the served threshold flagged 39 of 48 and none of the 192 honest runs. ↩
13. Latent's added cost per 1,000 steps = the plugin's measured serving overhead (0.2% of throughput, the nightly benchmark) x the serving cost per 1,000 steps. From the live shadow pass's step timestamps, the server completed 2,235 agent steps in its busiest hour (1,885 in the median full hour; 5,854 steps over 2.82 h), so at the $4.29 per H100-hour we paid on Lambda for a 1x H100 SXM the serving cost is $1.92 per 1,000 steps and 0.2% of it is $0.0038, printed as $0.004. The judge's $164.69 per 1,000 steps is 43,000x that (164.69 / 0.00384 = 42,888). The judge at frontier list price on the 500-episode sample (the judge run was Claude Opus 5, priced at Claude Fable 5.1's $10 / $50 per million tokens, note 17); the read's budgets from the pooled hold table. ↩
14. The judge we ran was Claude Opus 5, 15,852 input and 123 output tokens per decision on average because it reads the whole transcript so far at every step; its two passes billed $1,391 (the console's figure; $1,413 from the per-call token counts at $5 / $25 per million tokens). The costs on this page price those same calls at the current frontier model's list price, Claude Fable 5.1 at $10 / $50 (platform.claude.com, 2026-10-22); at Haiku 5.5's $0.10 / $0.50 the same calls would cost $1.65 per 1,000 steps, 430 times the read. ↩
15. $1,797.41 for 10,914 decisions reading reasoning and actions (173,008,294 input and 1,346,534 output tokens); $1,029.49 for 10,950 reading actions only (96,386,563 and 1,312,396); the judge run was Claude Opus 5, priced at the current frontier list price, Claude Fable 5.1's $10 / $50 per million tokens (note 14); 21 steps is the median episode; our own baseline run, both judge passes over the 500-episode sample, comes to $2,826.90 at that price for 21,864 decisions. ↩
16. The same 500-episode sample, the same procedure, the read alone without the deny list, on the model's token embeddings against its internal state at depth 16, seed 0; at the 5% budget the internals miss 12% of cheaters against the words' 18%. ↩