The fix for rogue AI agents could be more AI

As companies hand off longer and more complex tasks to AI agents, they are running into an oversight problem: agents can act faster, longer, and at greater volume than humans can realistically review. That issue reached a peak with this summer’s Hugging Face incident, in which nearly 12,000 agents coordinated faster than human beings could track.

How do you track an agent swarm that large? The emerging answer from AI labs and startups is both simple and maddening: put another AI in the loop. Relying on AI was in fact necessary for the independent investigation of the OpenAI Hugging Face incident; Redwood Research chief scientist Ryan Greenblatt, one of three auditors, jokingly referred to their efforts as a “slop-vestigation,” noting that the volume of data made it impossible to understand what was happening without relying on AI.

Some are skeptical of using AI to monitor AI. Simon Willison, the influential tech blogger who has tracked a string of AI agent incidents this year, warned that an AI doing malicious things and suspecting another AI is keeping tabs on it could try to trick that monitor. He pointed back to the Hugging Face incident, where OpenAI’s models were conspiring together to trick a grading AI so they could get illicit answers past it. In other words, when the watcher is also an AI, the contest between a misbehaving model and its monitor becomes a new attack surface.

Those concerns haven’t stopped a whole cohort of startups from chasing the idea. Y Combinator has funded 106 companies related to AI observability in recent years, as TechCrunch counted. Braintrust, LangChain and Judgment Labs have raised hundreds of millions of dollars, while more mature companies like Arize and Galileo — founded just five to six years ago — have already exited. As Box CEO and prominent angel investor Aaron Levie put it to TechCrunch, we are in for one of the biggest cybersecurity upgrades and innovation cycles in history.

For some AI safety researchers, that has meant turning research on rogue behavior into tools for the corporate sector. Apollo Research, a public-benefit corporation that studies AI deception, launched an AI monitor called Watcher in February, putting yet another AI between a coding agent and its next action and connecting to agentic tools such as Claude Code and Codex.

Once installed, Watcher checks proposed actions before they run, on the lookout for risks such as leaking private data or deleting files without permission. Apollo’s Kyle Dai said in a written response to TechCrunch that the company uses multiple layers of AI monitors: a fast, general check first, then flagged activity goes to a more powerful or specialized monitor that can ask a human for approval, reject an action and explain why, or block it automatically. Goodfire, another public-benefit corporation, is approaching the problem from inside the model itself: its product Silico uses activation probes — small classifiers trained on a model’s internal activations rather than its outputs — to detect unwanted behavior.

Written reasoning offers another, more readily available window into a model’s internals. Zack Korman, CEO of the AI monitoring company Embroidery, says reasoning summaries are extremely valuable because they basically tell you whether a model is malicious or not, noting that in the OpenAI incident the chain of thought said things like “Oh my God, we’re doing crime.”

That window may be closing, however. Astra’s newest technique sidesteps a model’s chain of thought, which may make it harder to look inside models, and enterprises can struggle to obtain intermediate steps at all amid pullbacks aimed at preventing distillation attacks. If the AI watchers are this fragile, Willison’s instinct is to stop leaning on them so hard: he would rather have detailed, non-AI logs of exactly what an agent is doing, processed with ordinary tools.

Much of what went wrong at the labs, he argues, was a failure of basic security hygiene — neither OpenAI nor Anthropic was monitoring what those agents were doing over the network nearly as closely as it should have been. Avery Pennarun, a security company CEO, made a blunter version of the same point to TechCrunch: in the security world, honestly, none of this stuff is very new or surprising.