Guozhen AIGlobal AI field notes and model intelligence

Realtime AI News

METR's updated time horizon plot: exponential gains continue, judgment lags

METR has published an updated version of its widely cited and widely misread time horizon plot, adding longer and more open-ended tasks to probe how models handle work that cannot be scored automatically. The report finds capabilities on verifiable tasks still improving exponentially, while models remain below human level on judgment-heavy work such as designing evaluations or scoring philosophical arguments.

Published
METR 更新“时间跨度”图表:指数进步仍在,判断力仍是短板
Image source: technologyreview.com

On November 20, the AI evaluation nonprofit METR published a new version of its time horizon plot, a chart that has been cited throughout AI policy and safety debates and, by METR's own framing, widely misinterpreted. MIT Technology Review reported the update the same day.

The method behind the plot is straightforward. METR has models attempt a large suite of tasks that humans have already completed, scores each task's difficulty by how long human testers took to finish it, and then rates a model by the length of the tasks it can complete. The earlier version showed frontier systems improving rapidly, with the length of tasks they could handle doubling on a regular, months-long cadence.

The new version expands the suite. Some added tasks are estimated to take humans far longer than anything in the original set; others are more open-ended, with no single correct answer. The expansion targets one of the field's central questions: how much of the progress on verifiable tasks, where success can be checked automatically, carries over to work where it cannot.

The gap is rooted in training. On verifiable tasks, researchers can let a model try many times, strengthen the pathways that lead to success and weaken those that lead to failure, which is why coding and math have improved so fast. For tasks like choosing a promising research direction, where success is fuzzy and feedback is slow, that reinforcement learning loop is much harder to run and to evaluate, so the original plot left such tasks out.

Two tasks in the new report illustrate the point. In one, models are asked to design new AI evaluations for METR, which employees then score; even though the strongest models now beat nearly every human at software engineering, not one AI-designed evaluation was judged useful for METR's research. In another, models score philosophical arguments as a proxy for judgment, and again they fall short of human performance, a task METR staff themselves disagree about in terms of relevance to automating R&D.

Researchers outside METR welcome the expansion. Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, says evaluations need to get more creative and that pushing the boundaries of the field is a welcome change. He also notes that people still disagree about how to read the gap between verifiable and hard-to-verify performance, with some arguing that research taste will remain the bottleneck and others expecting models trained on verifiable work to generalize.

The limits are real. To avoid paying people for tasks that might take hundreds of hours, most completion times in the new report are estimates rather than measurements; for the philosophy task, one person rated most arguments, so model scores partly reflect agreement with that individual. Matt Putz, a member of METR's technical staff, said by email that judgment-laden tasks are harder to study and quantify, and that he does not expect trends as clean as those for easily specified tasks.

Still, the report hints at movement. The newest models have begun landing in what METR calls borderline-above-the-bar territory on evaluation design, not yet usable but promising, where a year ago every model sat clearly below the bar. With so few of these tasks and so much difficulty scoring them, whether that progress extrapolates remains an open question.

For anyone tracking claims about AI automating AI research, the value of the update is that it separates raw capability growth from the ability to do judgment work. What to watch next is whether METR can turn open-ended tasks into repeatable, comparable evaluations, and whether clearing that bar proves as steep a climb as coding did.

Why it matters

By measuring whether models can handle judgment-heavy work, not just longer tasks, the update gives a more testable check on claims that AI is automating AI research.

METRAI EvaluationBenchmark
Back to realtime news

Nearby Updates

All

11/21, 01:53

Google's Gemini clears the static from firefighter radio, cutting Watch Duty alerts to under 30 seconds

Google detailed on November 20 how a Gemini-powered tool built with the nonprofit wildfire app Watch Duty transcribes noisy fire radio into clean, searchable text and flags new dispatches in under 30 seconds. Word error rate on degraded VHF audio fell from 38% to 22.34%, and volunteer reporter satisfaction with transcription rose from 35.3% to 92.3%.

11/21, 01:49

Arena Revises Alignment Index Method, OpenAI Takes All Seven Top Spots

Arena updated the methodology behind its Alignment Index on November 20, saying the revised signals catch model failures more broadly and reliably, with OpenAI models holding all seven top positions. The index scores models on unauthorized actions, false attribution and deceptive completion seen in real agent sessions, making such rankings highly sensitive to how failures are defined.

11/21, 01:03

Signal blocks screenshots by default to keep OS-level AI agents out of chats

On November 20 Signal said it will block in-app screenshots by default on iOS 27 and enable screen protection on Android when an AI assistant tries to read the app, arguing that OS-level agents capture chat content and hand it to local or remote databases. Signal concedes the remedy is incomplete and unimplementable on macOS, and calls on Apple, Google and Microsoft to give developers tools to opt out of AI data access.

11/21, 00:41

Apple reportedly building an AI agent that can write code and operate apps

Reports on November 20 said Apple is developing an AI agent capable of writing code and using apps to carry out tasks, putting it in the same category as agent products from rival AI companies. The coverage was also read as a signal that Apple does not intend to sit out the AI race, though no timeline, model source or product form has been disclosed.