SemiAnalysis Ships a Real-World Work Benchmark, and No AI Agent Tops 12 Percent

SemiAnalysis has released a new agent benchmark built around real-world work tasks, and no AI agent scored above 12 percent on it, according to a report from the-decoder.com published on November 5. The headline finding is unusually blunt.

The benchmark's orientation differs from exam-style evaluation. Rather than decomposing work into discrete questions, it hands an agent a complete task and asks whether the final deliverable is acceptable. Scores therefore sit closer to 'can this cover a day of someone's job' than to 'did it get a sub-task right.'

What the 12 percent ceiling actually says is that in end-to-end work, very few agents deliver completely, and partial completion is not delivery. Getting halfway through a workflow and leaving a mess for a human to finish is not a completed task in most real operations.

That result sits at odds with current product narratives. In demos, agents call tools in sequence, move across systems and produce polished outputs; move them to real tasks that have not been pre-trimmed, and every ambiguous instruction and failed API call compounds into error.

A single score should not be extrapolated into 'agents are useless,' however. The report does not lay out task counts, grading criteria, how much human intervention was allowed, or whether runs were single-attempt, and those settings often move the number more than the model does.

Reproducibility is the thing to watch. Whether SemiAnalysis publishes the task set, the grading scripts and failure cases decides whether this becomes an industry yardstick or another number that can be cited but not checked. For buyers, a benchmark they can re-run themselves is the only one with procurement value.