Anthropic's Fable 5.2 tops a freelance-work benchmark and still fails 58% of tasks

Scale AI's Remote Labor Index, updated on November 12, offers a set of numbers worth sitting with: Anthropic's Fable 5.2 leads the leaderboard with 41.3% of complete freelance projects judged to meet a professional reference standard.

The next-highest model is Opus 5.5 at 21.25%, putting the leader at nearly double its closest rival. For a benchmark that grades finished deliverables rather than single-turn answers, that is a substantial jump. Anthropic reported the result in its November 9 release notes, labeling it a “High effort” score produced with safeguards enabled.

RLI does not ask whether a model can write code or answer a question correctly; it asks whether the model can deliver a complete piece of freelance work on its own. The benchmark was introduced in October 2025 by researchers at the Center for AI Safety and Scale AI and covers 240 projects across 23 categories of freelance work, each with a brief, the source files needed to do the job, and a professional human reference deliverable.

The deliverables span documents, audio, video, 3D models and computer-aided design files, so the test grades two things at once: whether the model understood what the client asked for, and whether the finished artifact is professionally usable.

The judging is strict. Trained human reviewers check the AI's output against both the brief and the reference, with independent evaluations and majority voting deciding the outcome. A project passes only if the deliverable satisfies the brief at least as well as the reference, at a quality a reasonable client would accept. A promising prototype can therefore fall short of the bar.

That is what the 58% figure means: even the strongest model available still misses on roughly six in ten projects. Passing requires a finished deliverable that clears a professional reference bar, which is why a benchmark-topping score coexists with a majority failure rate.

There are real limits to how far the number travels. The original study excludes work that inherently requires client interaction, teamwork or physical labor, along with projects whose outcome must be awaited before it can be evaluated. A pass rate is also not a measure of jobs displaced, cash earned on freelance platforms, or the cost of deployment — the public disclosures do not establish per-project spending or the amount of human assistance involved.

The direction, though, is clear. Agentic systems are getting meaningfully better at real work, and the failure rate is the variable that matters most for whether that improvement translates into commercial value. For vendors selling agentic AI as a productivity promise, the six-in-ten gap deserves more explanation than the first-place ranking.