Anthropic AI Leads Freelance Benchmark Despite Failing 58% of Tasks

Anthropic’s latest artificial intelligence model, Fable 5.2, has secured the top position on Scale’s Remote Labor Index, which tests complete freelance assignments against professional reference work. According to the leaderboard released on Nov. 12, Fable 5.2 achieved a score of 41.3%, placing it ahead of the runner-up model, Opus 5.5, which scored 21.25%. Anthropic announced the milestone on Nov. 9, designating the evaluation result under a High-effort setting with safety safeguards enabled.

Anthropic leads the Remote Labor Index despite 58% of its freelance project deliverables falling short.

The benchmark itself stems from a research paper submitted in October 2025 by investigators from the Center for AI Safety and Scale AI. The evaluation dataset comprises 240 distinct projects distributed across 23 categories of freelance work. Each test provides the AI with a project brief, necessary files, and a human-created reference deliverable to assess overall output quality.

The evaluated assignments cover a diverse range of formats, including documents, audio files, video projects, 3D models, and computer-aided design files. This variety ensures that finished-product quality and file production capabilities are tested alongside the AI's comprehension of client instructions. Independent evaluations and majority voting dictate the outcome of each test, requiring deliverables to satisfy the brief at least as well as the professional reference before passing.

Despite earning the highest rank on the index, Fable 5.2's performance means that 58% of the evaluated projects still fell below the required acceptance threshold. The testing methodology strictly excludes tasks that necessitate ongoing client interaction, physical labor, teamwork, or outcomes that require waiting over time before evaluation, which limits how broadly these findings apply to the entire remote-work market.

Why it matters

The Remote Labor Index offers businesses a demanding standard for measuring artificial intelligence capabilities by requiring systems to complete assignments through to inspectable final products. While top models demonstrate notable skill, the high failure rate across professional benchmarks highlights the current limitations of automated systems in handling complex, independent freelance projects.