OpenAI audit finds ~30% of SWE-Bench Pro tasks broken, retracts recommendation

OpenAI released a detailed technical analysis on July 8 auditing SWE-Bench Pro, one of the most widely used coding benchmarks for AI models. The audit found that approximately 30% to 34% of the benchmark's tasks contain breaking issues that render evaluations unreliable.

SWE-Bench Pro was designed as an upgrade to SWE-Bench Verified, created in collaboration with Scale AI to test models on longer-horizon, more realistic coding tasks that better track agentic coding capabilities. Over the past eight months, frontier models improved from a 23.3% pass rate to 80.3% on its 731-task public split.

OpenAI's audit employed a two-stage process: an automated data quality pipeline flagged 200 (27.4%) potentially broken tasks, followed by a human annotation campaign where five experienced software engineers independently reviewed the flagged subset, identifying 249 (34.1%) tasks as broken.

The issues fall into four categories: overly strict tests that enforce specific implementation details not specified in the prompt; underspecified prompts that omit requirements hidden tests enforce; low-coverage tests where incomplete fixes can pass; and misleading prompts that point models toward incorrect behavior.

OpenAI attributes the problems to how SWE-Bench Pro tasks are sourced — extracted programmatically from feature change histories in public repositories. Pull request descriptions, merged code, and unit tests originally designed for human collaboration do not always align to form clean evaluation tasks.

Significantly, OpenAI has formally retracted its earlier recommendation for the industry to adopt SWE-Bench Pro, echoing its earlier decision to stop using SWE-Bench Verified. The company now advises model developers to carefully examine benchmark results rather than relying on headline numbers.

The findings carry broad implications for AI evaluation. If benchmark data has systematic flaws, it can misrepresent model capabilities, distort safety cases, and misdirect research priorities. OpenAI calls on the wider evaluation community to develop new benchmarks built specifically to test model capabilities by experienced software engineers.