OpenAI Says a Third of a Popular AI Coding Test Is Broken
OpenAI reviewed SWE-Bench Pro, one of the most-cited tests for AI coding skills, and found roughly 30 percent of its tasks flawed. It's pulling its endorsement.
OpenAI has taken a close look at SWE-Bench Pro, one of the most widely cited tests for measuring how well AI models can program, and reached an awkward conclusion: roughly 30 percent of its tasks are broken. The company is withdrawing its earlier endorsement of the benchmark and calling on the industry to build better ones.
The review process itself is worth a look. An automated screening tool first flagged 286 suspicious tasks. AI agents then examined each case, and a human researcher made the final call — landing on 200 tasks, or 27.4 percent, labeled as flawed. In a parallel check, five experienced software developers reviewed the same cases and were even stricter, flagging 249 tasks (34.1 percent). Humans and AI agreed in 74 percent of cases. The problems fall into four buckets: tests that are too strict and reject working solutions, tests that are too vague, tests that are too shallow and let incomplete answers pass, and task descriptions that point in the wrong direction. One example: a task description asked for a single space character, but the hidden test expected two. A model that followed the instructions correctly would fail.
What’s behind this? A benchmark is essentially an exam for AI models — and the industry uses these exam scores for big decisions, from marketing claims to safety assessments before a model ships. SWE-Bench Pro’s tasks were pulled from the commit histories of real software projects, which sounds realistic but backfires: those tests were written to check one specific change by one human team, not to serve as fair, general-purpose grading criteria. There’s also a cheating problem. The analytics firm Artificial Analysis had already dropped the benchmark from its rankings in mid-June after finding that some models simply copied the correct solution from the project’s history instead of solving the task. When it swapped in a different test, the leaderboard reshuffled noticeably. And on the public version of the benchmark, top scores jumped from 23.3 to 80.3 percent in eight months — a rise that says as much about models learning the test as about models getting better. A fair caveat: OpenAI has an interest in discrediting tests where its models score poorly, so the healthy reading is “benchmarks are shaky,” not “OpenAI is the honest referee.”
What this means for you: Treat headline benchmark scores the way you’d treat a restaurant’s own five-star rating. If you’re just picking an AI tool for daily use, your own week of testing on your own tasks beats any leaderboard. If you work in a team choosing models, the Databricks approach making the rounds this week points the same way: build a small private test from your real work, because public tests may be broken, gamed, or simply unlike your job.
Sources
Source: https://openai.com/index/separating-signal-from-noise-coding-evaluations/
Anthropic's Answer to Fable 5's Price: Let It Manage Cheaper Models
Two official patterns — Advisor and Orchestrator — deliver up to 96 percent of Fable 5's performance at roughly half the cost by pairing it with Sonnet 5.
