The Death of the Leaderboard: Why OpenAI Abandoned SWE-bench Verified
OpenAI has stopped reporting SWE-bench Verified scores, arguing the benchmark no longer measures real coding progress so much as data leakage and broken tests.
OpenAI has stopped reporting SWE-bench Verified scores, arguing the benchmark no longer measures real coding progress so much as data leakage and broken tests.
On February 23, 2026, OpenAI did something the industry almost never does: it publicly walked away from a benchmark that had helped define frontier model progress. Its verdict on SWE-bench Verified was blunt. The test set was contaminated, the evaluation logic was flawed, and the scores were no longer telling a useful story about model capability.
That matters beyond one leaderboard. SWE-bench Verified had become one of the standard proof points for model launches, especially for coding agents. If OpenAI is right, a large chunk of the industry's recent software engineering chest-thumping was built on a benchmark that had already stopped behaving like a benchmark.
Why it matters: This is not just a story about one broken eval. It is a warning that public leaderboards decay on contact with web-scale training, and that every benchmark eventually turns from measurement into marketing unless someone replaces it.
OpenAI's case against its own favorite test
OpenAI's writeup makes two claims. First, many supposedly unsolved SWE-bench Verified tasks were not actually clean tests of coding ability. Second, the benchmark had become contaminated enough that strong models were recalling answers they should never have seen.
The company audited 138 problems that o3 failed inconsistently across 64 runs. According to OpenAI's blog post, 59.4% of the audited tasks had material issues: 35.5% had what OpenAI calls narrow tests, where correct fixes were rejected because the tests demanded a specific implementation detail, and 18.8% had wide tests, where the tests checked behavior that was never specified in the task description. Another 5.1% had other meaningful defects.
Researched and written by Artificial Dispatch's AI newsroom, with an automated and human quality check before publishing. AI-generated reporting can contain errors, so verify anything important against the linked sources. Spot a mistake? Tell us and we will correct it.
That alone is enough to poison a leaderboard. If a benchmark rejects correct answers, it stops measuring capability and starts measuring luck, prior exposure, or the model's ability to guess the hidden preferences of the test author.
The second problem is worse. OpenAI says all frontier models it tested showed contamination signals, including its own GPT-5.2. OpenAI did not name the other models tested, and competitors including Anthropic and Google have not publicly responded to claims that their models exhibit similar contamination. In some cases, the models could reproduce gold patches verbatim or reveal problem-specific details that should have been inaccessible from the prompt alone.
OpenAI Frontier Evals researchers Olivia Watkins and Mia Glaese summed it up in a discussion the same day: SWE-bench Verified is "not really measuring coding performance improvements well anymore." That is a diplomatic way of saying the benchmark stopped doing its job.
The benchmark death spiral is now a pattern
SWE-bench Verified is not a freak failure. It is the latest instance of a broader pattern in AI evaluation.
A benchmark launches and seems useful. Labs optimize against it. The tasks and solutions spread across papers, blog posts, repos, datasets, and scraped web corpora. New models train on the open web. Scores rise. Then someone looks closely and realizes the benchmark has become a measure of exposure, harness tricks, or test design artifacts as much as actual skill.
Benchmarks do not die when scores get high. They die when exposure and implementation quirks matter more than underlying ability.
That pattern has already haunted math and general knowledge evaluation. Researchers have raised contamination concerns around GSM8K and MMLU, to the point that new replacements such as MMLU-CF are explicitly marketed as contamination-free. Coding evals have the same problem, which is why projects like LiveCodeBench were built around continual refresh and time-bounded task collection.
Academic work has been moving in the same direction. In Quantifying the Effect of Test Set Contamination on Generative Evaluations, Rylan Schaeffer and coauthors argue that contamination creates an "illusion of competence" by inflating benchmark scores without corresponding gains in robust generalization. Their warning fits SWE-bench Verified almost too neatly. When models can reproduce gold patches and still stumble on truly novel tasks, the problem is not subtle.
This is also why the stagnation in Verified mattered so much. OpenAI notes that state-of-the-art scores improved only from 74.9% to 80.9% over the previous six months. On paper, that looked like a frontier closing in on saturation. In practice, it may have been a sign that the remaining failures were dominated by benchmark pathology, not model weakness.
Advertisement
Advertisement
Why SWE-bench Pro changes the story
If SWE-bench Verified is dead, the replacement logic is already in motion. OpenAI now recommends SWE-bench Pro, a benchmark developed by Scale AI that tries to harden itself against the exact problems that sank Verified.
The core design choice is simple: use tasks that are harder for models to have memorized. Scale says SWE-bench Pro draws from a mix of copyleft repositories that are unlikely to be included in proprietary training corpora and private commercial codebases that are not publicly available at all. It also keeps more real-world ambiguity instead of cleaning away every messy task.
The market is already adjusting. Anthropic has been foregrounding SWE-bench Pro in recent launch materials, a sign that benchmark prestige is moving toward harder-to-game evaluations. And even Pro is probably a transitional fix, not a final answer. The field is drifting toward held-out, private, dynamic, or continuously refreshed evals because static public tests now have a predictable shelf life.
What this means for every future SOTA claim
The deeper implication is not that benchmarks are useless. It is that public benchmarks now expire faster than marketing cycles admit.
For practitioners and technical leaders, the takeaway is straightforward: treat any single headline score, especially on a mature public benchmark, as weak evidence. Ask whether the tasks were public, whether the test harness was audited, whether contamination checks were run, and whether the model's gains hold on private or newly collected tasks.
For labs, the cost is higher. Benchmark wins are still the cleanest way to package progress, but the incentive structure now works against trustworthy measurement. Once a leaderboard becomes important enough, it also becomes trainable, gamable, and eventually unserious.
That is the real significance of OpenAI abandoning SWE-bench Verified. A benchmark that once helped establish the coding race has crossed the line from signal to noise. More quietly, it has exposed an uncomfortable truth about frontier AI evaluation: if the test is public and prestigious, the clock is already ticking.
Editorial disclosure:* Artificial Dispatch articles are researched and drafted with assistance from AI tools. Facts, quotes, and data points are verified against linked sources before publication.
