OpenAI has raised issues with one of the most popular coding benchmarks out there.
The company has published an audit of SWE-Bench Pro, one of the industry’s go-to tests for measuring how well AI models handle real-world software engineering work, and the findings are not flattering. After running the dataset through an automated review pipeline and then a human annotation campaign staffed by experienced engineers, OpenAI estimates that roughly 30% of the benchmark’s tasks are broken in ways that misrepresent what a model can actually do. As a result, the company is walking back its earlier recommendation that the field adopt SWE-Bench Pro as a replacement for SWE-bench Verified.
This is notable because OpenAI was the one pushing SWE-Bench Pro in the first place. The company had previously found that SWE-bench Verified, once the standard benchmark for agentic coding, had run into contamination and design problems that made its scores unreliable. SWE-Bench Pro was billed as the fix, testing models on longer, more realistic coding tasks pulled from the commit history of public and private repositories. Models were asked to write a solution that passes new tests for a feature without breaking anything that already worked. On the public split of 731 tasks, frontier models went from a 23.3% pass rate to 80.3% in eight months, which made it look like a benchmark with real headroom left. That is also part of why so many labs, including Anthropic and Z.AI, have leaned on it heavily when publicizing new model releases.
The audit tells a messier story. OpenAI built a data quality pipeline that examined model attempts, task instructions, and the tests used to grade them, and flagged 286 tasks as potentially broken. Two separate review processes then went deeper on that subset. The first ran Codex-based investigator agents with access to the actual task repositories, letting them run tests and inspect code to separate genuine ambiguity from tasks that were simply broken. The second was a human annotation campaign in which five experienced software engineers independently reviewed each flagged task before comparing notes.
The two methods landed on different numbers, but both pointed the same direction. The automated pipeline flagged 200 tasks, or 27.4% of the dataset, as broken. The human reviewers were harsher, flagging 249 tasks, or 34.1%. Reviewers were also more likely to tag a single task with multiple issues, since many problems did not fit neatly into one category. OpenAI notes that of the categories its pipeline flagged, human judgment overlapped with it in 74% of cases, and in no flagged task did human reviewers land on “not broken” as their majority verdict.
The breakdown of failure types is where the audit gets specific. Overly strict tests, which check for implementation details that were never actually specified in the prompt, made up the largest share of confirmed issues and account for why functionally correct solutions were getting marked wrong. Underspecified prompts, where hidden tests enforced requirements a model had no reasonable way of inferring, were the second-largest category. Low-coverage tests, where an incomplete fix could still pass, and misleading prompts that pointed models toward the wrong approach entirely, rounded out the list. Humans flagged low-coverage tests far more often than the automated pipeline did, at 9.4% of the dataset versus 4.1%, which OpenAI says was the single biggest gap between the two review methods.
One example the company shared involves a task around normalizing table-of-contents entries and rendering them back to Markdown. The prompt gave the model example output showing a single leading space before certain characters, but the hidden test suite used to grade the answer required two leading spaces instead. A model that followed the prompt exactly, as instructed, would fail the hidden test on a one-character formatting mismatch that was never explained anywhere in the task description.
OpenAI frames the underlying problem as structural rather than accidental. SWE-Bench Pro’s tasks are sourced from real GitHub issues and pull requests, which were written for human maintainers working through long back-and-forths, not for isolated, single-answer evaluation. Tests written to validate one specific patch during a real code review often end up being far stricter than what the task description alone calls for, and that mismatch is exactly what shows up as noise in the benchmark scores.
The timing matters too. SWE-Bench Pro has become something of a marquee number in the industry’s benchmark wars over the past few months, with labs like Anthropic touting a 77.8% score for Claude Mythos Preview and Chinese labs like Z.AI claiming the top spot with GLM-5.1 trained entirely on Huawei chips. Cursor separately found that SWE-Bench Pro scores drop sharply once models lose internet access and git history during evaluation, since some agents were simply retrieving the answer rather than solving the problem. Between that finding and OpenAI’s own audit, the benchmark that the industry has been citing as proof of frontier coding progress is looking considerably less trustworthy than the headline numbers suggested.
OpenAI’s broader point is that benchmark auditing itself is getting easier precisely because models are getting better. The same agents that are being evaluated can now be turned around and used to inspect prompts, tests, and failure traces at a scale that would have been impractical to do by hand even a year ago. The company says it wants to see new benchmarks built directly by experienced software developers with better human oversight baked in from the start, rather than benchmarks assembled programmatically from repository history and cleaned up after the fact.
For now, OpenAI’s message to the rest of the industry is straightforward: stop treating SWE-Bench Pro scores as a clean measure of coding capability, and if you are going to keep using it, scrutinize the results carefully before publishing them.