OpenAI has withdrawn its support for the SWE-Bench Pro coding test after a review identified roughly 30 percent of its tasks as flawed. The assessment found that many tasks were derived from real software projects, making them too strict, too vague, or misleading for AI models. This undermines the accuracy of evaluating AI capabilities. The issues highlight the need for more reliable benchmarks in assessing AI programming skills.
To evaluate the test, OpenAI first used an automated screening tool that flagged 286 suspicious tasks. AI agents based on Codex then analyzed these cases, with a human researcher making the final decision. This process identified 200 tasks (27.4 percent) as flawed. In a separate review, five software developers flagged 249 tasks (34.1 percent) as problematic. Both groups agreed on 74 percent of the cases, showing some alignment but also differing levels of strictness.
The tasks were pulled from commit histories of real software projects, originally intended for human collaboration rather than AI evaluation. According to OpenAI, these tests tend to be too strict because they were designed to verify specific changes, not to serve as general-purpose requirements. On the public version of the test with 731 tasks, top models saw their accuracy jump from 23.3 to 80.3 percent in eight months. SWE-Bench Pro was meant to replace the older SWE-bench Verified, which OpenAI had already dismissed for similar reasons.
Source: thedecoder
