OpenAI conducted an audit of SWE-Bench Pro, a coding benchmark, and found that approximately 30% of its tasks are flawed. The findings highlight the challenges of creating reliable benchmarks for evaluating AI models' coding capabilities. The audit revealed that many tasks either enforce unnecessary implementation details or lack clear requirements, leading to incorrect assessments of model performance. These issues can mislead developers and affect research priorities. The audit also showed that human reviewers were more likely to identify broken tasks than automated systems, indicating a need for more rigorous benchmarking processes. The results underscore the importance of ensuring that evaluation tasks accurately reflect model capabilities and do not introduce biases or errors. Source: openai

OpenAI's audit of SWE-Bench Pro revealed that 27.4% of tasks were flagged as broken by an automated pipeline, while human reviewers identified 34.1% of tasks as flawed. The issues primarily fell into four categories: overly strict tests that enforce specific implementation details not mentioned in the prompt, underspecified prompts that omit necessary requirements, low-coverage tests that do not fully check the requested feature, and misleading prompts that direct models toward incorrect behavior. These flaws can result in models passing tasks that do not meet the actual requirements, thereby providing a false sense of their capabilities. The audit also found that human reviewers were more likely to assign multiple labels to tasks, suggesting that some issues were not captured by the automated system. Source: openai

In the audit, OpenAI used a combination of automated and human review to assess the quality of SWE-Bench Pro tasks. The initial automated filter flagged 286 potentially broken tasks, which were then reviewed by both human-supervised agent reviews and a human annotation campaign involving experienced software engineers. The human reviewers were more likely to identify broken tasks and were also more likely to assign multiple labels, indicating that some issues were not captured by the automated system. The findings suggest that while the automated pipeline captured the main failure modes, it undercounted cases where human reviewers identified additional or overlapping issues. This highlights the need for more comprehensive and human-informed benchmarking processes. Source: openai