OpenAI announced that its GPT-5.6 Sol model achieved a 38.3% score on the ARC-AGI-3 benchmark using two specific API settings, outperforming Anthropic's Opus 5, which scored 30.2%. The results were obtained through OpenAI's custom Responses API with 'Retained Reasoning' and 'Compaction' features. These settings allow the model to maintain its chain of thought between steps and summarize old context rather than truncating it. OpenAI argues that benchmarks should consider both the model and the technical setup around it. | Source: thedecoder
ARC Prize co-founder François Chollet responded to OpenAI's results, distinguishing between two kinds of test setups. He noted that harnesses 'custom-made to solve the benchmark or that contain knowledge about the benchmark format' are off-limits, while general-purpose API settings 'that were not developed for ARC-AGI-3 and that are available to all API users' are fair game. Chollet acknowledged that ARC Prize's own GPT-5.6 Sol score put OpenAI at a disadvantage, highlighting the ongoing discussions about how to best test models, especially with regard to compaction. | Source: thedecoder
In the official test harness, GPT-5.6 Sol scored just 7.8% because the model's reasoning gets discarded after each action. ARC Prize emphasized that official ARC scores use a standardized approach without provider-specific settings to ensure fair comparisons. The disagreement centers on whether ARC Prize used an older 'OpenAI-style completions API' that lacked features the Claude API already offered, which would make the comparison unfair to OpenAI. | Source: thedecoder
