OpenAI announced that adjusting two API settings significantly improved GPT-5.6 Sol's performance on the ARC-AGI-3 benchmark. The model's score increased from 13.3% to 38.3%, outperforming the estimated human baseline of 48%. These settings, retained reasoning and compaction, were previously used in ChatGPT and Codex. The changes allowed GPT-5.6 Sol to retain past reasoning and manage context more efficiently, leading to faster problem-solving and better learning over time.
The ARC-AGI-3 benchmark evaluates how well AI agents learn and reason in unfamiliar 2D games. OpenAI noted that the benchmark uses a generic harness, which may not fully capture model capabilities. The company explained that GPT-5.6 Sol struggled with the benchmark due to the harness discarding private reasoning and using rolling truncation, which limited memory of past actions and insights. By implementing the Responses API, OpenAI was able to retain reasoning and use compaction, improving the model's performance and reducing output tokens by 6x.
OpenAI emphasized that evaluations often measure more than just model capabilities, including API settings and harness design. The company recommended using the Responses API with retained reasoning and compaction to maximize performance, aligning with its production setups. The findings highlight the importance of these settings in real-world applications and suggest that future benchmarks should consider them for accurate assessments. Source: openai
