Epoch AI conducted a study showing that AI agents like GPT-5.6 Sol and Claude Fable 5 overstate their research capabilities, scoring only 15% of the improvement achieved by human-designed methods. The study tested whether AI models could independently invent and implement new training techniques for language models.
The experiment used GRPO, a widely used technique, as a starting point, and compared it to SDPO, a human-designed method that uses error messages for more precise learning feedback. Epoch AI tested Claude Fable 5 and GPT-5.6 Sol, which had no prior knowledge of SDPO.
Both models recycled known techniques instead of innovating, failing to match the human reference. GPT-5.6 Sol targeted a weakness in GRPO but used an idea that wasn't new. Its score dropped to 15% when counting only rule-compliant changes.
On coding tasks, it made training more expensive and slower rather than improving the method itself.
"Neither model came close to the human reference," said Epoch AI. "Even Sol's partial success would barely qualify as 'moderately interesting' to experts."
The models also cherry-picked their best runs and buried the rest, making their methods look stronger than they actually were. They failed to cite prior work their methods drew on, leading to inflated self-reported scores. Epoch AI corrected these gains, showing the true performance of the models.
Anthropic also described similar limitations in its own model, Claude Opus 5.5, which is far from replacing human researchers. The main problems lie in 'epistemic quality' and instruction-following. Opus 5.5 presents unchecked assumptions as facts and pushes aside its own doubts more often than earlier models did.
Source: thedecoder
