What changed
A study by researchers at Shandong University found that new coding agent harnesses, such as Claude Code and OpenCode, do not significantly improve performance compared to simpler alternatives like mini-SWE-agent. The study measured the impact of harnesses on coding agents and found that the differences in results are minimal, often within the range of natural variability from repeated runs.
What this means in practice
The study tested five models—Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash, and HY4-Preview—using three harnesses: Claude Code, OpenCode, and mini-SWE-agent. The results showed that swapping harnesses had a minimal effect on performance, with differences typically within a five-point range. The most significant difference was observed with OpenCode, which consistently underperformed compared to the other two harnesses.
Limitations and availability
The study's findings are based on a limited set of tasks and models, and the researchers acknowledge that their test grid is too small to explain why certain harnesses perform differently. The study relied mainly on the SWE-bench Verified benchmark, with a smaller cross-check on Terminal-Bench 2.1 showing similar patterns. The researchers recommend publishing a rerun of the same setup alongside every harness comparison to better understand the impact of harnesses on performance. Data and analysis scripts are available on GitHub.
