What changed
Researchers from Alibaba and several Asian universities developed a new benchmark called Co-Design Bench to evaluate AI coding agents. This benchmark is based on real user requests and includes tasks that previously challenged AI agents. The best model tested, GPT-6 Astra, meets about 70% of a task's requirements on average but fully completes only 26.4% of tasks correctly.
What this means in practice
The new benchmark focuses on tasks that require multiple changes, such as updating dependent totals from receipts or overhauling entire projects. AI agents often handle most of the work but leave at least one detail unfinished, necessitating manual cleanup by users. The strict grading criteria ensure that only fully functional features are counted as correct. The study also found that performance drops as tasks involve more types of changes, which are common in real projects.
Limitations and availability
The benchmark evaluates one fixed request per task and does not include follow-up questions or mid-task corrections. AI judges can make mistakes, and there is a gap between benchmark results and real-world usability. The study also notes that the working environment significantly affects results, and models can perform differently in various harnesses. The authors recommend testing both the model and its environment together when selecting a coding agent.
