Researchers from Microsoft and City University of Hong Kong developed Taste-Bench, a benchmark to evaluate how well AI agents make long-term decisions. The best model tested, GPT-5.6 Sol, scored 59.7% on the benchmark, showing significant room for improvement.
Taste-Bench measures an agent's ability to make good choices during multi-step tasks, such as writing research papers or building software projects. These decisions often have delayed effects, with their true cost only becoming apparent after significant resource use.
The benchmark uses a method called 'decision forks,' where models must choose between two paths before later results reveal which was better. It includes 502 questions drawn from real agent runs on coding and AI research tasks, including SWE-bench Pro and MALT.
"The later path of a trajectory shows if an earlier choice worked," said the researchers. "This branching point, called a decision fork, allows us to test an agent's judgment over time."
The benchmark highlights a gap in current AI evaluation, as most benchmarks only check if a task is completed, not the quality of the steps taken. The team also showed that taste can be trained, improving model performance on tasks like SWE-bench Pro.
Microsoft and City University of Hong Kong did not say how widely Taste-Bench will be adopted, and the team raised questions about whether models can improve further with more reasoning. The benchmark and code are publicly available.
Source: thedecoder
