What changed
IBM Research's ALTK-Evolve, using GPT-4.1, achieved a 67.5% Pass^4 score on τ²-bench retail tasks, surpassing all retail entries in the October 12, 2026 leaderboard snapshot. This was accomplished by appending learned guidelines to the system prompt of a basic ReAct agent, without updating model weights or requiring human annotation.
What this means in practice
The ALTK-Evolve approach involves analyzing an agent's training-task attempts to distill successes and failures into short behavioral rules. These guidelines, such as authenticating users before taking action or confirming changes with users, are then added to the agent's prompt for subsequent conversations. The same GPT-4.1 model powers both the guideline extraction and the agent, eliminating the need for weight updates or human intervention. This method was tested on customer service tasks, where success requires following domain policies, understanding customer requests, and executing the right tool actions.
Limitations and availability
The approach was tested on τ²-bench's retail and airline domains, using GPT-4.1 as the agent model and GPT-5.5 as the customer-simulator model. The learned guidelines are appended as a plain markdown list after the domain policy in the system prompt. The study evaluated three guideline configurations: baseline (no guidelines), sup3 (guidelines with support ≥ 3), and lossless (all guidelines). The results showed that more guidelines improved performance in retail tasks, while in airline tasks, the compact sup3 core provided better reliability. The cost of using guidelines is reflected in increased prompt input tokens, not in longer conversations. The supplied announcement does not specify the exact availability or access route for this method.
