IBM Research has launched the Open Agent Leaderboard, an open benchmark designed to assess the performance of AI agent systems by evaluating both quality and cost. The leaderboard includes six diverse benchmarks covering tasks such as coding, customer service, and technical support. According to IBM, the leaderboard aims to provide a comprehensive evaluation of agent systems, not just the models within them. The initiative is paired with the Exgentic framework for running and reproducing evaluations, and a paper detailing the methodology and results.
All resources are available from day one. IBM emphasizes that the leaderboard measures how well agents perform across different settings, focusing on generality and cost efficiency. The benchmarks were selected to test a variety of tasks, each with different tools, rules, and constraints. The unified protocol ensures that all benchmarks have a shared structure, allowing for standardized evaluation.
The leaderboard reports the average success rate, average cost per task, and per-benchmark breakdowns for each agent system. The current top five configurations show that the same model can yield different results based on the agent system used. IBM notes that the results highlight the importance of agent design, as the same model can produce varying outcomes when paired with different agent systems. The leaderboard also reveals that general-purpose agents are already competitive with specialized ones, as some agents without benchmark-specific tuning matched or outperformed specialized systems.
The evaluation also shows that agents differ significantly in how they handle failures, with some failing quickly and others incurring higher costs. IBM's paper details the full methodology and empirical analysis of the evaluation.
Source: huggingface
