Google has revamped its Android Bench benchmark to better assess how large language models (LLMs) perform in Android app development. The updated leaderboard now includes eight new models, such as Claude Fable 5, and introduces metrics like cost and efficiency. Developers are encouraged to run their own tests and submit feedback to shape the future of Android Bench. The company said it aims to create a more accessible framework for evaluating AI agents in development tasks. Source: arstechnica
The new leaderboard features models like Claude Sonnet 5, GLM 5.2, and Qwen 3.7 Plus, among others. Fable 5 leads with an accuracy of 84.5 percent in the 100-problem, 10-run benchmark. However, Fable 5 and GPT 5.5 have high operating costs, exceeding $130 per benchmark run. In contrast, Gemini 3.1 Pro, while scoring lower, costs only $87 to run. The benchmark also highlights that Gemini 3.5 Flash, despite being marketed as a cost-effective option, has the highest cost at $165 per run due to its 28-hour runtime. Source: arstechnica
Google’s Android Bench is designed to evolve with new workflows and development tasks. The company has transitioned to the Harbor framework, which it claims makes it easier for developers to run, evaluate, and share results. Google re-ran previous tests with Harbor to establish a new baseline for LLM performance, though the underlying tests remain unchanged. The historical data is archived online, and developers can now submit their own tasks for potential inclusion in the official benchmark. Source: arstechnica
