What changed
Artificial Analysis overhauled its Intelligence Index, resulting in Kimi K3.1 Pro dropping five spots to ninth place in version 5. The gap between open models and the top of the leaderboard increased to 12 points from 3 points previously. For the first time, 60 percent of the index relies on private test sets to prevent 'benchmaxxing,' the practice of training models specifically on known test questions.
What this means in practice
The updated index suggests that previous rankings may have overestimated some models' performance. Kimi K3.1 Pro's drop indicates that the old index may have made some models appear better than they were. Grok 4.7 also lost 12 points. AI researcher Nathan Lambert notes that many training gains occur in reinforcement learning environments with tasks resembling benchmarks. Keeping up across multiple domains requires sophisticated RL infrastructure, which not all labs possess.
Limitations and availability
Closed test sets used in the new index prevent outsiders from identifying errors in prompts. The AA-Omniscience knowledge test demonstrated that such errors exist. Artificial Analysis had to revise ambiguous and flawed questions in that benchmark for the new index. The supplied announcement does not specify the exact criteria or conditions for the private test sets used in the updated index.
