What changed
A new report from the AI evaluation nonprofit METR updates its time horizon plot, which tracks AI progress by task difficulty. The report adds more tasks, including those that require judgment, which were previously excluded due to their difficulty in scoring. METR evaluates models by comparing their performance on tasks to human completion times, with the original plot showing AI systems improving rapidly on verifiable tasks.
What this means in practice
The updated report includes tasks that are more open-ended or require human judgment, such as designing AI evaluations or scoring philosophical arguments. METR researchers note that these tasks are harder to evaluate and score, and the new report includes fewer of them due to the challenges in designing and quantifying them. The most advanced models, such as Opus 5.5, show some improvement on these tasks, achieving 'borderline above the bar' performance, but they are not yet usable for METR's research. The report also highlights that progress on verifiable tasks continues, but it remains unclear how quickly models might improve on more complex, judgment-laden tasks.
Limitations and availability
The new report faces challenges in accurately measuring performance on open-ended and judgment-heavy tasks due to the difficulty in designing and scoring them. Completion times for these tasks are mostly estimates, as paying people to complete them is prohibitively expensive. Additionally, some tasks, like scoring philosophical arguments, rely on limited expert input, which may affect the reliability of scores. METR acknowledges these limitations and notes that predicting progress on such tasks remains difficult. The report does not specify when these tasks will be fully integrated into the evaluation framework or how they will be scored in the future.
