What changed
A new open-source benchmark called CloudOpsBench has been developed to evaluate how well AI agents perform typical DevOps tasks in cloud environments. The benchmark, created by experienced DevOps engineers at the University of Michigan, includes 45 tasks on AWS and was designed to assess AI agents' ability to handle tasks such as fixing services, removing resources, restricting access rights, rolling out updates, and decommissioning projects.
What this means in practice
The benchmark reveals that even the best models struggle with many of these tasks. For instance, Claude Fable 5.1 solved 60 percent of the tasks, while GPT-6 Astra managed 53 percent. According to the team led by Ang Chen, the main issue is not a lack of knowledge but reasoning. About 80 percent of failures occur because agents misjudge the scope of a task, fail to check all relevant services, or stop too early. Roughly 20 percent of errors involve wrong actions, some of which have irreversible consequences for the infrastructure. No model completes every task, and in most cases, the bottleneck is reasoning.
Limitations and availability
All tasks run identically in the real AWS cloud and in an emulator. Extensions for GCP, Azure, and OCI are planned. The project on GitHub is licensed under Apache 2.0, and a detailed failure analysis is also available. The supplied announcement does not specify the exact version of the models tested or the conditions under which the benchmark was conducted.
