What changed
IBM's CUGA harness demonstrated superior performance in a controlled experiment where the same open-weight model was tested with different harnesses. When run on AppWorld, a benchmark of multi-step tasks across 9 apps and 457 APIs, CUGA achieved 81.1% task completion compared to 46.7% for Hermes and 30.0% for Deep Agents. The harness also showed better consistency, passing 66.7% of 30 scenarios versus 26.7% and 6.7% for the other harnesses.
What this means in practice
For enterprise users, CUGA's performance highlights the importance of selecting a harness that aligns with specific workload requirements. Unlike Hermes, which is suited for personal assistants, and Deep Agents, which excel at long, multi-step tasks, CUGA is designed for enterprise environments where agents interact with business systems like CRM and payment platforms. It enforces rules through a policy layer that moves business logic out of the model's prompt and into the harness, ensuring compliance with organizational policies such as payment approvals and data privacy.
Limitations and availability
The results are based on a 90-task sample on one benchmark, so generalization to other tasks may vary. The setups were not identical, and the runs did not use policies, which are part of CUGA's enforcement mechanisms. CUGA is available as open-source on GitHub and supports both open-weight and hosted models, with deployment options including sovereign clouds and air-gapped environments. The supplied announcement does not specify exact hardware requirements or licensing conditions for CUGA.
