What changed
AWS has introduced a new agent evaluation playbook for Amazon Bedrock AgentCore, integrating built-in evaluators, DeepEval, and AutoEval. This framework helps teams assess agent behavior beyond basic metrics, focusing on business outcomes and hidden performance issues.
What this means in practice
The playbook guides teams to start with business outcomes rather than metrics. For example, a customer support agent's success might be defined by first-contact resolution, while a research agent's success could be producing a complete, well-sourced answer. Teams are encouraged to break these outcomes into observable behaviors and map them to specific evaluators at the trace or session level. AWS provides a broad catalog of evaluators, including built-in ones, third-party managed evaluators from DeepEval and AutoEval, and custom evaluators. These evaluators cover various dimensions such as response quality, task completion, tool use, and safety. The multi-dimension evaluators allow for scoring multiple criteria simultaneously, providing detailed insights into agent performance.
Limitations and availability
The supplied announcement does not specify the exact release date or availability of the new evaluation tools. The evaluators are part of Amazon Bedrock AgentCore, and teams can use them to assess their agents' behavior. Custom evaluators can be created based on existing ones, offering greater control over the evaluation process. The tools are available for use within the AWS ecosystem, and the full catalog of evaluators can be accessed via the ListEvaluators API.
