AI teams face challenges when evaluating agent frameworks due to the lack of standardized evaluation tools. Most systems are tied to specific SDKs or LLM clients, making it hard to evaluate agents built with different frameworks. Amazon Bedrock AgentCore Evaluations addresses this by decoupling evaluation from the framework choice. The solution leverages OpenTelemetry to read and score agent telemetry regardless of the underlying SDK. This allows teams to evaluate agents built with frameworks like LangGraph, LlamaIndex, or Google ADK without rewriting evaluation pipelines. The service automatically reads spans and event records from CloudWatch, reconstructing sessions to score agents based on predefined metrics. This framework-agnostic approach simplifies the evaluation process for developers using various agent frameworks on Amazon Bedrock AgentCore runtime.
The evaluation service uses OpenTelemetry to read traces generated by agents, which include spans representing model calls, tool invocations, and user interactions. These spans are exported via the OpenTelemetry Protocol (OTLP) and collected by the AWS Distro for OpenTelemetry (ADOT), which routes them to Amazon CloudWatch. The service reads three key span roles: invoke agent spans, inference spans, and execute tool spans. These spans carry the necessary data to score an agent’s performance, such as user prompts, model responses, and tool results. Additional spans, like those for retrieval or guardrail checks, are treated as context and not required for evaluation. The service is designed to be forward-compatible, handling new span types without requiring configuration changes.
Amazon Bedrock AgentCore Evaluations supports multiple frameworks, including Strands Agents, LangGraph, the OpenAI Agents SDK, LlamaIndex, Google ADK, and the Claude Agent SDK. These frameworks use either OpenTelemetry or OpenInference instrumentation to record telemetry data. The service bridges both schemas to ensure consistent evaluation across different frameworks. Instrumentation packages are used to stamp spans with scope names, which the evaluation service uses to determine how to interpret the data. This approach allows teams to evaluate agents without modifying their code, as the correct handling is activated automatically by the installed instrumentation package. The service requires that agents include a session.id attribute to group spans into traces and sessions, which is automatically injected by ADOT on AgentCore runtime.
Source: awsml
