AWS and Motorway have developed a production-ready blueprint for evaluating AI agents, combining the Strands Agents SDK with Amazon Bedrock AgentCore to improve reliability and performance. The solution addresses challenges such as tool selection errors, semantic search misinterpretations, and context drift in multi-turn conversations. By implementing an end-to-end evaluation pipeline, the team reduced incorrect results from 1 in 8 queries to 1 in 50 and cut issue detection time from hours to minutes. This approach enables developers to build and monitor AI agents with greater precision and confidence, ensuring they meet quality and reliability standards before deployment. The blueprint provides a deployable framework that can be adapted for various applications, emphasizing the importance of structured testing and continuous monitoring in production environments.
The blueprint outlines a two-phase evaluation strategy that spans build-time testing with strands-agents-evals and production monitoring with Amazon Bedrock AgentCore Evaluations. It also includes a three-layer framework for assessing tool usage, reasoning, and output quality, along with a five-stage deployment pipeline featuring quality gates that block releases when metrics fall below defined thresholds. The framework uses the pass^k metric to ensure consistency and reliability, and it supports domain-specific customization through custom evaluator subclasses. These features enable developers to create AI agents that are both effective and safe for real-world use, addressing critical challenges such as data freshness, dealer scoping, and safety guardrails.
The Motorway dealer stock search agent serves as a worked example, demonstrating how the blueprint can be applied in practice. The agent uses the Strands Agents SDK and Amazon Bedrock AgentCore to handle complex queries, such as filtering vehicles by type, age, and other attributes. With around 1,500 concurrent users during peak hours, the agent must operate reliably to maintain user trust. The end-to-end request flow involves natural language queries from dealers, which are processed through Amazon Bedrock AgentCore Runtime to orchestrate calls to eight different tools. The runtime uses Amazon Bedrock models (Claude for reasoning, Amazon Titan for embeddings) to generate accurate and relevant results for users.
Source: awsml
