What changed
AWS has introduced a new method for developers to route coding agent requests to the most appropriate model on Amazon Bedrock using NVIDIA NeMo Switchyard. This allows developers to define which models can be used and the routing strategy to apply. The solution includes two routing strategies: Stage routing, which selects between a capable model and a lower-cost efficient model based on recent tool activity and results, and Random routing, which splits traffic among configured models with optional weights for A/B testing.
What this means in practice
Developers can now optimize the cost of their coding tasks by routing requests to the most suitable model based on the task's requirements. The cost per solved task is calculated by dividing the total cost of runs by the number of runs whose tests pass. This approach helps developers evaluate routing strategies against their own workloads. The solution uses AWS Cloud Development Kit (AWS CDK) to deploy LiteLLM, an open-source gateway for large language models (LLMs), with the Switchyard routing plugin. LiteLLM calls the models on Amazon Bedrock, and the sample benchmark runners measure cost per solved task for routed model groups and single-model baselines.
Limitations and availability
The sample uses the experimental Switchyard LiteLLM integration, and its configuration and APIs may change between releases. The solution is intended for non-production usage and requires a non-production AWS account with Amazon Bedrock access to Claude Opus 5.5 and NVIDIA Nemotron 3 Super 120B. The pair has been tested in US East (N. Virginia), and regional availability should be checked for other models and tool-calling support. The deployment involves setting up a hostname with a public Route 53 hosted zone or an ACM certificate, and the solution includes security measures such as AWS WAF and private subnets for the ECS task.
