What changed
Amazon Bedrock AgentCore Evaluations now supports decision models as part of its evaluation framework. These models are designed to answer specific, repeatable questions about agent responses by returning calibrated probabilities over a fixed set of allowed answers, rather than free-form text.
What this means in practice
The addition of decision models allows for more precise and programmable scoring during agent evaluations. These models can be used alongside built-in evaluators to provide a direct confidence measure that can be thresholded and acted upon. Decision models are particularly useful for tasks where the judgment can be framed as a choice among a fixed set of options, such as determining if a response is grounded in the provided context or if the agent completed the user's goal.
Limitations and availability
Decision models are available through custom evaluators defined as either an LLM-as-a-judge or a code-based evaluator. The supported decision models include Jev, a hosted decision model from TypeSafe, and Strands Decider 2B, an open-source decision model. These models can be integrated into the evaluation pipeline through code-based evaluators, allowing for a screening layer that identifies responses requiring deeper assessment by built-in evaluators. The feature is available in regions where Amazon Bedrock AgentCore Evaluations is supported, with the sample defaulting to US West (Oregon).
