What changed
Amazon SageMaker AI introduced KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints. This feature uses the cache state reported by the model container to direct requests to instances that already hold a long cached match for the prompt and aren't too busy.
What this means in practice
The change is aimed at teams serving large language models (LLMs) with multi-turn conversations, long shared prompts, or heavy loads. To enable KV-aware routing, users must add an environment variable to their model so the container reports its cache contents and set the RoutingStrategy to KV_AWARE in the endpoint configuration. Clients continue to call the endpoint with the same APIs as before.
KV-aware routing works by having the model container report its cache state, SageMaker AI maintaining a managed cache index of that state, and the endpoint using the index to select an instance for each request.
When a request arrives, the endpoint tokenizes the prompt and looks up the tokens in the index. The index returns instances or inference component copies that hold part of the prompt in their cache, and how much of it each one holds.
The endpoint then selects among these candidates by weighing how much of the prompt each one has cached against how busy it is. If no candidate has a useful match or all are busy, the request falls back to load-aware routing.
Limitations and availability
KV-aware routing is active only with at least two instances or two copies of an inference component. With a single instance, the setting has no effect. The feature is available for real-time endpoints and requires a supported container with prefix caching and KV cache events. The index also follows the fleet, so entries for removed instances are dropped when scaling in, and new instances start with empty caches when scaling out or updating, which may temporarily increase latency. The feature is recommended for LLM workloads on a supported container with multi-turn conversations, frequent cache eviction, or requests that share only part of their start.
