What changed

AWS announced a new skill for the Agent Toolkit that enables distributed key-value (KV) caching for self-hosted large language models (LLMs). This skill uses Amazon ElastiCache to extend GPU-local caches with a shared tier, allowing servers to reuse KV states across sessions, routing shifts, and server failures. The feature is designed to cut self-hosted LLM inference costs by up to 40% and reduce time to first token (TTFT) by up to 80%.

What this means in practice

The skill helps coding agents like Kiro, Claude Code, and OpenAI Codex evaluate and set up distributed KV caching. It guides users through checking if their workload benefits from the caching, setting up a cache cluster for testing or production, and troubleshooting. The skill works with existing engines like vLLM and SGLang, adding the LMCache connector to manage configuration across inference servers. It also includes a tool called KVTrace to estimate potential savings based on model and traffic data before deployment.

Limitations and availability

The skill requires a coding agent that supports Agent Skills, such as Kiro or Claude Code, and GPU instances that support EFA RDMA write, like p5 instances. It also needs LMCache 0.5.8rc1 or later, vLLM 0.11.1 or later, and ElastiCache for Valkey 9.1 or later on cache.i8ge.48xlarge nodes. The skill is available for use with the Agent Toolkit for AWS, and the setup process involves confirming the cache node count, creating security groups, and configuring Valkey clusters with custom parameter groups and caching eviction policies.

Source facts for AWS Introduces Distributed KV Caching for Self-Hosted LLMs: Consider a legal AI assistant that can accumulate hundreds of thousands of tokens of case history in each conversation. When a lawyer returns to a thread after lunch, responses from the LLM can take minutes. The GPU server that picks up the request has none of that history in its memory.
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. awsml: Cut self-hosted LLM cost and latency with the distributed KV caching agent skill