What changed
AWS announced a new skill for the Agent Toolkit that enables distributed key-value (KV) caching for self-hosted large language models (LLMs). This skill uses Amazon ElastiCache to extend GPU-local caches with a shared tier, allowing servers to reuse KV states across sessions, routing shifts, and server failures. The feature is designed to cut self-hosted LLM inference costs by up to 40% and reduce time to first token (TTFT) by up to 80%.
What this means in practice
The skill helps coding agents like Kiro, Claude Code, and OpenAI Codex evaluate and set up distributed KV caching. It guides users through checking if their workload benefits from the caching, setting up a cache cluster for testing or production, and troubleshooting. The skill works with existing engines like vLLM and SGLang, adding the LMCache connector to manage configuration across inference servers. It also includes a tool called KVTrace to estimate potential savings based on model and traffic data before deployment.
Limitations and availability
The skill requires a coding agent that supports Agent Skills, such as Kiro or Claude Code, and GPU instances that support EFA RDMA write, like p5 instances. It also needs LMCache 0.5.8rc1 or later, vLLM 0.11.1 or later, and ElastiCache for Valkey 9.1 or later on cache.i8ge.48xlarge nodes. The skill is available for use with the Agent Toolkit for AWS, and the setup process involves confirming the cache node count, creating security groups, and configuring Valkey clusters with custom parameter groups and caching eviction policies.
