Amazon SageMaker HyperPod Uses Tiered KV Cache with Curvine for Large LLMs

Amazon SageMaker HyperPod introduces a tiered KV cache architecture that extends the cache hierarchy beyond GPU and CPU memory into a shared, distributed NVMe pool. This approach allows for cross-replica reuse of KV cache at near-local-disk speeds, improving performance for large language model (LLM) inference at scale. The solution leverages two HyperPod capabilities—Managed Tiered KV Cache and Intelligent Routing—and adds Curvine, a lightweight distributed cache filesystem, as the shared L2 tier. The implementation enables reuse of KV cache across replicas, reducing time-to-first-token (TTFT) and infrastructure costs. The architecture supports a three-tier hierarchy: L0 (GPU HBM), L1 (local CPU/host memory), and L2 (Curvine, a shared cross-node cache). This setup helps address the trade-off between GPU costs and TTFT degradation for large models.

The tiered KV cache system works by offloading GPU blocks to CPU memory when needed, using LMCache to catch them in host DRAM before they’re lost. This is managed automatically by the SageMaker HyperPod Inference Operator when enableL1Cache is set to true. The L2 tier, powered by Curvine, pools local NVMe drives from G6e/P5 instances into a single namespace, which is mounted as a ReadWriteMany PVC into every inference Pod. LMCache reads and writes through its fs:// connector, making the distributed pool appear as a local directory. This allows a KV block written by one replica to be immediately readable by others, enhancing cache reuse across replicas. The system also includes an Intelligent Routing feature that directs requests to the replica most likely to produce a cache hit, based on prefix-aware or kv-aware strategies.

The solution is designed to address the limitations of current systems, where each vLLM replica operates in isolation, leading to cold starts and reduced cache hit rates. By extending the cache beyond what fits on a single Pod, the architecture reduces the need for oversized GPU instances and improves performance for workloads with high prompt overlap. This approach allows workloads that previously required P5 instances to run on lower-cost G6e instances, reducing per-endpoint cost. The implementation includes a detailed process for enabling HyperPod Tiered Storage, deploying Curvine workers on node-local NVMe, and patching the Inference Operator for filesystem-backed L2.

Source: awsml