AMD Introduces 4-bit KV Caching in LMCache for MI355X

AMD has introduced a new 4-bit key-value (KV) caching solution called LMCache for its MI355X GPU, designed to address memory constraints in large language model (LLM) serving. The technology combines 4-bit quantization with a tiered cache system that moves data between GPU HBM and CPU DRAM, improving efficiency for context-heavy agents. This approach allows for more efficient use of memory, reducing the need for frequent recomputation of reused context. According to AMD, the solution maintains accuracy while significantly improving performance metrics like goodput and latency. The LMCache system is built on top of vLLM and integrates with AMD's ROCm platform, supporting mixed bf16 and 4-bit KV stacks for optimized memory usage. This innovation is part of AMD's ongoing efforts to enhance AI model serving efficiency on its hardware.

The 4-bit KV caching solution leverages AMD's TurboQuant technology, which compresses the KV cache to 4-bit, making each token approximately 3.76 times smaller. This compression allows more tokens to remain in fast HBM memory before spilling to slower CPU DRAM, while also reducing the data payload during transfers. AMD reported that this approach increases the number of KV tokens that can reside in HBM by about 3.2 times compared to full-precision bf16 caching, significantly improving cache hit rates. The LMCache system also supports a tiered memory hierarchy, with data being moved between GPU HBM and CPU DRAM as needed. This tiered approach helps maintain performance while managing memory constraints, especially for large context sizes that exceed HBM capacity.

The implementation of LMCache involves integrating it with vLLM's KV-connector hook, allowing for mixed bf16 and 4-bit KV stacks to be managed across GPU and CPU memory. AMD's solution keeps the most accuracy-sensitive layers in bf16 while compressing the middle layers to 4-bit, balancing precision and memory efficiency. The system also includes a layout-aware connector that ensures data is moved accurately between memory tiers without losing precision. This approach enables efficient memory usage while maintaining the integrity of the model's output. AMD tested the solution on a 230 GB FP8 mixture-of-experts model, using two MI355X GPUs with tensor parallelism, and reported significant improvements in goodput and latency under high-load conditions.

Source: amd