AMD has announced the release of ROCm™ AMD Infinity Context (ROCm AIC), a new software and hardware solution designed to manage key-value (KV) cache in distributed inference workloads. The technology addresses the increasing challenges of handling long-context requests, agentic workflows, and multi-turn conversations in AI systems. As models grow larger and context windows expand, the demand for efficient KV cache management has become critical for maintaining performance and cost efficiency in production environments.
The solution combines AMD hardware with open-source frameworks to enable low-latency data transfer between network-attached storage and GPU High-Bandwidth Memory (HBM), bypassing the CPU memory bus entirely. This approach allows for faster access to KV cache data, reducing the computational overhead associated with reprocessing large context windows. ROCm AIC is compatible with AMD Instinct MI300X, MI350, MI450, and MI455X platforms, offering support for both front-side Ethernet and high-bandwidth scale-out networks. The system is designed to work with existing deployment ecosystems and integrates with popular open-source inference frameworks like vLLM and LMCache.
The problem of KV cache management has become a significant bottleneck in AI inference, particularly as models like GPT-OSS-120B and Qwen3 support context windows of up to 128K–1M tokens. At 100K tokens, the computational cost can exceed trillions of FLOPs per turn, with single 1M-token requests generating over 600 GB of KV data—more than the HBM capacity of an entire MI455X node. Local NVMe storage is limited by node-bound constraints, and general-purpose network file systems are too slow for modern inference demands.
Source: amd
