AMD released UltraQuant, a 4-bit KV-cache method, on September 25, 2026, saying it improves decode throughput and cuts latency under load without a measurable drop in reasoning accuracy. It is the company's first software update for agentic serving since its earlier optimizations for long-context models.
AMD reported a 29% increase in decode throughput and a 24% reduction in per-token latency, measured on a real agentic replay with 32 in-flight requests. That compares with the fastest 8-bit KV backend under the same conditions.
UltraQuant is built on AMD Instinct GPUs and targets long-context agentic serving with Qwen3.8-MXFP4. Availability begins with the release of the UltraQuant kernels, initially for developers using AMD Quark and vLLM.
"UltraQuant stores keys and values on the FP4 grid the MI355X already understands, with one shared power-of-two scale per group of 32 channels," said Aditi Ghai Rana. The method reduces memory reads by roughly half per decode step, improving performance under high concurrency.
The announcement follows AMD's release of the MXFP4 Qwen3.8 checkpoint. AMD framed the significance as a step toward more efficient large-scale model serving, emphasizing the balance between performance and resource constraints.
AMD did not say how UltraQuant will perform with other models beyond Qwen3.8-MXFP4, and raised the question of whether the technique will scale to even longer contexts. The company noted further testing and benchmarking will be conducted in the coming months.
Source: amd
