What changed
AMD has introduced a new feature called online quantization, which allows AI models to be quantized during the model loading process rather than beforehand. This approach eliminates the need for a separate quantization script, calibration pass, and the creation of a new checkpoint file. The serving engine reads a high-precision checkpoint and quantizes each layer's weights inside its weight-loading hook, just before the model starts serving.
What this means in practice
The feature is particularly useful in scenarios such as reinforcement learning rollouts, requantization between different quantized formats, and fast exploration of mixed-precision schemes. Online quantization enables users to make minor modifications to existing quantized checkpoints without regenerating the entire model. For example, it allows for quantizing additional layers or adjusting specific modules without the need for a full re-export. This approach significantly reduces the iteration cost, as switching or mixing quantization schemes can be done with a simple configuration edit and a reload.
Limitations and availability
Online quantization does not support calibration data or advanced accuracy-recovery algorithms like rotation, SmoothQuant, GPTQ, or AWQ. It only uses plain round-to-nearest (RTN) quantization, which may result in lower accuracy compared to offline methods. Additionally, peak memory usage and load time are higher than with offline quantization because the full high-precision checkpoint is still read. However, layerwise loading helps keep these runtime costs bounded. The feature is available in AMD Quark online quantization and integrates with serving engines like vLLM, ATOM, and SGLang.
