LiquidAI announced the release of QAD Q4_0 GGUFs for LFM2.5 models, enabling developers to run these models at Q4_0 memory and speed without significant quality loss. The updated checkpoints are available for LFM2.5-230M, LFM2.5-350M, LFM2.5-1.2B-Instruct, and LFM2.5-2.6B. These models are trained using Quantization-Aware Distillation (QAD), where a high-precision teacher model is distilled into a quantized student model. The QAD checkpoints maintain the low memory footprint and high throughput of native Q4_0 GGUFs, with 97% of their BF16 average accuracy recovered from quantization.
According to LiquidAI, the QAD checkpoints significantly outperform post-training quantization (PTQ) versions across various benchmarks. The models were tested on a suite of benchmarks including GPQA Diamond, MMLU-Pro, IFEval, IFBench, Multi-IF, and BFCLv4. For smaller models, QAD Q4_0 checkpoints match Q5_K_M quality at a 4-33% higher decode throughput.
For larger models, they match Q4_K_M quality at a 3-14% higher throughput. The QAD checkpoints also match Unsloth's UD-Q4_K_XL, where applicable, for the 230M and 1.2B models. Developers can use these GGUFs with llama.cpp or any runtime that supports GGUF Q4_0 artifacts.
The QAD GGUFs are now available on Hugging Face. LiquidAI encourages the community to explore the potential of these models. Source: huggingface
