Distillation has become a foundational technique in the post-training processes of 2026's leading models. The method allows large models to train smaller, more efficient versions while preserving performance. This technique is also used to merge multiple domain-specific experts into a single model, improving versatility and efficiency. According to Hugging Face, distillation is widely adopted across various stages of model development, with different flavors of the technique being applied depending on the goals of the training process.
The original use of distillation involves training a smaller student model to match the performance of a larger teacher model. Gemma 4's tech report describes a similar post-training recipe, indicating that distillation is still a key component in model development. Additionally, DeepSeek-R1-Distill demonstrates how reasoning traces from a large model can be distilled into smaller models through fine-tuning. This approach, known as off-policy distillation, involves either matching the teacher's next-token distribution or directly training on the teacher's generated text.
The newer applications of distillation focus on integrating multiple domain-specific experts into a single model. This method, called on-policy distillation, involves training a student model that generates its own rollouts while being graded by specialized teachers. For example, DeepSeek-V4 uses this technique to train a unified model through on-policy distillation, where the student optimizes the reverse KL loss against domain-specific teachers. This approach allows models to learn from multiple sources of expertise while maintaining efficiency and performance.
Source: huggingface
