What changed
Hugging Face's TRL (Transformers Reinforcement Learning) now supports Multi-Teacher On-Policy Distillation (MOPD). The AsyncDistillationTrainer routes each prompt to its own expert teacher, and support for the synchronous DistillationTrainer is on its way in #7215. This update enables models to be trained using MOPD, a technique that allows a single model to excel across multiple domains by learning from multiple domain-specific experts.
What this means in practice
MOPD is a post-training technique that involves three stages: General Supervised Fine-Tuning (SFT), Domain Expert Training, and Fusion. During the fusion phase, a model learns from multiple domain-specific experts, each trained to be highly proficient in a single domain. This approach helps avoid the issue of conflicting training signals from different domains, as each prompt is scored by its corresponding domain's expert.
TRL's implementation of MOPD allows users to route prompts to the appropriate expert teacher, with each expert providing token-level feedback.
The training process uses reverse KL divergence between the student and the teacher's distribution, with the teacher only providing feedback on the tokens the student generates.
This method ensures that the student model learns to align with the distribution of the expert teacher for each domain.
Limitations and availability
The supplied announcement does not specify the exact hardware requirements or licensing conditions for using MOPD in TRL. The example provided uses the SmolLM3-3B model and three domain-specific experts trained with GRPO for math, code, and instruction following. The availability of expert checkpoints is still limited, as most labs only release the final model. However, NVIDIA has released five domain-specific teachers on the Hub for STEM, competition coding, instruction following, general reasoning, and chat. These resources are expected to become more widely available in the coming months.
