What changed
Hugging Face introduced a method to extend Mixture-of-Experts (MoE) models by adding new domain-specific experts, such as those trained for Rust and Haskell programming, without retraining the base model. These experts are modular and can be combined with other domain experts in a single checkpoint.
What this means in practice
The approach allows users to enhance a model's capabilities in specific domains by adding new experts trained on domain-specific data. These experts are trained independently and can be integrated into the model without affecting its existing capabilities. The new experts are added to the model's parameter space, and the model's router learns when to activate them. This method avoids the interference issues seen with traditional fine-tuning techniques like LoRA.
Limitations and availability
The method was tested on a 35 billion parameter model, with each new expert requiring approximately 125 million parameters. The approach is available for use with Hugging Face checkpoints, and the new experts can be combined with other domain experts. The results show improvements in domain-specific tasks without a drop in general capabilities. The supplied announcement does not specify the exact conditions under which the new experts are available for use beyond the described testing environment.
![Source facts for Hugging Face Extends MoE Models with Domain-Specific Experts: Back to Articles Bring Your Own Expert! This is called a Mixture-of-Experts (MoE) architecture [1][2]. The efficiency gain is substantial: the model has the capacity of its full size, but only runs a fraction of it on any given input.](/images/articles/hugging-face-extends-moe-models-with-domain-specific-experts-editorial-facts.png)
