HuggingFace researchers trained NanoColibri-Instruct, a 2.7B-parameter Mixture-of-Experts model, for approximately $200 using rented GPUs. The model, which has 0.34B active parameters per token, was trained sequentially across multiple contributors' GPUs. The team used a relay pretraining method, where each contributor took turns training a segment of the model, ensuring no overlap in training steps. The model's 5.4B-token budget outperformed token-matched dense models on 5 of 7 zero-shot tasks, despite using fewer active parameters than either baseline. The training process was fully reproducible, with code, checkpoints, benchmarking tools, and a public ledger of contributions available in the training repository. Source: huggingface
The training process involved a unique relay system, where contributors took turns training segments of the model using a shared repository on the HuggingFace Hub. Each contributor claimed a segment, trained it, and pushed the updated weights back to the repository. To prevent conflicts, the system used a compare-and-swap mechanism, ensuring that only one contributor could train a segment at a time. This system also included a dashboard that showed real-time metrics like loss, throughput, and who held the training baton. The process resulted in 20,000 updates across multiple contributors, with a final 1,500-update chat SFT on smol-smoltalk, requiring about 90 H100-hours of compute. Source: huggingface
The model was built using the serving engine's native architecture, HYV3, with 24 layers, a shared expert, and top-2 routing. The architecture included a fat 2048-wide shared expert that never left memory, GQA 4× attention with per-head QK-norm, and a sigmoid+bias router with aux-loss-free load balancing. The int4 export of the model leveraged these features, allowing it to run efficiently on disk-based streaming systems. Source: huggingface
