Hugging Face has introduced the Base Optimization Stack (B:OS), a new framework designed to automate the process of optimizing large language models for specific hardware. The system transforms open weights into a tuned BaseRT release tailored for a particular device, eliminating the need for manual engineering. The pipeline leverages AI research agents to handle porting and performance tuning under strict constraints, ensuring correctness before speed. The B:OS was demonstrated on NVIDIA's Nemotron 3 Nano, a hybrid mixture-of-experts model that previously lacked support for BaseRT. The results showed significant improvements in inference speed, with the model achieving 1.90× faster performance than llama.cpp and 1.43× faster than MLX on decode tasks. The pipeline's success highlights its potential to streamline the deployment of complex models across diverse hardware platforms.
The B:OS pipeline operates through a series of stages, starting with open weights and ending with a BaseRT release optimized for a specific model and device. Each stage is driven by research agents working within a controlled environment that tracks git history and maintains a global database of benchmark results. Every modification is logged as a commit and paired with a benchmark evaluation, ensuring reproducibility. The pipeline prioritizes correctness first, followed by speed, with strict constraints on quantization, kernel implementation, and performance testing. A tuned build of Nemotron 3 Nano achieved a 2.0% lower perplexity than the untuned baseline, demonstrating the effectiveness of the optimization process. The system's ability to learn and apply insights from previous tuning sessions further enhances its efficiency as it scales to new models and hardware.
The blog post outlines the challenges faced when deploying models on unsupported hardware, such as the need for custom kernels and performance tuning. The B:OS addresses these issues by automating the porting and optimization process, reducing the reliance on popular model-hardware combinations. The system's adaptability is evident in its ability to handle complex architectures like Nemotron 3 Nano, which features a hybrid design with Mamba-2 state-space blocks and sparse MoE components. The optimization process involves both correctness checks and performance benchmarks, ensuring that any improvements in speed do not compromise the model's accuracy. The results from the B:OS pipeline suggest a promising future for on-device inference, where models can be deployed at frontier speeds without requiring extensive manual intervention.
Source: huggingface
