Hugging Face demonstrated that its VKUE engine can run a 34.7B parameter model on an 8 GB laptop and a CPU server with no GPU. The model, Ourbox-35B-JGOS, achieved 20.01 tok/s on an 8 GB laptop and 17 tok/s on a CPU-only server. These results were measured across different hardware tiers, from a datacenter B200 GPU to a consumer-grade device. The performance highlights the efficiency of the VKUE engine, which leverages sparse MoE architecture to reduce memory traffic. Source: huggingface
The model's sparse MoE structure allows only about 3B parameters to be active per token, significantly reducing memory bandwidth usage. This design enables the model to run efficiently on hardware with limited resources. Hugging Face measured the throughput across various setups, including a datacenter B200 GPU, an A10G GPU, an 8 GB laptop, and a CPU-only server. The results show that the same model weights can operate across a wide range of hardware, from high-end GPUs to low-end devices. Source: huggingface
The VKUE engine is part of Hugging Face's efficiency-serving line, designed for accessibility rather than raw speed. It allows frontier-class models to run on minimal hardware, such as a CPU server or an 8 GB laptop. The demonstration included live demos where users can compare GPU and CPU performance side by side. The results show that while GPU performance is faster, the model can still run effectively on a CPU without any GPU. Source: huggingface
