Hugging Face has released POCKET, a 35-billion-parameter model designed to run on mobile devices and personal computers without requiring a GPU. The model, part of the FINAL-Bench series, is built from Darwin-36B-Opus, a flagship model from VIDRAFT, and is quantized to reduce computational demands while maintaining high performance. POCKET is available in multiple variants tailored for different devices and languages, including Korean and English, with options for iPhones, Android phones, and PCs. The model's efficiency allows it to run on devices with as little as 8 GB of RAM, making it accessible for a wide range of users. Source: huggingface
POCKET achieves significant performance improvements over existing models, particularly on CPUs and GPUs. On a 12-core AMD mini-PC with 32 GB of RAM, it generates at around 20 tokens per second, faster than users can read, without a graphics card. The model's sparse Mixture-of-Experts (MoE) architecture activates only about 3 billion parameters per token, reducing memory usage while maintaining quality. This design allows POCKET to outperform the leading 1-bit 27B model by 2.7× on CPUs and 2.2× on GPUs, with the same quality. Source: huggingface
The model's performance is further enhanced by domain-specific pruning and mixed-precision techniques. For example, Korean and English versions of POCKET are optimized differently, with Korean benefiting more from pruning and English from compression. The English iPhone build, for instance, uses a GGUF format to support mixed-precision, while the Korean version uses MLX for Apple-native performance. These optimizations ensure that POCKET can run efficiently on a variety of devices without sacrificing quality. Source: huggingface
Source: huggingface
