HuggingFace released UltraBERT on October 14, 2026, saying it is a family of English encoders with native 32K context support. It is the company's first encoder update since the release of the original BERT model.
HuggingFace reported 91.0 on GLUE and 47.7 on BEIR for the 380M model, measured on standard benchmarking tasks. That compares with 89.4 on GLUE and 45.5 on BEIR for the 150M model.
UltraBERT is built on a modern architecture with RoPE, SwiGLU, and alternating local/global attention and targets classification and retrieval tasks. Availability begins immediately, initially for developers and researchers.
"Encoders stop improving after 1–2T tokens, even while the loss keeps going down," said Jungwoo Park, lead researcher at dmis-lab. This suggests that further gains require changing the training recipe itself.
The announcement follows the release of FineWeb-Edu, a dataset filtered by encoder scores. HuggingFace said the release of UltraBERT addresses the trade-off between classification and retrieval performance.
HuggingFace did not say whether UltraBERT will be extended to other languages, and raised the open question of whether the trade-off between classification and retrieval is a law of nature or an artifact of training. The company said it will continue to explore training recipes for future models.
Source: huggingface
