Hugging Face released an updated BERT tokenizer on October 15, 2026, saying it significantly improved encoding speed. It is the company's first major update to the tokenizer library since its v0.23 release.
Hugging Face reported a 50× speed increase in tokenization throughput, measured on English text encoding. That compares with a 30× improvement in tokenizers v0.23.
The updated tokenizer is built on the BPE algorithm and targets models using BERT-like tokenization. Availability begins with the Pull Request stage, initially for developers and researchers.
"Normalizer: 88%" said Simon Brandeis, Hugging Face contributor. The majority of encoding time is spent on normalization, which includes Unicode processing and character checks.
The announcement follows the revamping of the tokenizers library. Hugging Face framed the update as a step toward stabilizing improvements in a v1.0.0 release.
Hugging Face did not say how the update will affect other tokenizers, and the company raised the question of whether further optimization is possible. The changes are still in Pull Request stage.
Source: huggingface
