Hugging Face has released DoctoBERT, a pretrained medical encoder designed to process French medical text. The model is tailored for tasks such as named entity recognition and classification, where understanding clinical documents is crucial. DoctoBERT is pretrained on FineMed, a medical corpus sourced from the open web, which provides scale and diversity not achievable with smaller hand-built corpora. The model's effectiveness is demonstrated through its ability to generate clinically relevant terms, such as pneumothorax and amoxicilline, when compared to general-purpose models like CamemBERT.
DoctoBERT comes in two versions: DoctoBERT-fr-base and DoctoModernBERT-fr-base. The latter, based on ModernBERT, is recommended for its efficiency in batched inference and support for longer context lengths. Both models are available under the Apache-2.0 license on the Hugging Face Hub. The release includes a data curation pipeline and annotated corpora, such as FineMed, which is a 19.2B-word annotated corpus and FineMed-rephrased, a 4.5B-word subset rephrased by large language models.
The blog post provides practical examples of how to fine-tune DoctoBERT for tasks like named entity recognition, classification, and semantic similarity. It includes code examples and guidance on installing dependencies and loading datasets for these tasks. The release marks a significant step in advancing clinical NLP applications using French medical text.
Source: huggingface
