Hugging Face and EleutherAI have launched the FineBooks OCR Leaderboard, a benchmarking tool that evaluates 14 open-source OCR models on 2,165 historical book pages from the Biodiversity Heritage Library. The initiative aims to assess whether current open OCR models are sufficient to unlock historical knowledge at scale. The leaderboard provides scores for each model, allowing users to compare performance and verify results. The project also includes a ground-truth dataset and an evaluation framework for reproducibility.
The leaderboard tests models on a diverse set of historical documents, including single-column English natural history texts, multilingual table of contents, and engraved plates. These pages were transcribed by expert researchers, creating a reliable benchmark for OCR accuracy. The evaluation includes multiple metrics, such as Character Error Rate (CER), Recall, Over-extraction, and Loop Rate, to provide a comprehensive view of model performance. The results highlight the challenges of OCR on historical texts, which often feature archaic letterforms, complex layouts, and languages no longer in common use.
The project builds on the work of the IMPACT project and BHL-Europe, which produced expert-corrected transcriptions of six BHL volumes between 2011 and 2012. These transcriptions, released under an open license, form the basis of the ground-truth dataset used in the evaluation. The Biodiversity Heritage Library was chosen as the first collection due to its open data availability via AWS Open Data, enabling the creation of a modern ground-truth dataset.
Source: huggingface
