A collaboration between Hugging Face and EleutherAI, the FineBooks project tested 14 open-source OCR models on more than 2,000 pages from historical books to assess their ability to convert scanned texts into usable training data for AI language models. The best-performing models achieved character accuracy above 97% at a cost of less than two U.S. dollars per thousand pages, according to the researchers. The team noted that while the output quality is sufficient for AI training, the models remain too error-prone for scholarly or scientific applications. The project focused on the Biodiversity Heritage Library, which holds over 300,000 digitized natural history documents totaling more than 64 million pages. The BHL offers its collection as a bulk download via AWS, and the team used transcriptions from the IMPACT project and BHL-Europe as ground-truth data for evaluation. The evaluation framework is openly available, and new models are added to the leaderboard on a rolling basis. The team plans to reprocess about 200,000 public-domain BHL documents with one of the top models and release the text as an open dataset. The evaluation covers only Antiqua typefaces in four languages and does not account for Fraktur, non-Latin scripts, or handwriting. FineBooks is limited to single-column book pages. The team noted that the models produce far fewer errors than older pipelines, and reprocessing a collection the size of BHL is realistic at the measured costs. However, libraries face a different problem, as their systems rely on ALTO XML, a format with word-level coordinates. The new models output Markdown or plain text without word positions, so they can't plug into existing library infrastructure. For scholarly transcriptions, accuracy still falls short, but not because the models misread characters. They silently modernize them, swapping the long 's' or ligatures for modern equivalents. Targeted fine-tuning could fix this, the team says. The FineBooks authors judge results by intended use. For training language models, the top performers work well enough, they write. The models produce far fewer errors than the older pipelines, and reprocessing a collection the size of BHL is realistic at the measured costs. Libraries face a different problem, the team notes. Their systems rely on ALTO XML, a format with word-level coordinates. The new models output Markdown or plain text without word positions, so they can't plug into existing library infrastructure. For scholarly transcriptions, accuracy still falls short, but not because the models misread characters. They silently modernize them, swapping the long 's' or ligatures for modern equivalents. Targeted fine-tuning could fix this, the team says. AI News Without the Hype – Curated by Humans to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section. now
Source: thedecoder
