A new late-interaction retriever called GLInt has surpassed its predecessor, LateOn, in retrieval performance. Developed by researchers at Hugging Face, GLInt demonstrates superior effectiveness in late-interaction retrieval tasks. The model achieves a mean nDCG of 57.43 on the 15 BEIR Tasks, surpassing the 57.22 score of LateOn. This marks GLInt as the strongest retriever under 300M parameters in this comparison. Additionally, it sets the highest Quora and HotPotQA scores among models below 7B parameters. The model and training data are available on the Hugging Face platform.
The development of GLInt involved several key techniques, including multi-vector mining and MaxSim geometry. Researchers found that hard negatives mined by a multi-vector retriever produce better late-interaction models than those mined by a dense retriever. The training process included knowledge distillation and a shift from SFT to distillation using a Jina teacher with Mixed KL/InfoNCE. These methods contributed to the model's improved performance. The experiments also highlighted the importance of geometry-matched selection in the MaxSim space, which helps late-interaction students.
The project began with a narrow question: whether hard negatives mined by a multi-vector retriever would produce a better late-interaction model than those mined by a dense retriever. The results showed that the entire data pipeline being multi-vector had significant effects, including compressed score distributions and increased false-negative exposure. The findings underscore the importance of adapting training methods to the specific geometry of late-interaction retrieval. The model's success highlights the potential of geometry-matched hard negatives in improving retrieval performance.
Source: huggingface
