txtai has introduced LEMUR, a late-interaction retrieval model designed to enhance search accuracy by preserving detailed token-level information. Unlike traditional methods that collapse queries and documents into single embeddings, LEMUR retains vectors for each token, enabling more nuanced comparisons. This approach uses a MaxSim score to evaluate matches between query and document tokens, resulting in a more precise retrieval process. The model is integrated into txtai's framework to work with fixed-vector indexes, bridging the gap between late-interaction models and conventional retrieval pipelines. Source: huggingface
LEMUR stands for Learned Multi-Vector Retrieval, and its implementation in txtai includes a feature encoder, output normalization statistics, and a sample of token vectors. During inference, a query is represented as the sum of its learned token features, while documents are transformed using ordinary least squares weights over stored samples. The fixed-vector inner product approximates the original late-interaction score, ensuring compatibility with existing retrieval systems. Training involves encoding corpus text, building standardized MaxSim targets, and learning a feature map to produce an inference artifact. The model's performance is corpus-specific and requires training before it can be used in txtai's embeddings index. Source: huggingface
The exact-search benchmark demonstrated LEMUR's effectiveness, showing improvements in NDCG scores across multiple datasets. At equal vector size, LEMUR improved NDCG by 56.6% on nfcorpus, 49.4% on scifact, and 61.9% on arguana. These results were achieved using a 2,048-dimensional LEMUR encoding, compared to MUVERA's 10,240-dimensional default. The LEMUR indexes also used one-fifth the storage of default MUVERA indexes, highlighting efficiency gains. However, the benchmark results are limited to one model, one machine, and exact search, emphasizing the need for further testing in varied environments. Source: huggingface
