Erik Kaunismäki has developed lattice-retrieval, a static embedding model trained on 660M curated query/document pairs. The model achieves an NDCG score of 0.4749 on decontaminated BEIR after fine-tuning, surpassing the 0.4334 score of sentence-transformers/static-retrieval-mrl-en-v1. The model is notable for its compact size and efficiency, embedding all 6.4M English Wikipedia articles in 7 minutes and 26 seconds on an 8-core Apple M2 MacBook Air.
The lattice-retrieval model operates using a simple architecture: a single learned lookup table followed by mean pooling. Unlike transformer-based models, it does not use attention mechanisms or contextualization, making it fast and compact. The model's simplicity allows for aggressive quantization, with the best quality/size trade-off found at int4-row with 512 dimensions, resulting in a 7.94 MB weight file that scores 0.4697, effectively matching the performance of fp32 at the same dimension.
The model's architecture is based on a 30,522 by 1,024 matrix, using the BERT-uncased vocabulary and Matryoshka training over dimensions [1024, 512, 256, 128, 64, 32]. The training process involved two stages: contrastive pre-training on the curated pairs and hard-negative fine-tuning. The model uses MultipleNegativesRankingLoss from Sentence Transformers. The training data was sourced from lightonai/embeddings-pre-training-curated, which includes 665M query/document pairs curated from 34 sources.
Source: huggingface
