Qdrant and Hugging Face have released Qdrant-FineWeb-10B, a massive 10-billion vector dataset designed for benchmarking internet-scale vector search. This dataset is the result of a collaboration with Vultr, which provided the raw compute infrastructure and ran a custom generation pipeline to produce the initial embeddings. The dataset is part of a broader effort to address the limitations of current vector search benchmarks, which often fall short in scale and complexity. The release also includes the open-source Supernova framework, which enables the community to generate, manipulate, and benchmark large-scale datasets independently of proprietary tools.
The Qdrant-FineWeb-10B dataset comprises 24.47 TB of vectors and 28.66 TB of source text and metadata. It was created using Hugging Face’s FineWeb corpus and the gte-multilingual-base model. Supernova was used to compute exact top-1000 brute-force ground-truth nearest neighbors for 100,000 queries across the entire 10-billion vector space, involving over one quadrillion distance calculations run in parallel on GPU-accelerated hardware. This dataset and framework are intended to support the development and evaluation of vector search systems at an unprecedented scale.
The release also includes additional community datasets such as PubMed-Multi-Vector and Coyo-Vector-Embeddings, which are designed to benchmark hybrid retrieval methods and multimodal retrieval, respectively. These datasets are freely available on Hugging Face, and the community can expect more releases in the future. Supernova, the open-source benchmarking engine, is designed to automate the core phases of building and running vector search benchmarks, enabling scalable and reproducible experimentation.
Source: huggingface
