HuggingFace has released an updated version of the MTEB leaderboard, which is substantially faster and offers enhanced filtering and transparency. The new version improves the user experience by providing more detailed insights into model performance and enabling better customization. The update addresses previous limitations in speed and reliability, making the leaderboard more accessible and efficient for users.
The new leaderboard is built on a more reliable and scalable framework using FastAPI and Svelte, allowing for faster performance and a richer set of features. Users can now filter benchmarks by domain, language, modality, and individual tasks, making it easier to find models that match their specific needs. Additionally, the leaderboard includes annotations that indicate whether a model has been trained on a task's training set or is encountering it for the first time, improving transparency and trust in the evaluation process.
The update also focuses on encouraging broader improvements across the model frontier rather than just the top-performing models. Quick views of the front page now display the top models for their size bracket, and performance-by-runtime analytics are provided to help users make informed decisions. The leaderboard now allows users to compare models directly, with pinned models reordered and highlighted for easy comparison. Users can also access the leaderboard's data through a downloadable CSV or via an API, ensuring flexibility and accessibility.
Source: huggingface
