What changed
AWS has launched the Amazon SageMaker JumpStart Model Leaderboard, a new feature within SageMaker Studio that allows users to compare hundreds of open-weight foundation models side by side. The leaderboard provides rankings based on quality scores from three public benchmarks and performance and cost metrics measured using standardized test workloads.
What this means in practice
The leaderboard is accessible from the Models page in SageMaker Studio, next to the Categories and All models views. It uses the same search bar and filters as other views and opens on a leaderboard of large language models (LLMs). For each model, it displays quality scores on three benchmarks—BIG-Bench Hard, GPQA Diamond, and MMLU-Pro—and performance and cost metrics based on standardized test workloads. The leaderboard covers 70+ models from providers like Meta, Google, Qwen, IBM, and Arcee, with more to be added over time.
Limitations and availability
The leaderboard is available to users with access to SageMaker Studio and permission to deploy JumpStart models. It does not include data transfer, storage, or inference component costs, only the estimated cost of the SageMaker instance. The leaderboard does not specify which models are currently available for deployment, though users can filter by license, provider, and input/output modality. The leaderboard does not provide independent verification of benchmark scores, as all evaluations are conducted by AWS using the same setup for every model.
