What changed

AWS has launched the Amazon SageMaker JumpStart Model Leaderboard, a new feature within SageMaker Studio that allows users to compare hundreds of open-weight foundation models side by side. The leaderboard provides rankings based on quality scores from three public benchmarks and performance and cost metrics measured using standardized test workloads.

What this means in practice

The leaderboard is accessible from the Models page in SageMaker Studio, next to the Categories and All models views. It uses the same search bar and filters as other views and opens on a leaderboard of large language models (LLMs). For each model, it displays quality scores on three benchmarks—BIG-Bench Hard, GPQA Diamond, and MMLU-Pro—and performance and cost metrics based on standardized test workloads. The leaderboard covers 70+ models from providers like Meta, Google, Qwen, IBM, and Arcee, with more to be added over time.

Limitations and availability

The leaderboard is available to users with access to SageMaker Studio and permission to deploy JumpStart models. It does not include data transfer, storage, or inference component costs, only the estimated cost of the SageMaker instance. The leaderboard does not specify which models are currently available for deployment, though users can filter by license, provider, and input/output modality. The leaderboard does not provide independent verification of benchmark scores, as all evaluations are conducted by AWS using the same setup for every model.

Source facts for AWS Launches SageMaker JumpStart Model Leaderboard for Model Comparison: That choice is powerful, but it also creates a real question: Which model should you start with? Then you still must guess how each model will perform, and what it will cost, for your use case. What is the JumpStart Model Leaderboard?
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. awsml: Introducing the Amazon SageMaker JumpStart Model Leaderboard