What changed

AMD has announced the release of InferaSim, a tool designed to cut the costs and time associated with trial and evaluation of large language model (LLM) serving configurations. The tool replaces extensive hardware-based measurements with fast projections, significantly reducing the need for GPU hours.

What this means in practice

InferaSim measures once per set of kernels and calculates every configuration that runs them, allowing for efficient evaluation of multiple serving setups. This approach reduces the GPU hours required for testing from approximately 204 to just 9.1 for 226 configurations. The tool provides insights into whether a recipe fits in HBM, the required concurrency for latency targets, and the optimal split between prefill and decode for different models and latency tiers. It also identifies the throughput-versus-interactivity Pareto frontier for models, both with and without speculative decoding and KV offload to host DRAM.

Limitations and availability

The tool is available for use with ROCm and requires installation on a machine with ROCm software. It is currently in preview and is intended for use in simulation mode, which does not require GPU resources for projections. The full method and evaluation are detailed in the InferaSim paper. The tool is designed to work with AMD Instinct™ MI355X GPUs and uses the Origami kernel models that build against ROCm. The source code is available on GitHub for developers to try and implement the tool in their workflows.

Source facts for AMD Introduces InferaSim to Cut LLM Serving Trial Costs: Software tools & optimizations AI/ML, LLM, Optimization, Performance AI Serving an LLM well means making a lot of choices before you know the answer. You pick an engine, a parallel layout, precisions and scheduler settings, and for disaggregated serving a split between prefill and decode. That leaves thousands of candidates, and each trial ties up one or more multi-GPU nodes for minutes.
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. amd: ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration