AMD has introduced ROCm Infera, a distributed inference reference solution designed for large-scale deployments. The tool aims to address orchestration challenges in agentic AI systems by improving goodput per GPU for realistic workloads. According to AMD, initial internal testing shows that Infera can enhance goodput by up to 2.6×, as detailed in the company’s blog post. The solution is open source from day one and is available on GitHub at github.com/AMD-AGI/Infera.
Infera tackles orchestration gaps by coordinating engine instances across three dimensions: KV-aware routing, prefill-decode disaggregation, and KV-cache tiering. This orchestration increases inference goodput, which is crucial for end-to-end agentic AI performance. The tool works with existing inference engines like vLLM, SGLang, and AMD's ROCm-native ATOM engine, allowing them to evolve independently while preserving current tooling and optimizations.
The source explains that agentic systems run long, multi-turn loops that reuse large prefixes, leading to challenges in KV-cache management. Infera addresses these by managing request placement, execution phases, and reusable KV state across the deployment. This approach increases KV-cache reuse, reduces redundant prefill computation, and lowers time to first token for requests with repeated prefixes, directly improving responsiveness and system efficiency.
Source: amd
