What changed
AMD introduced ATOM, a native ROCm inference engine, to serve AgentX workloads on AMD Instinct MI355X GPUs. The framework is designed to handle agentic workloads, which involve long shared histories, short outputs, and frequent tool calls. On eight MI355X GPUs, ATOM serves Kimi-K3 at up to 17.5K total tokens per second per chip with 56 concurrent sessions, and achieves a P90 interactivity of 152 tokens per second per user on the SemiAnalysis InferenceX AgentX benchmark.
What this means in practice
The ATOM framework addresses challenges in serving agentic workloads by optimizing memory, scheduling, and compute. Memory optimizations include reusing history across GPU HBM and CPU DRAM (LMCache), compacting speculative state (ReplaySSM), and sharding MLA KV across GPUs (DCP). Scheduling optimizations involve delaying short prefills (PrefillDelayer) to keep active agents generating. Compute optimizations fuse decode steps into persistent kernels (Mono decode, written in FlyDSL). These techniques help reduce per-step overhead and improve efficiency for workloads with long histories and bursty sub-agent requests.
Limitations and availability
The supplied announcement does not specify the exact deployment settings or concurrency configurations for ATOM. The benchmark results are based on the SemiAnalysis InferenceX AgentX benchmark, which replays recorded coding sessions from the cc-traces-weka-062126 corpus. The framework is available for use with AMD Instinct MI355X GPUs and is part of AMD's ROCm software ecosystem. The effectiveness of ATOM may depend on specific workload characteristics and system configurations.
