What changed

AMD introduced ATOM, a native ROCm inference engine, to serve AgentX workloads on AMD Instinct MI355X GPUs. The framework is designed to handle agentic workloads, which involve long shared histories, short outputs, and frequent tool calls. On eight MI355X GPUs, ATOM serves Kimi-K3 at up to 17.5K total tokens per second per chip with 56 concurrent sessions, and achieves a P90 interactivity of 152 tokens per second per user on the SemiAnalysis InferenceX AgentX benchmark.

What this means in practice

The ATOM framework addresses challenges in serving agentic workloads by optimizing memory, scheduling, and compute. Memory optimizations include reusing history across GPU HBM and CPU DRAM (LMCache), compacting speculative state (ReplaySSM), and sharding MLA KV across GPUs (DCP). Scheduling optimizations involve delaying short prefills (PrefillDelayer) to keep active agents generating. Compute optimizations fuse decode steps into persistent kernels (Mono decode, written in FlyDSL). These techniques help reduce per-step overhead and improve efficiency for workloads with long histories and bursty sub-agent requests.

Limitations and availability

The supplied announcement does not specify the exact deployment settings or concurrency configurations for ATOM. The benchmark results are based on the SemiAnalysis InferenceX AgentX benchmark, which replays recorded coding sessions from the cc-traces-weka-062126 corpus. The framework is available for use with AMD Instinct MI355X GPUs and is part of AMD's ROCm software ecosystem. The effectiveness of ATOM may depend on specific workload characteristics and system configurations.

Source facts for AMD Introduces ATOM for Efficient AgentX Workloads on MI355X GPUs: Agentic workloads combine long shared histories and short outputs, interrupted by tool calls and parallel sub-agents. This post uses Kimi-K3, a hybrid model with multi-head latent attention (MLA) and Kimi Delta Attention (KDA) layers, as the case study. It runs on ATOM, AMD’s ROCm-native inference engine, and is measured with the AgentX benchmark.
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. amd: Keep the Agent Warm: How ATOM Serves AgentX Workloads on AMD Instinct MI355X GPUs