AMD's case study highlights an unexpected outcome when enabling FP8 KV cache on a Kimi-K2.5-W4A8 model deployed on 8× AMD Instinct MI308X. While the MLA decode kernel ran 34% faster, the total GPU time for the same workload increased by 0.08%, indicating the new configuration introduced additional costs. The analysis reveals that the FP8 KV cache created two new GPU time categories that did not exist in the BF16 baseline, contributing to the overall increase. The challenge lies in understanding these new cost paths and ensuring optimizations target the correct areas.
According to AMD, the key to accurate profiling is combining paired trace diffs, semantic analysis, and source validation to identify and resolve performance bottlenecks. The case study underscores the importance of a comprehensive approach to performance analysis when adopting new technologies like FP8 KV cache. The findings suggest that while individual kernel improvements can be significant, they may not translate to overall system performance gains if new cost paths are introduced.
The report also emphasizes the role of LLMs in semantic analysis, but stresses the need for explicit rules and source-code validation to ensure accuracy. The insights from this case study provide a framework for understanding the trade-offs and complexities involved in optimizing AI workloads with FP8 KV cache. The results highlight the necessity of a multi-step analysis process to accurately interpret performance data and guide effective optimization strategies.
Source: amd
