AMD has introduced a new optimization for video sparse attention (VSA) on its ROCm platform, aimed at improving the efficiency of video generation models. The optimization leverages Triton-based kernels and AMD’s kernel optimization agent, GEAK-V3, to enhance performance on AMD GPUs. According to AMD, the optimized VSA implementation achieves significant speedups in both training and inference tasks, offering a more efficient approach to handling the computational demands of video generation models. The optimization is part of AMD’s broader effort to support advanced AI research and deployment on its hardware platform.
The optimized VSA implementation delivers 1.37× and 1.33× speedups in the forward and backward passes, respectively, compared to the original VSA implementation. Additionally, it provides a 1.81× and 1.96× speedup over full attention in the same passes. The optimization also integrates with the Hummingbird-XT framework, achieving a 1.13× end-to-end training acceleration. Furthermore, the optimized VSA offers a 1.23× speedup for supervised fine-tuning of the Wan2.2-5B model. AMD also highlights that the optimization addresses training instability issues in joint step-distillation by introducing a lightweight linear-attention branch as an auxiliary source of global context, which significantly improves stability and visual quality.
The source discusses the challenges of self-attention in video generation models, where dense self-attention scales quadratically with token count, leading to increased computational costs. Video sparse attention mitigates this by approximating full attention with a subset of informative token interactions. However, the practical efficiency of VSA depends heavily on hardware-aware implementations, which AMD has now optimized for ROCm. The optimization combines heuristic kernel design with agent-based optimization, resulting in substantial performance gains. The source also notes that existing VSA implementations often rely on Triton and work in two stages, constructing a sparse block map and transforming it into index representations for efficient computation.
Source: amd
