AMD has introduced Verl, a framework designed to scale reinforcement learning (RL) on its Instinct MI355X GPUs. The framework supports both synchronous and asynchronous modes, allowing for greater flexibility in training large language models (LLMs). Verl enables asynchronous training by decoupling the rollout and training phases, which helps reduce long-tail stalls and improves GPU utilization. The framework also integrates with ROCm and supports multiple training strategies, including Group Relative Policy Optimization (GRPO) and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO).
The asynchronous mode of Verl allows for overlapping training and rollout processes, which is particularly beneficial for tasks with long and variable response times, such as vision-math or multi-turn reasoning. This approach helps keep training GPUs busy and reduces idle time caused by slow samples. In contrast, the synchronous mode ensures minimal policy-version lag and simpler debugging, but it may lead to inefficiencies due to batch barriers. Verl's asynchronous framework also supports resource isolation, parallel generation and training, and flexible synchronization options through the ROCm Collective Communication Library (RCCL).
The blog provides a hands-on walkthrough of running Verl's asynchronous examples on AMD Instinct MI355X GPUs. It includes end-to-end training of GRPO on Qwen2.5-VL-7B-Instruct with the Geometry3k vision-math dataset and DAPO on Qwen2.5-Math-7B with the DAPO-Math-17k and AIME-2024 datasets. The framework also presents throughput comparisons between MI355X and NVIDIA B300 for synchronous use cases. These studies highlight the performance and scalability of Verl on AMD hardware.
Source: amd
