Amazon has introduced a new infrastructure for multi-turn reinforcement learning (RL) training, enabling the development of enterprise agents that handle multi-step workflows. This system, built on Amazon SageMaker HyperPod, allows for training complex agents that can query databases, call APIs, and recover from mid-process failures. The solution is designed to optimize over entire interaction sequences rather than individual responses, addressing limitations in standard reinforcement learning from human feedback (RLHF) approaches.
The multi-turn RL infrastructure for Amazon Nova on SageMaker HyperPod is a fully managed, serverless capability that provides compute, orchestration, and reward-routing layers for training agents on complex workflows. It includes a two-phase deployment model that separates long-lived foundational resources from ephemeral per-run resources, reducing costs and improving iteration speed. The solution is event-driven, starting training when data is uploaded to Amazon S3, and uses a pipeline that provisions compute, routes rewards, and runs multi-turn RL training automatically.
The infrastructure is built on three main components: a SageMaker HyperPod cluster for training and generation, an ECS on Fargate environment for reward workers, and the Nova Forge SDK for routing messages between the model and reward environment. The architecture is split into two phases, with the first phase involving a one-time AWS Cloud Development Kit (CDK) deployment for foundational resources, and the second phase spinning up ephemeral resources for each training run. The system is designed to keep GPU compute from sitting idle between runs and allows for iteration without full redeployment.
Source: awsml
