Amazon has introduced Amazon Nova, a new customizable content moderation tool designed to address the challenge of over-deflection in foundation models. Organizations using these models often face issues where content moderation safeguards block legitimate business-critical use cases. For instance, a security team might be denied access to generate a sample phishing email for training purposes, despite the intent being defensive. The solution involves a novel unlearning technique called Reverse Direct Preference Optimization (rDPO), which allows for selective adjustment of model behavior without full retraining. This approach enables models to process approved policy areas while maintaining alignment with responsible AI principles elsewhere. Source: awsml
Amazon Nova Customizable Content Moderation Settings (CCMS) enables approved customers to selectively adjust safeguards across four responsible AI (RAI) pillars: Safety, Sensitive Content, Fairness, and Security. The tool enforces essential, non-configurable controls for responsible AI use, such as protections against harm to children and privacy violations. The science behind CCMS is unlearning, a technique for selectively removing learned behaviors from a model’s parameters without retraining from scratch. By training Low-Rank Adaptation (LoRA) adapters, Amazon allows for the creation of custom model variants that generate content in customer-approved policy areas while remaining aligned with broader responsible AI principles. Source: awsml
The development of rDPO addresses the key scientific challenge of unlearning: stopping content deflection in targeted policy areas while preserving general model capabilities like instruction following, coding, and math. Direct fine-tuning approaches risk degrading model quality, and while Negative Preference Optimization (NPO) has been used to teach models to forget, it often results in degraded output quality. rDPO, by reversing the preference pair in the DPO objective, guides the model to not only forget but also generate high-quality responses in the unlearned policy areas. This dual objective improves training efficiency and response quality. Source: awsml
