Amazon Nova 2 introduces a new method called Self-Distilled Reasoning (SDR) to enhance supervised fine-tuning (SFT) for models lacking reasoning traces. This technique leverages the model’s own reasoning traces during training to improve performance without requiring a separate teacher model. SDR helps retain the base model’s capabilities while boosting target performance, particularly in math and coding tasks. According to Amazon, SDR mitigates catastrophic forgetting, a common issue in SFT where previous knowledge is lost. This approach is designed to work with existing SFT datasets across various domains, offering a more effective alternative to model merging. The method is part of a growing trend in self-distillation, which aims to improve model performance by distilling information from the model itself during training.
SDR reuses the chain of thought from the base Amazon Nova 2 Lite model as a substitute for non-reasoning datasets. This technique improves target performance while preserving general performance, as demonstrated by experiments on the LLaVA CoT dataset. The results showed that SDR can recover nearly all the base model’s math performance, which dropped to 6% after vanilla SFT. The approach also outperforms model merging in retaining both target and general performance, achieving a 6.5% improvement in target performance on average. These findings align with recent research emphasizing the effectiveness of self-distillation in mitigating catastrophic forgetting.
The reasoning suppression problem occurs when SFT is applied to datasets without reasoning traces, leading to a loss of reasoning ability during inference. This issue arises because the training objective penalizes tokens that don’t directly contribute to the final output, encouraging the model to bypass its reasoning mechanisms. The impact of reasoning on target performance was quantified through experiments on the LLaVA CoT dataset, showing significant improvements when reasoning is enabled during both training and inference. Model merging is currently recommended as a recovery mechanism, but SDR provides a more effective solution by preserving both target and general performance without requiring human annotation.
Source: awsml
