Hugging Face has introduced TAVR, a new method for generating talking avatars using short video references. Unlike traditional systems that rely on single images, TAVR uses video clips to create more accurate and consistent avatars. This approach allows for better identity preservation across different scenes and backgrounds. The model's performance was tested on a new cross-scene benchmark, where it achieved the highest overall quality score of 16.42, significantly outperforming existing methods.
TAVR's three-stage training strategy bridges the domain gap between reference and target scenes. The first stage involves same-scene pretraining, where the model learns to reproduce identity and motion from intra-scene video data. The second stage focuses on cross-scene fine-tuning, forcing the model to learn genuine identity aggregation. The third stage uses reinforcement learning with a task-specific DPO stage that rewards identity similarity using ArcFace metrics. This training approach ensures the model can generalize to new environments while maintaining identity fidelity.
Most talking avatar systems start from a single photograph, which captures only one moment in time. This limits the model's ability to accurately represent a person's identity across different scenes. TAVR changes this by using video references, which provide a richer signal of a person's appearance, expressions, and lighting conditions. This allows the model to create avatars that look and move more like the person, even in new backgrounds and scenes. Source: huggingface*
