Audio-Visual Synergy for High-Fidelity Portrait Animation: A Two-Stage Framework with Identity-Motion Disentanglement

Portrait animation technology has advanced rapidly in recent years. However, the quality of the animation degrades significantly when the identity in the source image mismatches that in the driving sequence. The primary cause of this degradation is the tight coupling between identity and motion in the learned representations. This entanglement causes the driving motion to carry its own identity features, which, during the feature fusion stage, contaminate the source identity and ultimately impair the quality of the generated sequence. This limitation has prompted the exploration of multi-modal driving. The core challenge is that each modality is inherently incomplete: visual features from frames capture structure but only short-term motion, whereas audio features provide a continuous temporal guide but no structural information. Consequently, any method based on a single modality is fundamentally constrained. To address this, a novel multi-modal portrait animation framework is proposed: it first learns joint and collaborative audio-visual representations via a self-supervised framework incorporating contrastive learning and masked data modeling; second, it employs a dedicated audio-visual motion estimation module to integrate these features and infer accurate facial motion. Experiments demonstrate that our model achieves the state-of-the-art performance on the VoxCeleb and VoxCeleb2 datasets. The code is available at https://anonymous.4open.science/r/AVS-for-Portrait-Animation-85F2.