From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection

Authors: Ke Liu, Jiwei Wei, Wenyu Zhang, Shuchang Zhou, Ruikun Chai, Yutao Dai, Chaoning Zhang, Yang Yang

Published: 2026-05-27 04:33:39+00:00

Comment: Accepted by ICML 2026

AI Summary

The paper introduces the Singing Head DeepFake (SHDF) dataset to address the performance degradation of audio-visual deepfake detection in singing scenarios due to rhythmic vocalization. It proposes T-AVFD, a Text-guided Audio-Visual Forgery Detection framework, which learns generalizable facial authenticity patterns via text-guided alignment and adaptively integrates them with cross-modal consistency through differential weighting. This enables robust detection across both talking and singing deepfakes.

Abstract

With rapid advances in audio-visual generative models, reliable forgery detection becomes increasingly critical. Existing methods for audio-visual deepfake detection typically rely on cross-modal inconsistencies. In singing, rhythmic vocalization weakens this coupling and introduces a nontrivial domain shift, substantially degrading detection performance. We construct the Singing Head DeepFake (SHDF) dataset using rhythm-aware generative models to fill the gap in singing benchmarks. To cope with cross-scenario domain shifts, we propose a Text-guided Audio-Visual Forgery Detection (T-AVFD) framework that generalizes across both talking and singing scenarios. T-AVFD comprises a facial authenticity pattern learner and a multi-modal differential weight learning module. The pattern learner aligns facial features with multi-granularity textual descriptions to learn generalizable authenticity patterns. The weight learning module preserves intrinsic audio-visual consistency and adaptively integrates it with authenticity patterns via differential weighting. Extensive experiments on multiple talking head deepfake datasets and SHDF show consistent improvements over existing baselines and strong robustness under diverse perturbations.

Key findings

T-AVFD consistently achieves superior performance over existing baselines on both talking (AVLips, FKAV, THB) and the newly introduced singing (SHDF) deepfake datasets, demonstrating strong cross-scenario generalization. The framework also exhibits significant robustness against diverse visual perturbations like blur, compression, and noise. This improvement is primarily attributed to its ability to learn generalized facial authenticity patterns and adaptively fuse multi-modal cues, mitigating the challenge of domain shift from talking to singing.

Approach

The T-AVFD framework employs a Facial Authenticity Pattern Learner (FAPL) that uses Alpha-CLIP to align facial features with multi-granularity textual descriptions, learning generalizable authenticity patterns. A Multi-Modal Differential Weight Learning (MMDWL) module then preserves audio-visual consistency using a pre-trained lip-reading expert and adaptively integrates these features with the authenticity patterns through content-conditioned weighting, generalizing across talking and singing scenarios.

Datasets

Singing Head DeepFake (SHDF), AVLips, FakeAVCeleb (FKAV), TalkingHeadBench (THB), LRS3

Model(s)

Alpha-CLIP, CLIP (text encoder), pre-trained lip-reading expert, MLP. Baselines: CViT, EfficientViT, RealForensics, LipFD, AVAD, AVH-Align.

Author countries

China

← Previous