Teacher-Student Structure for Domain Adaptation in Ensemble Audio-Visual Video Deepfake Detection

Authors: Elham Abolhasani, Maryam Ramezani, Hamid R. Rabiee

Published: 2026-06-13 05:11:15+00:00

AI Summary

This paper proposes EAV-DFD, a generalized deep ensemble audio-visual model combined with a teacher-student framework for enhanced deepfake detection across unseen domains. The method aims to improve generalization ability and adapt to new deepfake generation methods by leveraging an ensemble of audio, visual, and audio-visual sub-networks, alongside a domain adaptation mechanism. It effectively transfers knowledge from a teacher model trained on a primary domain to a student model adapted to unseen data, even with limited new data.

Abstract

The rapid advancement of generative AI models is leading to more realistic deepfake media, encompassing the manipulation of audio, video, or both. This raises severe privacy and societal concerns. Numerous studies in this area have yielded promising intra-domain results; however, these models frequently exhibit decreased efficacy when faced with data from dissimilar domains. Consequently, recent deepfake detection approaches focus on enhancing the generalization ability through multiple techniques that incorporate all input modalities, including audio, images, and their interactions. In this regard, we propose the EAV-DFD method, a generalized deep ensemble audio-visual model (EAV-DFD) combined with a domain adaptation mechanism utilizing a teacher-student framework to enhance the model's ability to perform and generalize effectively across unseen domains. To evaluate the model's performance, we used the FakeAVCeleb dataset as the primary domain and the DFDC, Deepfake_TIMIT, and PolyGlotFake datasets as an unseen domain. Our experimental results demonstrate that the proposed framework is efficient in domain adaptation, improving AUC performance of the model by 4.09%, 17.94%, and 0.5% on three unseen datasets, using only a small portion of them to train the student model. This leads to a novel deepfake detection model capable of adapting to new domains and interpreting which modality has been manipulated, highlighting the potential of our approach for real-world applications.

Key findings

The proposed framework is efficient in domain adaptation, improving the AUC performance of the student model by 4.09% on DFDC, 17.94% on Deepfake TIMIT, and 0.5% on PolyGlotFake datasets compared to the teacher model. This adaptation is achieved using only a small portion of unseen data for student training, demonstrating the model's robustness and ability to adapt to new deepfake methods without significant performance loss on the primary domain.

Approach

The proposed EAV-DFD uses a deep ensemble architecture consisting of separate audio, visual, and audio-visual sub-networks, along with a decision-making module. It then employs a teacher-student framework for domain adaptation, where a teacher model (trained on a primary domain) transfers knowledge to a student model (adapted to unseen domains) using specialized loss functions, including binary cross-entropy, mean squared error, and KL divergence.

Datasets

FakeAVCeleb, DFDC, Deepfake_TIMIT, PolyGlotFake

Model(s)

CNN Encoder, Masked Transformer (HuBERT for audio), Xception (for visual), Cross-Attention Transformers (for audio-visual fusion), MLP Classifiers

Author countries

Iran

← Previous