DF-MoE: Generalizable Deepfake Detection via Multimodal Sparse Mixture-of-Experts

Authors: Vlad Hondru, Florinel Alin Croitoru, Iuliana Georgescu, A. Sophia Koepke, Radu Tudor Ionescu

Published: 2026-08-24 15:13:08+00:00

Comment: Accepted at BMVC 2026

AI Summary

This paper introduces DF-MoE, a novel audio-visual deepfake detection framework that improves generalization across various deepfake generation methods. It achieves this by extracting a wide array of high-level multimodal cues using frozen pre-trained models and integrating them via a sparse Mixture-of-Experts (MoE) backbone. The framework, combined with a contractive-repulsive objective (CRO) loss, demonstrates superior performance in both in-domain and cross-domain deepfake detection benchmarks.

Abstract

Audio-visual deepfake detection is an actively studied topic, where one of the main challenges is to develop detectors able to generalize across deepfake generation methods. We conjecture that overfitting can be mitigated by extracting multiple high-level cues from the available audio and visual modalities via pre-trained models. We therefore assemble a wide variety of pre-trained models to extract features that encode mouth movements, face parsing, facial expressions, head pose, gaze tracking, heart rate, audio emotion and speech activity. We further integrate both unimodal and multimodal cues via a Mixture-of-Experts (MoE) backbone to detect deepfakes. We perform in-domain and cross-domain experiments on five benchmarks for deepfake detection (MAVOS-DD, AVLips, PolyGlotFake, BioDeepAV, FakeAVCeleb) to compare our framework (DF-MoE) with state-of-the-art methods. Our results indicate that DF-MoE obtains superior deepfake detection results, surpassing all competing methods. We release our code at https://github.com/vladhondru25/DF-MoE.

Key findings

DF-MoE consistently outperforms state-of-the-art deepfake detectors across all five challenging benchmarks (MAVOS-DD, AVLips, PolyGlotFake, BioDeepAV, FakeAVCeleb) in both in-domain and cross-domain settings. Ablation studies confirm that combining diverse high-level features through the MoE architecture, along with the proposed CRO loss, significantly boosts generalization and interpretability, demonstrating the complementary value of each cue.

Approach

DF-MoE extracts multiple high-level semantic cues from audio and visual modalities using various pre-trained models (e.g., mouth movements, facial expressions, audio emotion, gaze tracking). These extracted features are then integrated and processed by a sparse Mixture-of-Experts (MoE) transformer. A novel Contractive-Repulsive Objective (CRO) loss is employed to structure the latent space, reducing intra-class variance while increasing inter-class separation, thereby enhancing generalization.

Datasets

MAVOS-DD, AVLips, PolyGlotFake, BioDeepAV, FakeAVCeleb, VoxCeleb2, CREMA-D, Celeb-DF (v2)

Model(s)

YOLOv11 (face detection), DeepSORT (face tracking), HopeNet (head pose), ResNet-34 (gaze estimation), Bilateral Segmentation Network (face segmentation), Whisper-Tiny (speech embeddings), EfficientNet-B0 (facial expression), custom MLP classifier (audio emotion), custom AV transformer (lip sync), AVFF, MoE (Sparse Mixture-of-Experts transformer)

Author countries

Romania, Germany

← Previous