Shared-Private Cross-Modal Discrepancy Learning for Imbalanced Multi-Source Audio-Visual Deepfake Detection

Authors: Qiwu Wen, Jahangir Alam

Published: 2026-10-18 14:34:38+00:00

AI Summary

This paper addresses imbalanced multi-source audio-visual deepfake detection, a challenging scenario where training data is heterogeneous and unevenly distributed across sources. The proposed Pre-MSCT Shared-Private Feature Fusion method enhances existing detectors by routing audio-visual tokens into shared and source-specific private streams, followed by a domain-aware gated residual fusion. This approach effectively disentangles cross-source reusable evidence from source-dependent variations, leading to improved detection performance, particularly in terms of macro and worst-source AUC.

Abstract

Audio-visual deepfake detectors are often trained under a single dataset or manipulation regime, but multi-source training collections may contain heterogeneous attacks and uneven source distributions. Training separate detectors for each source is inefficient, while a single shared detector may be biased toward dominant source-specific patterns. We study known-source imbalanced multi-source audio-visual deepfake detection using FakeAVCeleb and LAV-DF, where source identity is supplied during training and inference. To address this setting, we build on the Multi-Scale Cross-Modal Transformer Encoder (MSCT) and propose Pre-MSCT Shared-Private Feature Fusion. Our method routes \\texttt{[CLS]}-augmented audio and visual tokens into a shared stream and source-specific private streams before the first MSCT block, while retaining its internal multi-scale self-attention (MSSA), differential cross-modal attention (DCA), and feed-forward operations. The shared stream models cross-source reusable evidence, while the private streams retain source-dependent residual variations. A domain-aware gated residual fusion module injects selected private evidence into the shared representation for the main fused prediction. Experiments under two controlled source-composition protocols show improved macro and worst-source AUC compared with the MRDF and MSCT baselines. These results support shared-private cross-modal discrepancy learning in the evaluated two-source setting with known source identity.

Key findings

The Pre-MSCT Shared-Private Feature Fusion method significantly improves macro AUC and worst-source AUC compared to MRDF and MSCT baselines under two controlled source-composition protocols (1:3 and 1:9 ratios). The gated residual fusion module and domain loss are crucial components, demonstrating that explicitly modeling shared and private characteristics for heterogeneous sources leads to more robust deepfake detection.

Approach

The method builds on the Multi-Scale Cross-Modal Transformer Encoder (MSCT) but introduces shared and source-specific private streams for audio and visual tokens before the first MSCT block. A shared stream captures reusable cross-source evidence, while private streams handle source-dependent variations. A domain-aware gated residual fusion module then selectively injects private evidence into the shared representation for the final prediction.

Datasets

FakeAVCeleb, LAV-DF

Model(s)

Multi-Scale Cross-Modal Transformer Encoder (MSCT), MRDF (Cross-Modality and Within-Modality Regularization for Audio-Visual DeepFake Detection)

Author countries

Canada

← Previous