CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation

View on arXiv ← Back to list

Authors: Yuxuan Du, Zhendong Wang, Yuhao Luo, Caiyong Piao, Zhiyuan Yan, Hao Li, Li Yuan

Published: 2025-05-21 08:11:07+00:00

AI Summary

This paper introduces CAD, a general multimodal framework for video deepfake detection that addresses the limitations of existing methods by harmoniously integrating modality-specific forensic traces and modality-shared semantic misalignments. CAD employs a dual-path architecture consisting of a Cross-modal Alignment module to detect high-level semantic inconsistencies (e.g., lip-speech mismatches) and a Cross-modal Distillation mechanism to preserve modality-specific forensic cues while mitigating feature conflicts during fusion. Experiments show that CAD significantly outperforms previous methods on various deepfake benchmarks, demonstrating its robustness and the necessity of a holistic multimodal approach.

Abstract

The rapid emergence of multimodal deepfakes (visual and auditory content are manipulated in concert) undermines the reliability of existing detectors that rely solely on modality-specific artifacts or cross-modal inconsistencies. In this work, we first demonstrate that modality-specific forensic traces (e.g., face-swap artifacts or spectral distortions) and modality-shared semantic misalignments (e.g., lip-speech asynchrony) offer complementary evidence, and that neglecting either aspect limits detection performance. Existing approaches either naively fuse modality-specific features without reconciling their conflicting characteristics or focus predominantly on semantic misalignment at the expense of modality-specific fine-grained artifact cues. To address these shortcomings, we propose a general multimodal framework for video deepfake detection via Cross-Modal Alignment and Distillation (CAD). CAD comprises two core components: 1) Cross-modal alignment that identifies inconsistencies in high-level semantic synchronization (e.g., lip-speech mismatches); 2) Cross-modal distillation that mitigates feature conflicts during fusion while preserving modality-specific forensic traces (e.g., spectral distortions in synthetic audio). Extensive experiments on both multimodal and unimodal (e.g., image-only/video-only)deepfake benchmarks demonstrate that CAD significantly outperforms previous methods, validating the necessity of harmonious integration of multimodal complementary information.

Key findings
CAD achieved state-of-the-art performance with 99.96% AUC on the IDForge dataset and 99.6% AUC on FakeAVCeleb, significantly outperforming existing unimodal and multimodal detectors. Ablation studies confirmed that both cross-modal alignment and cross-modal distillation are critical, as their removal or substitution led to noticeable performance drops. The model also demonstrated strong generalization capabilities in cross-manipulation evaluations against unseen forgery types.
Approach
The CAD framework utilizes a dual-path architecture comprising two core components: Cross-modal Alignment, which identifies inconsistencies in high-level semantic synchronization (e.g., lip-speech mismatches) by aligning audio-visual features using CLIP and Whisper with KL divergence, and Cross-modal Distillation, which preserves modality-specific forensic traces (e.g., spectral distortions in synthetic audio) using LoRA and SimSiam loss to enhance video representation learning and improve cross-modal consistency.
Datasets
FakeAVCeleb, IDForge-v2, FaceShifter, Celeb-DF
Model(s)
CLIP ViT-Base-16, Whisper-Small, LoRA (Low-Rank Adaptation), SimSiam, Cross-Attention, MLP
Author countries
China
← Previous
Next →