Voice-Face Homogeneity Tells Deepfake

Authors: Harry Cheng, Yangyang Guo, Tianyi Wang, Qi Li, Xiaojun Chang, Liqiang Nie

Published: 2022-03-04 09:08:50+00:00

Comment: 13 pages for peer review. Code will be released at https://github.com/xaCheng1996/VFD

AI Summary

This paper introduces Voice-Face matching Detection (VFD), a novel deepfake detection method that exploits the empirical observation of identity mismatch between voices and faces in fake videos. VFD measures the homogeneity between these two modalities using a pre-training then fine-tuning paradigm to enhance generalizability. The approach involves pre-training on generic audio-visual data and subsequently fine-tuning on deepfake datasets to learn cross-modal consistency.

Abstract

Detecting forgery videos is highly desirable due to the abuse of deepfake. Existing detection approaches contribute to exploring the specific artifacts in deepfake videos and fit well on certain data. However, the growing technique on these artifacts keeps challenging the robustness of traditional deepfake detectors. As a result, the development of generalizability of these approaches has reached a blockage. To address this issue, given the empirical results that the identities behind voices and faces are often mismatched in deepfake videos, and the voices and faces have homogeneity to some extent, in this paper, we propose to perform the deepfake detection from an unexplored voice-face matching view. To this end, a voice-face matching method is devised to measure the matching degree of these two. Nevertheless, training on specific deepfake datasets makes the model overfit certain traits of deepfake algorithms. We instead, advocate a method that quickly adapts to untapped forgery, with a pre-training then fine-tuning paradigm. Specifically, we first pre-train the model on a generic audio-visual dataset, followed by the fine-tuning on downstream deepfake data. We conduct extensive experiments over three widely exploited deepfake datasets - DFDC, FakeAVCeleb, and DeepfakeTIMIT. Our method obtains significant performance gains as compared to other state-of-the-art competitors. It is also worth noting that our method already achieves competitive results when fine-tuned on limited deepfake data.

Key findings

The proposed VFD method achieves state-of-the-art performance across DFDC, FakeAVCeleb, and DeepfakeTIMIT datasets, with AUCs around 85% on challenging datasets where baselines often perform poorly. It demonstrates high generalizability by focusing on global identity information rather than specific regional artifacts. Notably, VFD maintains competitive detection capabilities even when fine-tuned on limited deepfake data, indicating its efficiency and adaptability to new forgery algorithms.

Approach

The VFD method employs a dual-stream network with transformer-based feature extractors for voices and faces, guided by modality modulators. It is initially pre-trained on a generic audio-visual dataset using InfoNCE loss to learn identity-associated voice-face homogeneity. The model is then fine-tuned on deepfake datasets using an upgraded Real-Fake Contrastive (RFC) loss to discriminate between real (matched) and fake (mismatched) audio-visual pairs.

Datasets

Voxceleb2 (pre-training), DFDC, FakeAVCeleb, DeepfakeTIMIT (LQ, HQ)

Model(s)

Transformer-based multi-modal feature extractors, with 12 transformer blocks and 12-head self-attention.

Author countries

China, Singapore, Australia

← Previous