Self-supervised Transformer for Deepfake Detection

Authors: Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Weiming Zhang, Nenghai Yu

Published: 2022-03-02 17:44:40+00:00

AI Summary

This paper proposes a self-supervised transformer-based audio-visual contrastive learning method to enhance deepfake detection's generalization and robustness. The approach learns robust mouth motion representations by encouraging paired video and audio features to be close while pushing unpaired ones apart. After self-supervised pre-training, the model is partially fine-tuned for deepfake detection, demonstrating comparable or superior performance to supervised pre-training, particularly in generalization and robustness to post-processing.

Abstract

The fast evolution and widespread of deepfake techniques in real-world scenarios require stronger generalization abilities of face forgery detectors. Some works capture the features that are unrelated to method-specific artifacts, such as clues of blending boundary, accumulated up-sampling, to strengthen the generalization ability. However, the effectiveness of these methods can be easily corrupted by post-processing operations such as compression. Inspired by transfer learning, neural networks pre-trained on other large-scale face-related tasks may provide useful features for deepfake detection. For example, lip movement has been proved to be a kind of robust and good-transferring highlevel semantic feature, which can be learned from the lipreading task. However, the existing method pre-trains the lip feature extraction model in a supervised manner, which requires plenty of human resources in data annotation and increases the difficulty of obtaining training data. In this paper, we propose a self-supervised transformer based audio-visual contrastive learning method. The proposed method learns mouth motion representations by encouraging the paired video and audio representations to be close while unpaired ones to be diverse. After pre-training with our method, the model will then be partially fine-tuned for deepfake detection task. Extensive experiments show that our self-supervised method performs comparably or even better than the supervised pre-training counterpart.

Key findings

The self-supervised method achieves comparable or better performance than its supervised pre-training counterpart, especially exhibiting superior robustness to common video corruptions and better generalization across unseen forgery datasets and manipulation methods. Performance scales positively with the amount of pre-training data, highlighting the advantage of self-supervised learning in leveraging large unlabeled datasets. The transformer-based architecture for the video encoder backend significantly outperforms MSTCN in generalization capabilities.

Approach

The method employs a two-stage process. First, it uses self-supervised audio-visual contrastive learning to pre-train a video encoder (composed of 3D convolution, 2D ResNet, and a temporal Transformer) and an audio encoder (wav2vec2 backbone with Transformer). It learns generic lip motion representations by maximizing agreement between corresponding audio and visual features and disagreement for non-corresponding pairs. Second, the pre-trained video encoder's frontend is frozen, and a new MLP head is added to the backend (Transformer) to fine-tune the network for binary deepfake classification using a binary cross-entropy loss.

Datasets

VoxCeleb2, AV Speech, FaceForensics++ (FF++), Celeb-DF v2, DeeperForensics, DFDC (DeepFake Detection Challenge), FaceShifter

Model(s)

wav2vec2 (for audio encoder), 3D Convolutional layer, 2D ResNet (ResNet18, ResNet50), Temporal 1D Transformer, MLP head

Author countries

China, China, UNKNOWN, China, China

← Previous