Real Face Foundation Representation Learning for Generalized Deepfake Detection

View on arXiv ← Back to list

Authors: Liang Shi, Jie Zhang, Shiguang Shan

Published: 2023-03-15 08:27:56+00:00

AI Summary

This paper introduces Real Face Foundation Representation Learning (RFFR), a novel approach to deepfake detection that learns the distribution of real faces using masked image modeling (MIM). By identifying discrepancies between input and reconstructed fake images, RFFR enables the detection of artifacts indicative of deepfakes, achieving superior generalization performance compared to state-of-the-art methods.

Abstract

The emergence of deepfake technologies has become a matter of social concern as they pose threats to individual privacy and public security. It is now of great significance to develop reliable deepfake detectors. However, with numerous face manipulation algorithms present, it is almost impossible to collect sufficient representative fake faces, and it is hard for existing detectors to generalize to all types of manipulation. Therefore, we turn to learn the distribution of real faces, and indirectly identify fake images that deviate from the real face distribution. In this study, we propose Real Face Foundation Representation Learning (RFFR), which aims to learn a general representation from large-scale real face datasets and detect potential artifacts outside the distribution of RFFR. Specifically, we train a model on real face datasets by masked image modeling (MIM), which results in a discrepancy between input faces and the reconstructed ones when applying the model on fake samples. This discrepancy reveals the low-level artifacts not contained in RFFR, making it easier to build a deepfake detector sensitive to all kinds of potential artifacts outside the distribution of RFFR. Extensive experiments demonstrate that our method brings about better generalization performance, as it significantly outperforms the state-of-the-art methods in cross-manipulation evaluations, and has the potential to further improve by introducing extra real faces for training RFFR.

Key findings

RFFR significantly outperforms state-of-the-art methods in cross-manipulation and cross-dataset evaluations. The method demonstrates better generalization and resistance to overfitting, even without validation data. Adding more real face data for training RFFR further improves detection performance.

Approach

RFFR trains a model on real face images using masked image modeling (MIM) to learn a representation of real faces. When applied to fake images, the model produces discrepancies between the input and reconstructed images, revealing artifacts. These discrepancies, along with the original image blocks, are used to train a deepfake detector.

Datasets

Faceforensics++ (FF), Celeb-DF, DFDC, ForgeryNet (FN)

Model(s)

Masked Autoencoder (MAE) for representation learning, Vision Transformer (ViT) for deepfake detection

Author countries

China

← Previous