DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusion

Authors: Ke Sun, Shen Chen, Taiping Yao, Hong Liu, Xiaoshuai Sun, Shouhong Ding, Rongrong Ji

Published: 2024-10-06 06:22:43+00:00

Comment: Accepted by NeurIPS 2024

AI Summary

The paper introduces DiffusionFake, a novel plug-and-play framework for deepfake detection that enhances generalization to unseen domains. It achieves this by reversing the deepfake generative process using a frozen Stable Diffusion model, which guides the detection network to learn disentangled source and target identity features. This approach significantly improves cross-domain generalization for various detector architectures without introducing additional inference-time parameters.

Abstract

The rapid progress of Deepfake technology has made face swapping highly realistic, raising concerns about the malicious use of fabricated facial content. Existing methods often struggle to generalize to unseen domains due to the diverse nature of facial manipulations. In this paper, we revisit the generation process and identify a universal principle: Deepfake images inherently contain information from both source and target identities, while genuine faces maintain a consistent identity. Building upon this insight, we introduce DiffusionFake, a novel plug-and-play framework that reverses the generative process of face forgeries to enhance the generalization of detection models. DiffusionFake achieves this by injecting the features extracted by the detection model into a frozen pre-trained Stable Diffusion model, compelling it to reconstruct the corresponding target and source images. This guided reconstruction process constrains the detection network to capture the source and target related features to facilitate the reconstruction, thereby learning rich and disentangled representations that are more resilient to unseen forgeries. Extensive experiments demonstrate that DiffusionFake significantly improves cross-domain generalization of various detector architectures without introducing additional parameters during inference. Our Codes are available in https://github.com/skJack/DiffusionFake.git.

Key findings

DiffusionFake significantly boosts the cross-domain generalization ability of various detector backbones, for instance, improving EfficientNet-B4's AUC on Celeb-DF by 11% and achieving an average 6% gain across datasets. The framework introduces no additional parameters during inference and demonstrates strong performance against recent diffusion-based forgeries (DiffSwap). It effectively disentangles source and target features and leads to more discriminative feature distributions for real vs. fake samples.

Approach

DiffusionFake enhances existing deepfake detection models by integrating a frozen pre-trained Stable Diffusion model. It injects features extracted by the detection model's encoder into Stable Diffusion, compelling it to reconstruct the corresponding source and target images. This guided reconstruction process constrains the detector to capture robust, disentangled source- and target-related features, improving generalization.

Datasets

FaceForensics++ (FF++), Celeb-DF, DeepFake Detection (DFD), DFDC Preview (DFDC-P), WildDeepfake, DiffSwap

Model(s)

EfficientNet-B4 (En-B4), ViT-B, ResNet-34, EfficientNet-B0, ViT-Small (as detector backbones), Stable Diffusion 1.5 (as the guiding generative model)

Author countries

China, Japan

← Previous