Standing on the Shoulders of Giants: Reprogramming Visual-Language Model for General Deepfake Detection

Authors: Kaiqing Lin, Yuzhen Lin, Weixiang Li, Taiping Yao, Bin Li

Published: 2024-09-04 12:46:30+00:00

Comment: Accepted by AAAI 2025

AI Summary

This paper introduces RepDFD, a novel method for general deepfake detection that reprograms a pre-trained Vision-Language Model (VLM), such as CLIP. The approach manipulates the VLM's input through learnable visual perturbations and sample-level adaptive text prompts, leveraging face embeddings, without tuning the model's internal parameters. RepDFD significantly enhances cross-dataset and cross-manipulation deepfake detection performance while utilizing substantially fewer trainable parameters.

Abstract

The proliferation of deepfake faces poses huge potential negative impacts on our daily lives. Despite substantial advancements in deepfake detection over these years, the generalizability of existing methods against forgeries from unseen datasets or created by emerging generative models remains constrained. In this paper, inspired by the zero-shot advantages of Vision-Language Models (VLMs), we propose a novel approach that repurposes a well-trained VLM for general deepfake detection. Motivated by the model reprogramming paradigm that manipulates the model prediction via input perturbations, our method can reprogram a pre-trained VLM model (e.g., CLIP) solely based on manipulating its input without tuning the inner parameters. First, learnable visual perturbations are used to refine feature extraction for deepfake detection. Then, we exploit information of face embedding to create sample-level adaptative text prompts, improving the performance. Extensive experiments on several popular benchmark datasets demonstrate that (1) the cross-dataset and cross-manipulation performances of deepfake detection can be significantly and consistently improved (e.g., over 88\\% AUC in cross-dataset setting from FF++ to WildDeepfake); (2) the superior performances are achieved with fewer trainable parameters, making it a promising approach for real-world applications.

Key findings

The method significantly and consistently improved cross-dataset and cross-manipulation deepfake detection performances, achieving over 88% AUC in cross-dataset settings (e.g., FF++ to WildDeepfake). These superior results were obtained with remarkably fewer trainable parameters (0.078M) compared to state-of-the-art competitors (often >10M). The study also found that asymmetric text prompts, which incorporate face embeddings specifically for the fake class, were most effective.

Approach

The proposed RepDFD method reprograms a frozen pre-trained CLIP model by solely manipulating its input. It uses 'Input Transformation' to merge learnable visual perturbations (visual prompts) with input images for the image encoder and 'Face2Text Prompts' to generate sample-level adaptive text prompts, incorporating face embeddings, for the text encoder. This allows the model to adapt to deepfake detection without altering its core parameters.

Datasets

FaceForensics++ (FF++), Celeb-DF-v2 (CDF), DeepFake Detection Challenge preview (DFDCP), DeepFake Detection Challenge public (DFDC), WildDeepfake (Wild)

Model(s)

CLIP-ViT-L, CLIP-ViT-B (as foundation models), Transface (as face encoder), ArcFace, BlendFace (also tested as face encoders)

Author countries

China

← Previous