Generalizing Deepfake Video Detection with Plug-and-Play: Video-Level Blending and Spatiotemporal Adapter Tuning

Authors: Zhiyuan Yan, Yandan Zhao, Shen Chen, Mingyi Guo, Xinghe Fu, Taiping Yao, Shouhong Ding, Li Yuan

Published: 2024-08-30 07:49:57+00:00

AI Summary

This paper addresses generalization challenges in deepfake video detection by identifying and simulating a novel temporal artifact called Facial Feature Drift (FFD) using Video-level Blending (VB) data. It also proposes a lightweight Spatiotemporal Adapter (StA) that efficiently equips pre-trained image models with joint spatiotemporal reasoning capabilities for robust detection. The approach aims to learn more general forgery artifacts and enhance detection efficiency.

Abstract

Three key challenges hinder the development of current deepfake video detection: (1) Temporal features can be complex and diverse: how can we identify general temporal artifacts to enhance model generalization? (2) Spatiotemporal models often lean heavily on one type of artifact and ignore the other: how can we ensure balanced learning from both? (3) Videos are naturally resource-intensive: how can we tackle efficiency without compromising accuracy? This paper attempts to tackle the three challenges jointly. First, inspired by the notable generality of using image-level blending data for image forgery detection, we investigate whether and how video-level blending can be effective in video. We then perform a thorough analysis and identify a previously underexplored temporal forgery artifact: Facial Feature Drift (FFD), which commonly exists across different forgeries. To reproduce FFD, we then propose a novel Video-level Blending data (VB), where VB is implemented by blending the original image and its warped version frame-by-frame, serving as a hard negative sample to mine more general artifacts. Second, we carefully design a lightweight Spatiotemporal Adapter (StA) to equip a pretrained image model (both ViTs and CNNs) with the ability to capture both spatial and temporal features jointly and efficiently. StA is designed with two-stream 3D-Conv with varying kernel sizes, allowing it to process spatial and temporal features separately. Extensive experiments validate the effectiveness of the proposed methods; and show our approach can generalize well to previously unseen forgery videos, even the latest generation methods.

Key findings

The proposed method significantly outperforms prior image and video deepfake detectors across various cross-dataset and cross-manipulation generalization benchmarks, achieving an average AUC of 0.911 (Protocol-1) and 0.942 (Protocol-2). Both the Video-level Blending (VB) and the Spatiotemporal Adapter (StA) individually contribute to improved generalization, with the StA proving effective in providing spatiotemporal modeling abilities to existing image backbones with fewer parameters. The cross-attention module is identified as crucial for effectively fusing spatial and temporal information.

Approach

The method involves two main components: first, a novel Video-level Blending (VB) data generation strategy that simulates Facial Feature Drift (FFD) by perturbing facial landmarks frame-by-frame and blending warped images. Second, a lightweight, plug-and-play Spatiotemporal Adapter (StA) is designed with two-stream 3D-Convolutions (for spatial and temporal features) and a cross-attention layer to enable pre-trained image models (CNNs/ViTs) to jointly and efficiently capture spatiotemporal artifacts with minimal trainable parameters.

Datasets

FaceForensics++ (FF++), Celeb-DF-v2 (CDF-v2), DeepfakeDetection (DFD), Deepfake Detection Challenge (DFDC), DFDCP, DeeperForensics (DFo), WildDeepfake (WDF), FFIW, DF40.

Model(s)

CLIP ViT-l14 (default backbone), ResNet-34, CLIP ViT-b16, ViT-b16-21k, integrated with the proposed Spatiotemporal Adapter (StA) consisting of 3D-Conv layers and a cross-attention module.

Author countries

China

← Previous