Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Model

Authors: Yue-Hua Han, Tai-Ming Huang, Kai-Lung Hua, Jun-Cheng Chen

Published: 2024-04-08 14:58:52+00:00

AI Summary

This paper introduces a novel framework for generalized video-based Deepfake detection, adapting a vision-language foundation model (CLIP) with a side-network-based decoder. The approach incorporates Facial Component Guidance (FCG) to enhance spatial learning by focusing on key facial regions, alongside temporal modules for capturing inconsistencies. This method demonstrates improved generalizability, training data and parameter efficiency, and model robustness on challenging Deepfake datasets.

Abstract

Generative models have enabled the creation of highly realistic facial-synthetic images, raising significant concerns due to their potential for misuse. Despite rapid advancements in the field of deepfake detection, developing efficient approaches to leverage foundation models for improved generalizability to unseen forgery samples remains challenging. To address this challenge, we propose a novel side-network-based decoder that extracts spatial and temporal cues using the CLIP image encoder for generalized video-based Deepfake detection. Additionally, we introduce Facial Component Guidance (FCG) to enhance spatial learning generalizability by encouraging the model to focus on key facial regions. By leveraging the generic features of a vision-language foundation model, our approach demonstrates promising generalizability on challenging Deepfake datasets while also exhibiting superiority in training data efficiency, parameter efficiency, and model robustness.

Key findings

The framework significantly outperforms SOTA methods in cross-dataset evaluation, especially on challenging unseen datasets like CDF, DFDC, and WDF. It demonstrates superior parameter efficiency, being ~200 times smaller than comparable methods, and maintains robust performance even with limited training data. The Facial Component Guidance (FCG) is crucial for directing the model's attention to nuanced facial artifacts, contributing to enhanced generalizability and robustness against various perturbations.

Approach

The proposed method leverages a pre-trained ViT-based CLIP image encoder and adds a lightweight side-network-based decoder composed of spatial and temporal modules. The spatial module is guided by Facial Component Guidance (FCG) to focus on critical facial areas using InfoNCE loss, while the temporal module employs Patch-Temporal Multi-Head Self-Attention to detect inconsistencies over time. These components collectively extract spatio-temporal cues, which are then used by classification heads for Deepfake detection.

Datasets

FaceForensics++ (FF++), CelebDF-v2 (CDF), DeepFake Detection Challenge (DFDC), FaceShifter (FSh), DeeperForensics (DFo), WildDeepfake (WDF), HeyGen (AI Avatars), DiffusionForensics (CelebA-HQ subset)

Model(s)

ViT-based CLIP image encoder (specifically ViT-L/14), custom side-network-based decoder with spatial and temporal modules, Fully-Connected (FC) classification heads.

Author countries

Taiwan

← Previous