DIP: Diffusion Learning of Inconsistency Pattern for General DeepFake Detection

Authors: Fan Nie, Jiangqun Ni, Jian Zhang, Bin Zhang, Weizhe Zhang

Published: 2024-10-31 06:26:00+00:00

Comment: 13 pages, accepted with IEEE Trans. on Multimedia

AI Summary

This paper introduces DIP (Diffusion Learning of Inconsistency Pattern), a transformer-based framework for general deepfake video detection. DIP exploits distinct directional inconsistency patterns (horizontal and vertical) in the motion information of forged videos. It employs a spatiotemporal encoder, a directional inconsistency decoder with direction-aware attention and inconsistency diffusion, and a SpatioTemporal Invariant Loss to enhance generalizability and prevent overfitting.

Abstract

With the advancement of deepfake generation techniques, the importance of deepfake detection in protecting multimedia content integrity has become increasingly obvious. Recently, temporal inconsistency clues have been explored to improve the generalizability of deepfake video detection. According to our observation, the temporal artifacts of forged videos in terms of motion information usually exhibits quite distinct inconsistency patterns along horizontal and vertical directions, which could be leveraged to improve the generalizability of detectors. In this paper, a transformer-based framework for Diffusion Learning of Inconsistency Pattern (DIP) is proposed, which exploits directional inconsistencies for deepfake video detection. Specifically, DIP begins with a spatiotemporal encoder to represent spatiotemporal information. A directional inconsistency decoder is adopted accordingly, where direction-aware attention and inconsistency diffusion are incorporated to explore potential inconsistency patterns and jointly learn the inherent relationships. In addition, the SpatioTemporal Invariant Loss (STI Loss) is introduced to contrast spatiotemporally augmented sample pairs and prevent the model from overfitting nonessential forgery artifacts. Extensive experiments on several public datasets demonstrate that our method could effectively identify directional forgery clues and achieve state-of-the-art performance.

Key findings

DIP achieved state-of-the-art performance, particularly demonstrating superior generalizability in cross-dataset and cross-manipulation evaluations, and robustness against various distortions. The directional inconsistency modeling via DiCA and IDM, alongside the SpatioTemporal Invariant Loss, effectively captures discriminative forgery clues. Ablation studies confirmed the effectiveness of each proposed module and the asymmetric spatiotemporal attention design.

Approach

The proposed DIP framework utilizes a SpatioTemporal Encoder (STE) to represent spatiotemporal information, followed by a Joint Directional Inconsistency Decoder (DID). The DID incorporates Directional Cross Attention (DiCA) for directional interaction and an Inconsistency Diffusion Module (IDM) to learn motion diffusion patterns. Additionally, a SpatioTemporal Invariant (STI) Loss, combined with spatiotemporal data augmentation, helps the model learn general forgery representations and improve robustness.

Datasets

FaceForensics++ (FF++), Celeb-DF-v2 (CDF), WildDeepFake (WDF), Deepfake Detection Challenge (DFDC-P, DFDC), DeeperForensics-1.0 (DFR)

Model(s)

UNKNOWN

Author countries

China

← Previous