Generalized Deepfakes Detection with Reconstructed-Blended Images and Multi-scale Feature Reconstruction Network

Authors: Yuyang Sun, Huy H. Nguyen, Chun-Shien Lu, ZhiYong Zhang, Lu Sun, Isao Echizen

Published: 2023-12-13 09:49:15+00:00

AI Summary

This paper introduces a blended-based deepfake detection approach designed for robust applicability to unseen datasets. It combines a method for generating synthetic training samples, termed reconstructed blended images (RBIs), which incorporate potential deepfake generator artifacts, with a multi-scale feature reconstruction network (MFRN) for detecting generic boundary artifacts and noise distribution anomalies. Experiments show that this approach achieves superior performance in both cross-manipulation and cross-dataset detection on unseen data.

Abstract

The growing diversity of digital face manipulation techniques has led to an urgent need for a universal and robust detection technology to mitigate the risks posed by malicious forgeries. We present a blended-based detection approach that has robust applicability to unseen datasets. It combines a method for generating synthetic training samples, i.e., reconstructed blended images, that incorporate potential deepfake generator artifacts and a detection model, a multi-scale feature reconstruction network, for capturing the generic boundary artifacts and noise distribution anomalies brought about by digital face manipulations. Experiments demonstrated that this approach results in better performance in both cross-manipulation detection and cross-dataset detection on unseen data.

Key findings

The proposed method achieved superior performance in cross-manipulation detection on FF++ test sets, reaching 100% AUC for DF and F2F, and 98.90% for FSH. It also demonstrated robust cross-dataset detection, outperforming baselines on CDF-v2 (95.27% AUC), DFD (99.12%), and DFDC (73.31%), and was second-best on DFDC-P (83.66%). The approach showed strong resilience to moderate video compression but saw performance degradation under heavy compression.

Approach

The approach generates synthetic training samples called Reconstructed Blended Images (RBIs) by disentangling identity and background from genuine images, infusing random noise to simulate generator fingerprints, and then blending them with statistical transformations and a deformed mask. A Multi-scale Feature Reconstruction Network (MFRN) is then used, which employs separate EfficientNet-B4 backbones to capture RGB features, noise patterns, and edge features across multiple scales, integrating them via a feature fusion block to reconstruct manipulated boundaries and regions for classification.

Datasets

FaceForensics++ (FF++), Deepfakes (DF), Face2Face (F2F), FaceSwap (FS), NeuralTextures (NT), FaceShifter (FSH), DeepFakeDetection (DFD), Celeb-DF-v2 (CDF-v2), DeepFake Detection Challenge (DFDC), DFDC-P.

Model(s)

Multi-scale Feature Reconstruction Network (MFRN) with EfficientNet-B4 backbone, SimSwap (for RBI generation), Dlib (for landmark extraction), RetinaFace (for face detection), Bottleneck Attention Module (BAM).

Author countries

Japan, Taiwan, China

← Previous