DepthFake: a depth-based strategy for detecting Deepfake videos

Authors: Luca Maiano, Lorenzo Papa, Ketbjano Vocaj, Irene Amerini

Published: 2022-08-23 16:38:25+00:00

Comment: 2022 ICPR Workshop on Artificial Intelligence for Multimedia Forensics and Disinformation Detection

Journal Ref: Springer 2022

AI Summary

This paper introduces DepthFake, a novel approach to deepfake video detection that enhances classical RGB-based methods by incorporating depth-maps. Depth information is extracted from RGB images using monocular depth estimation techniques, creating an RGBD input for robust pre-trained architectures. The proposed method demonstrates a significant improvement in detection performance on the FaceForensics++ dataset, achieving an average gain of 3.20% and up to 11.7% for specific deepfake attacks.

Abstract

Fake content has grown at an incredible rate over the past few years. The spread of social media and online platforms makes their dissemination on a large scale increasingly accessible by malicious actors. In parallel, due to the growing diffusion of fake image generation methods, many Deep Learning-based detection techniques have been proposed. Most of those methods rely on extracting salient features from RGB images to detect through a binary classifier if the image is fake or real. In this paper, we proposed DepthFake, a study on how to improve classical RGB-based approaches with depth-maps. The depth information is extracted from RGB images with recent monocular depth estimation techniques. Here, we demonstrate the effective contribution of depth-maps to the deepfake detection task on robust pre-trained architectures. The proposed RGBD approach is in fact able to achieve an average improvement of 3.20% and up to 11.7% for some deepfake attacks with respect to standard RGB architectures over the FaceForensic++ dataset.

Key findings

The DepthFake approach consistently improves deepfake detection performance compared to standard RGB-only methods, showing an average improvement of 3.20% and up to 11.7% for certain deepfake types on the FaceForensics++ dataset. The addition of depth features proved particularly effective for challenging deepfake classes like NeuralTexture. Furthermore, the study found that using grayscale plus depth (GrayD) inputs can achieve comparable or even superior results to RGBD, suggesting a strong contribution of depth over color artifacts, with minimal impact on inference times.

Approach

The proposed DepthFake method extracts depth information from RGB video frames using a pre-trained monocular depth estimation model (FaceDepth). This estimated depth map is then concatenated with the original RGB image to form an RGBD input. A convolutional neural network, primarily Xception, is then trained to classify these RGBD frames as real or fake, leveraging both color and spatial depth inconsistencies.

Datasets

FaceForensics++

Model(s)

Xception, ResNet50, MobileNet-V1 (for deepfake detection); FaceDepth model by Khan et al. [39] (for monocular depth estimation).

Author countries

Italy

← Previous