Deepfake Detection using Spatiotemporal Convolutional Networks

View on arXiv ← Back to list

Authors: Oscar de Lima, Sean Franklin, Shreshtha Basu, Blake Karwoski, Annet George

Published: 2020-06-26 01:32:31+00:00

AI Summary

This paper explores the efficacy of spatiotemporal convolutional networks, commonly used in action recognition, for detecting AI-generated Deepfake videos. The authors created a benchmark using the Celeb-DF dataset, demonstrating that these methods can leverage temporal information to detect subtle artifacts. Their spatiotemporal approaches significantly outperformed state-of-the-art frame-based Deepfake detection methods.

Abstract

Better generative models and larger datasets have led to more realistic fake videos that can fool the human eye but produce temporal and spatial artifacts that deep learning approaches can detect. Most current Deepfake detection methods only use individual video frames and therefore fail to learn from temporal information. We created a benchmark of the performance of spatiotemporal convolutional methods using the Celeb-DF dataset. Our methods outperformed state-of-the-art frame-based detection methods. Code for our paper is publicly available at https://github.com/oidelima/Deepfake-Detection.

Key findings
Spatiotemporal convolutional methods achieved significantly higher ROC-AUC scores and accuracies (up to 99.73% ROC-AUC for R3D) compared to baseline frame-based methods on the challenging Celeb-DF dataset. The R3D network emerged as the top performer among the tested spatiotemporal models, outperforming even I3D. This indicates that temporal information is crucial for robust Deepfake detection, especially in datasets with fewer noticeable visual artifacts.
Approach
The authors propose using spatiotemporal convolutional networks, adapted from action recognition, to detect Deepfakes by analyzing both spatial and temporal inconsistencies in videos. They preprocess videos by face-cropping using RetinaFace and then feed sequences of frames into various 3D CNN architectures. These models are designed to capture motion features and temporal coherence across frames.
Datasets
Celeb-DF (v2), Kinetics (for pre-training), Charades (for I3D pre-training)
Model(s)
RCN, R3D, ResNet Mixed 3D-2D (MC3), ResNet (2+1)D, I3D
Author countries
USA
← Previous
Next →