Use of a Capsule Network to Detect Fake Images and Videos

Authors: Huy H. Nguyen, Junichi Yamagishi, Isao Echizen

Published: 2019-10-28 07:01:49+00:00

Comment: Fixing Table 2's scale

AI Summary

This paper introduces Capsule-Forensics, a capsule network-based method for detecting various types of attacks, from presentation attacks to deepfake videos, using significantly fewer parameters than traditional CNNs. The authors provide a detailed analysis and visualization to explain the theory behind applying capsule networks to digital media forensics. The proposed method demonstrates strong performance across different forgery detection tasks.

Abstract

The revolution in computer hardware, especially in graphics processing units and tensor processing units, has enabled significant advances in computer graphics and artificial intelligence algorithms. In addition to their many beneficial applications in daily life and business, computer-generated/manipulated images and videos can be used for malicious purposes that violate security systems, privacy, and social trust. The deepfake phenomenon and its variations enable a normal user to use his or her personal computer to easily create fake videos of anybody from a short real online video. Several countermeasures have been introduced to deal with attacks using such videos. However, most of them are targeted at certain domains and are ineffective when applied to other domains or new attacks. In this paper, we introduce a capsule network that can detect various kinds of attacks, from presentation attacks using printed images and replayed videos to attacks using fake videos created using deep learning. It uses many fewer parameters than traditional convolutional neural networks with similar performance. Moreover, we explain, for the first time ever in the literature, the theory behind the application of capsule networks to the forensics problem through detailed analysis and visualization.

Key findings

Capsule-Forensics achieved comparable or superior performance to state-of-the-art CNNs like XceptionNet on various fake image and video detection tasks, including deepfakes and presentation attacks, while using up to five times fewer parameters. The method demonstrated 100% accuracy on fully computer-generated image detection and presentation attack detection on specific datasets. Visualizations confirmed that primary capsules focus on critical regions like eyes, nose, and mouth, with dynamic routing consolidating these findings.

Approach

The Capsule-Forensics method preprocesses input images or video frames using a partial VGG-19 network as a feature extractor. These features are then fed into a capsule network architecture comprising several primary capsules and two output capsules ('real' and 'fake'). Dynamic routing, enhanced with random noise and dropout regularizations, calculates agreement between primary capsules to determine the final classification.

Datasets

FaceForensics++, dataset by Rahmouni et al. (for CGI vs PI), Idiap’s Replay-Attack database

Model(s)

Capsule Network (Capsule-Forensics), pre-trained VGG-19 (as feature extractor)

Author countries

Japan, UK

← Previous