FakeTransformer: Exposing Face Forgery From Spatial-Temporal Representation Modeled By Facial Pixel Variations

Authors: Yuyang Sun, Zhiyong Zhang, Changzhen Qiu, Liang Wang, Zekai Wang

Published: 2021-11-15 08:44:52+00:00

AI Summary

The FakeTransformer proposes a novel approach to detect face forgeries by exploiting inconsistencies in physiological signals from facial pixel variations. It utilizes Multi-Scale Eulerian Magnified Spatial-Temporal maps (MEMSTmaps) to enhance these subtle variations, which are then processed by a Vision Transformer to learn robust spatio-temporal descriptors for classification. The method demonstrates excellent performance in forgery detection and strong generalization capabilities across different datasets.

Abstract

With the rapid development of generation model, AI-based face manipulation technology, which called DeepFakes, has become more and more realistic. This means of face forgery can attack any target, which poses a new threat to personal privacy and property security. Moreover, the misuse of synthetic video shows potential dangers in many areas, such as identity harassment, pornography and news rumors. Inspired by the fact that the spatial coherence and temporal consistency of physiological signal are destroyed in the generated content, we attempt to find inconsistent patterns that can distinguish between real videos and synthetic videos from the variations of facial pixels, which are highly related to physiological information. Our approach first applies Eulerian Video Magnification (EVM) at multiple Gaussian scales to the original video to enlarge the physiological variations caused by the change of facial blood volume, and then transform the original video and magnified videos into a Multi-Scale Eulerian Magnified Spatial-Temporal map (MEMSTmap), which can represent time-varying physiological enhancement sequences on different octaves. Then, these maps are reshaped into frame patches in column units and sent to the vision Transformer to learn the spatio-time descriptors of frame levels. Finally, we sort out the feature embedding and output the probability of judging whether the video is real or fake. We validate our method on the FaceForensics++ and DeepFake Detection datasets. The results show that our model achieves excellent performance in forgery detection, and also show outstanding generalization capability in cross-data domain.

Key findings

The FakeTransformer achieves excellent forgery detection performance, matching or surpassing baselines on FaceForensics++ sub-datasets, including 98.98% accuracy on DeepFakes. It also demonstrates superior cross-data domain generalization, achieving 68% accuracy on the DeepFake Detection dataset. Furthermore, the model exhibits better robustness against video compression compared to other physiological information-based methods.

Approach

The method first applies Multi-Scale Eulerian Video Magnification (EVM) to the input video to amplify subtle physiological variations in facial pixels. These original and magnified video sequences are then transformed into Multi-Scale Eulerian Magnified Spatial-Temporal maps (MEMSTmaps). Finally, these maps are reshaped into column-unit frame patches and fed into a Vision Transformer for learning spatio-temporal features to classify videos as real or fake.

Datasets

FaceForensics++, DeepFake Detection dataset

Model(s)

Vision Transformer (ViT)

Author countries

China

← Previous