Individualized Deepfake Detection Exploiting Traces Due to Double Neural-Network Operations

Authors: Mushfiqur Rahman, Runze Liu, Chau-Wai Wong, Huaiyu Dai

Published: 2023-12-13 10:21:00+00:00

AI Summary

This study addresses individualized deepfake detection for facial images of public figures, which is crucial for journalism. The authors propose a detector conditioned on individual identity and exploit the near-idempotency property of neural networks by applying double deepfake operations during training. This method significantly improves the Area Under the Curve (AUC) from 0.92 to 0.94 and reduces its standard deviation by 17% for Faceswap-GAN deepfakes, while also curating and releasing a new dataset of public figures.

Abstract

In today's digital landscape, journalists urgently require tools to verify the authenticity of facial images and videos depicting specific public figures before incorporating them into news stories. Existing deepfake detectors are not optimized for this detection task when an image is associated with a specific and identifiable individual. This study focuses on the deepfake detection of facial images of individual public figures. We propose to condition the proposed detector on the identity of an identified individual, given the advantages revealed by our theory-driven simulations. While most detectors in the literature rely on perceptible or imperceptible artifacts present in deepfake facial images, we demonstrate that the detection performance can be improved by exploiting the idempotency property of neural networks. In our approach, the training process involves double neural-network operations where we pass an authentic image through a deepfake simulating network twice. Experimental results show that the proposed method improves the area under the curve (AUC) from 0.92 to 0.94 and reduces its standard deviation by 17%. To address the need for evaluating detection performance for individual public figures, we curated and publicly released a dataset of ~32k images featuring 45 public figures, as existing deepfake datasets do not meet this criterion.

Key findings

The proposed method achieved an improved AUC of 0.94 (from 0.92) and a 17% reduction in AUC standard deviation against Faceswap-GAN deepfakes, demonstrating better reliability and consistency across identities. It also achieved an AUC of 1.00 against DiffSwap deepfakes. Experimental verification confirmed the near-idempotence property of deepfake generators and highlighted the synergistic benefits of combining identity conditioning with the idempotence strategy.

Approach

The approach conditions the deepfake detector on the identity of the individual and leverages the near-idempotency property of deepfake generation networks. During training, an authentic image is passed through a deepfake simulating network (reconstruction operator) twice. A Siamese network then learns to differentiate between the significant changes from a first deepfake operation on an authentic image and the minimal changes from a second operation on an already deepfaked image, enhanced by an identity-aware feature extractor and identity decoder.

Datasets

Curated dataset (publicly released, ~32k images of 45 public figures), Celeb-DF [63], Cross-Age Celebrity Dataset (CACD) [64], Faceswap-GAN [1] (for deepfake generation), DiffSwap [2] (for deepfake generation). Mentioned for context/comparison: DFDC [60], DFD [61], Deeper Forensics [62], FaceForensics++ [66].

Model(s)

The proposed method employs a reconstruction operator (an autoencoder utilizing the encoder/decoder architecture from FaceSwap-GAN), an identity-aware feature extractor (a fine-tuned ResNet architecture pre-trained by Bulat et al. [50], with EfficientNetAutoAttB4ST [37] as a teacher network), an identity decoder (a single-layer fully-connected neural network), and a Siamese network (two identical single-layer neural networks). Baselines for comparison included Xception [35] and EfficientNetAutoAttB4ST [37].

Author countries

USA

← Previous