Deepfake Style Transfer Mixture: a First Forensic Ballistics Study on Synthetic Images

Authors: Luca Guarnera, Oliver Giudice, Sebastiano Battiato

Published: 2022-03-18 13:11:54+00:00

Journal Ref: ICIAP 2022

AI Summary

This paper presents a preliminary study on forensic ballistics for deepfake images subjected to style-transfer manipulations. The core contribution is investigating the detection of how many times a digital image has been processed by a generative architecture for style transfer (once or twice). Additionally, the authors explore mathematical properties of style-transfer operations to better understand the manipulation process.

Abstract

Most recent style-transfer techniques based on generative architectures are able to obtain synthetic multimedia contents, or commonly called deepfakes, with almost no artifacts. Researchers already demonstrated that synthetic images contain patterns that can determine not only if it is a deepfake but also the generative architecture employed to create the image data itself. These traces can be exploited to study problems that have never been addressed in the context of deepfakes. To this aim, in this paper a first approach to investigate the image ballistics on deepfake images subject to style-transfer manipulations is proposed. Specifically, this paper describes a study on detecting how many times a digital image has been processed by a generative architecture for style transfer. Moreover, in order to address and study accurately forensic ballistics on deepfake images, some mathematical properties of style-transfer operations were investigated.

Key findings

The deep learning approach using ResNet-18 achieved the best classification accuracy of 92.75% for distinguishing between singly and doubly style-transferred images. The analytical method achieved a peak accuracy of 81% with the Random Forest classifier. The study also revealed that common mathematical properties (neutral element, commutativity, associativity) are generally not strictly satisfied by style-transfer operations, though associativity was approximately satisfied for RGB color distributions.

Approach

The authors propose two methods: an analytical approach that analyzes Discrete Cosine Transform (DCT) coefficients to extract discriminative frequency statistics, and a deep learning approach utilizing a ResNet-18 model. Both methods aim to classify whether a deepfake image has undergone one or two style-transfer operations.

Datasets

Images generated by StarGAN-V2, specifically a train set of 2400 images (1200 for single style transfer, 1200 for double style transfer) and a test set of 400 images (200 for each class).

Model(s)

For the analytical method: k-NN, SVM (linear, poly, rbf, sigmoid kernels), Linear Discriminant Analysis (LDA), Decision-Tree, Random Forest, GBoost. For the deep learning method: RESNET-18 (pre-trained on ImageNet).

Author countries

Italy

← Previous