On the Exploitation of Deepfake Model Recognition

Authors: Luca Guarnera, Oliver Giudice, Matthias Niessner, Sebastiano Battiato

Published: 2022-04-09 16:48:23+00:00

AI Summary

This paper introduces a robust processing pipeline for Deepfake Model Recognition, aiming to identify the specific GAN model that generated a deepfake image. By exploring the latent space of 50 slightly different StyleGAN2 models and training a suitable encoder, the method extracts analytical fingerprints from generated images. A dedicated metric is then introduced to discriminate between these models with high accuracy, even for models not used during training.

Abstract

Despite recent advances in Generative Adversarial Networks (GANs), with special focus to the Deepfake phenomenon there is no a clear understanding neither in terms of explainability nor of recognition of the involved models. In particular, the recognition of a specific GAN model that generated the deepfake image compared to many other possible models created by the same generative architecture (e.g. StyleGAN) is a task not yet completely addressed in the state-of-the-art. In this work, a robust processing pipeline to evaluate the possibility to point-out analytic fingerprints for Deepfake model recognition is presented. After exploiting the latent space of 50 slightly different models through an in-depth analysis on the generated images, a proper encoder was trained to discriminate among these models obtaining a classification accuracy of over 96%. Once demonstrated the possibility to discriminate extremely similar images, a dedicated metric exploiting the insights discovered in the latent space was introduced. By achieving a final accuracy of more than 94% for the Model Recognition task on images generated by models not employed in the training phase, this study takes an important step in countering the Deepfake phenomenon introducing a sort of signature in some sense similar to those employed in the multimedia forensics field (e.g. for camera source identification task, image ballistics task, etc).

Key findings

The trained encoder achieved an average classification accuracy of 96.24% on images from 50 known StyleGAN2 models. The proposed metric demonstrated strong generalization, obtaining over 94% accuracy for recognizing images from unknown StyleGAN2 models. The method also effectively distinguished between images from different generative architectures (StyleGAN1 vs. StyleGAN2) with nearly 100% accuracy, establishing a novel forensic signature for deepfake attribution.

Approach

The authors fine-tune 50 distinct StyleGAN2-ADA models to create a dataset of visually similar deepfake face images. A Resnet-18 encoder, pre-trained on ImageNet, is then trained to classify these known generative models based on their output images. Finally, a metric learning approach is employed on SVD-reduced features extracted from the encoder's first layer to define a robust metric for recognizing both known and unseen generative models.

Datasets

Custom dataset generated by 100 fine-tuned StyleGAN2-ADA models, FFHQ (for StyleGAN2-ADA training), ImageNet (for Resnet-18 pre-training), images generated by StyleGAN1 models (for comparison).

Model(s)

Resnet-18 (as the encoder for model recognition), StyleGAN2-ADA (generative architecture whose models are recognized), StyleGAN1 (used for generalization testing).

Author countries

Italy, Germany

← Previous