Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Generated Images

Authors: Vishal Asnani, Xi Yin, Tal Hassner, Xiaoming Liu

Published: 2021-06-15 04:19:26+00:00

Comment: Accepted at Transactions on Pattern Analysis and Machine Intelligence

AI Summary

This paper introduces "model parsing," a novel problem of inferring generative model hyperparameters (network architecture and training loss functions) directly from their generated images. It proposes a framework comprising a Fingerprint Estimation Network (FEN) to extract subtle, model-specific patterns and a Parsing Network (PN) to predict these hyperparameters. The estimated fingerprints also demonstrate state-of-the-art performance in deepfake detection and image attribution.

Abstract

State-of-the-art (SOTA) Generative Models (GMs) can synthesize photo-realistic images that are hard for humans to distinguish from genuine photos. Identifying and understanding manipulated media are crucial to mitigate the social concerns on the potential misuse of GMs. We propose to perform reverse engineering of GMs to infer model hyperparameters from the images generated by these models. We define a novel problem, ``model parsing, as estimating GM network architectures and training loss functions by examining their generated images -- a task seemingly impossible for human beings. To tackle this problem, we propose a framework with two components: a Fingerprint Estimation Network (FEN), which estimates a GM fingerprint from a generated image by training with four constraints to encourage the fingerprint to have desired properties, and a Parsing Network (PN), which predicts network architecture and loss functions from the estimated fingerprints. To evaluate our approach, we collect a fake image dataset with $100$K images generated by $116$ different GMs. Extensive experiments show encouraging results in parsing the hyperparameters of the unseen models. Finally, our fingerprint estimation can be leveraged for deepfake detection and image attribution, as we show by reporting SOTA results on both the deepfake detection (Celeb-DF) and image attribution benchmarks.

Key findings

The proposed model parsing framework significantly outperformed baseline methods in inferring GM hyperparameters, revealing a strong and generalized correlation between generated images and GM design characteristics. The estimated fingerprints achieved state-of-the-art results in deepfake detection on Celeb-DF and image attribution on CelebA/LSUN datasets, while also showing effectiveness in detecting coordinated misinformation attacks with approximately 80% accuracy. The method also proved effective for unseen diffusion models.

Approach

The approach utilizes a two-component framework: a Fingerprint Estimation Network (FEN) extracts subtle, unique fingerprints from generated images, guided by four constraints (magnitude, spectrum, repetitive, and energy loss). A subsequent Parsing Network (PN) then predicts the generative model's network architecture and training loss functions from these estimated fingerprints, employing a hierarchical clustering-based strategy with both cluster-level and instance-level parsers.

Datasets

A custom dataset of 116,000 images generated by 116 different Generative Models (GMs) was collected for model parsing. For deepfake detection, the Celeb-DF dataset was used. For image attribution, CelebA and LSUN datasets were utilized. The approach was also evaluated on images generated by 7 diffusion models.

Model(s)

The core architecture consists of a Fingerprint Estimation Network (FEN), designed with a CNN structure similar to DnCNN, and a Parsing Network (PN). The PN integrates k-means clustering with two specialized parsers (cluster parser and instance parser) for hierarchical hyperparameter prediction. Additionally, shallow networks are used for downstream tasks like deepfake detection and image attribution.

Author countries

USA

← Previous