Fusing Global and Local Features for Generalized AI-Synthesized Image Detection

Authors: Yan Ju, Shan Jia, Lipeng Ke, Hongfei Xue, Koki Nagano, Siwei Lyu

Published: 2022-03-26 01:55:37+00:00

Comment: 5 pages, 3 figures, 2 tables

AI Summary

This paper introduces a two-branch model designed to improve the generalization ability of AI-synthesized image detection by fusing global and local features. It combines whole-image spatial information with informative patches selected by a novel patch selection module. A multi-head attention mechanism is utilized to effectively fuse these complementary features, achieving high accuracy and good generalization on a highly diverse dataset.

Abstract

With the development of the Generative Adversarial Networks (GANs) and DeepFakes, AI-synthesized images are now of such high quality that humans can hardly distinguish them from real images. It is imperative for media forensics to develop detectors to expose them accurately. Existing detection methods have shown high performance in generated images detection, but they tend to generalize poorly in the real-world scenarios, where the synthetic images are usually generated with unseen models using unknown source data. In this work, we emphasize the importance of combining information from the whole image and informative patches in improving the generalization ability of AI-synthesized image detection. Specifically, we design a two-branch model to combine global spatial information from the whole image and local informative features from multiple patches selected by a novel patch selection module. Multi-head attention mechanism is further utilized to fuse the global and local features. We collect a highly diverse dataset synthesized by 19 models with various objects and resolutions to evaluate our model. Experimental results demonstrate the high accuracy and good generalization ability of our method in detecting generated images. Our code is available at https://github.com/littlejuyan/FusingGlobalandLocal.

Key findings

The proposed method achieved superior accuracy and significantly better generalization ability across diverse unseen models compared to existing baselines. It attained a Total mAP of 91.732% and Global AP of 96.906%, outperforming other methods. The model also demonstrated good robustness to post-processing operations like Gaussian blur and JPEG compression.

Approach

The authors design a two-branch model: a global branch extracts spatial features from the whole image using a CNN backbone, and a local branch extracts fine-grained features from multiple informative patches. A novel Patch Selection Module (PSM) automatically selects these patches. An Attention-based Feature Fusion Module (AFFM) leveraging multi-head attention then fuses the global and local embeddings for binary classification.

Datasets

For training, a dataset from [6] with 362K real images (LSUN) and 362K ProGAN-generated images was used. For testing, a custom-composed diverse dataset of 128,424 images, synthesized by 19 different generative models (including Conditional GANs, Unconditional GANs, Perceptual Loss, Low-level Vision, DeepFakes like FF++ and Celeb-DF, and others like StyleGAN3, Transformers, BGM), was used.

Model(s)

The core model uses ResNet-50 (pre-trained on ImageNet) as the backbone for feature extraction. A novel Patch Selection Module (PSM) selects informative regions. Features are fused using a Multi-head Attention-based Feature Fusion Module (AFFM). The Adam optimizer and BCE loss function were used during training.

Author countries

USA

← Previous