On the Holistic Approach for Detecting Human Image Forgery

View on arXiv ← Back to list

Authors: Xiao Guo, Jie Zhu, Anil Jain, Xiaoming Liu

Published: 2026-01-08 08:33:22+00:00

AI Summary

The paper introduces HuForDet, a holistic detection framework designed to generalize across both localized facial manipulations and full-body synthetic human images. HuForDet employs a dual-branch architecture: one using heterogeneous experts (RGB and adaptive frequency domains) for facial analysis, and another leveraging an MLLM for contextualized full-body semantic consistency analysis. The framework achieves state-of-the-art performance on the newly curated HuFor dataset, addressing the fragmentation of existing deepfake detectors.

Abstract

The rapid advancement of AI-generated content (AIGC) has escalated the threat of deepfakes, from facial manipulations to the synthesis of entire photorealistic human bodies. However, existing detection methods remain fragmented, specializing either in facial-region forgeries or full-body synthetic images, and consequently fail to generalize across the full spectrum of human image manipulations. We introduce HuForDet, a holistic framework for human image forgery detection, which features a dual-branch architecture comprising: (1) a face forgery detection branch that employs heterogeneous experts operating in both RGB and frequency domains, including an adaptive Laplacian-of-Gaussian (LoG) module designed to capture artifacts ranging from fine-grained blending boundaries to coarse-scale texture irregularities; and (2) a contextualized forgery detection branch that leverages a Multi-Modal Large Language Model (MLLM) to analyze full-body semantic consistency, enhanced with a confidence estimation mechanism that dynamically weights its contribution during feature fusion. We curate a human image forgery (HuFor) dataset that unifies existing face forgery data with a new corpus of full-body synthetic humans. Extensive experiments show that our HuForDet achieves state-of-the-art forgery detection performance and superior robustness across diverse human image forgeries.

Key findings

HuForDet achieved state-of-the-art overall performance on the HuFor dataset, scoring 90.22% overall AUC, improving upon NPR (87.75%) by 2.47%. The method showed superior robustness and generalization across diverse forgery types, exhibiting strong performance on subsets covering both partial manipulations (UniAttack+) and full syntheses (Diff-Cele).

Approach

HuForDet uses a dual-branch architecture: a Face Forgery Detection Branch (Fface) employing heterogeneous experts in RGB and adaptive frequency (adaptive LoG) domains to capture local artifacts. The Contextualized Forgery Detection Branch (Fctx) uses a Multi-Modal Large Language Model (MLLM) to analyze full-body semantic consistency. The outputs of both branches are integrated via a confidence-aware fusion module, where Fctx's feature contribution is dynamically weighted by a self-assessed confidence score.

Datasets

HuFor, FaceForensics++ (FF++), UniAttack+, Diff-Cele, Celeb-DF

Model(s)

HuForDet (overall framework), DenseNet-121, adaptive Laplacian-of-Gaussian (adaLoG), CLIP-ViT, Vicuna-7B

Author countries

USA