Beyond Shortcut Learning: Hierarchical Spatial Prototype Regularization for Face Forgery Detection

Authors: Chao Shuai, Shaojing Fan, Kun Pan, Jinwu Hu, Weichen Lian, Ziqi Yang, Zhongjie Ba, Zhenguang Liu

Published: 2026-10-21 09:02:34+00:00

AI Summary

This paper addresses the challenge of generalizing deepfake detectors to unseen manipulations by tackling 'forensic representation collapse', where models overfit unstable, dataset-specific artifacts. The authors propose Hierarchical Spatial Prototype (HSP) regularization, a plug-in module that explicitly models the forgery manifold by learning a structured prototype subspace over spatial features. HSP enhances generalizability by preserving the geometric independence of latent forgery factors and promoting high-effective-rank utilization of the forensic representation, leading to substantial improvements in cross-dataset and cross-manipulation AUC scores.

Abstract

Recent advances in generative models have enabled highly realistic face forgeries, yet generalizing deepfake detectors to unseen manipulation pipelines remains a major challenge. One critical yet underexplored issue is that the learned forensic representation itself may fail to remain stable across different manipulation sources. In this paper, we provide new evidence that naively fine-tuned vision models suffer from severe \\textit{forensic representation collapse}: their extracted cues are often entangled with unstable, dataset-specific artifacts, resulting in structurally fragile representations and poor cross-domain generalization. Leveraging this insight, we propose \\textbf{H}ierarchical \\textbf{S}patial \\textbf{P}rototype (\\textbf{HSP}) regularization, which explicitly models the forgery manifold by learning a structured prototype subspace over spatial features. We further introduce prototype orthogonality and second-order assignment balance, which jointly preserve the geometric independence of latent forgery factors and promote high-effective-rank utilization of the forensic representation, thereby reducing reliance on dataset-specific shortcuts. Empirical results demonstrate that our method substantially improves generalizability, yielding average frame-level AUC gains of 5.6\\% and 5.3\\% in the cross-dataset and cross-manipulation settings, respectively. Moreover, when used as a plug-in module, HSP boosts the average video-level AUC of existing state-of-the-art detectors to 95.3\\% (+1.8\\%). These results demonstrate the practical value of HSP as an easy-to-adopt regularization module for enhancing deepfake detectors.