When AUC Misleads: Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift

Authors: Dat Nguyen, Cosmin Radoi, Romain Hermary, Marcella Astrid, Nesryne Mejri, Enjie Ghorbel, Djamila Aouada

Published: 2026-06-17 15:24:06+00:00

AI Summary

This paper introduces Cross-dataset AUC (Cross-AUC), a novel metric for evaluating deepfake detectors' generalization capabilities under domain shifts. It addresses the limitations of traditional AUC, which often overestimates performance in real-world scenarios by failing to account for mixed data sources and varying artifact types. Cross-AUC integrates the mean per-domain AUCs with a measure of prediction polarization, quantified by the Wasserstein Distance, to provide a more realistic and interpretable assessment of detector robustness.

Abstract

Recent advances in generative AI, such as diffusion models and face-swapping tools, have enabled the creation of highly realistic deepfakes, leading to real-world harms including financial fraud and non-consensual explicit content. In response, deepfake detection has become an active research area, with recent methods increasingly focusing on improving generalization to unseen manipulations. This is typically evaluated using the Area Under the ROC Curve (AUC) measured separately across multiple datasets. However, such an evaluation fails to reflect real-world scenarios where detectors face a mixture of data sources and varying artifact types. To address this limitation, we introduce a novel metric, Cross-dataset AUC (Cross-AUC) that averages per-domain AUCs with a measure of prediction polarization for taking into account the robustness to domain shift. The polarization extent is quantified by the Wasserstein Distance between class score distributions. Cross-AUC not only assesses the generalization capabilities of deepfake detectors under domain shifts more realistically, but it is also interpretable as it better explains the reason behind a drop in performance. Experiments performed on seven benchmark datasets demonstrate its practical relevance.

Key findings

Experiments on seven benchmark datasets demonstrate that conventional average AUC consistently overestimates performance, with Cross-AUC aligning more closely with performance on combined, realistic datasets. Most state-of-the-art deepfake detectors show noticeable performance drops and reduced prediction polarization when confronted with domain shifts, indicating weaker generalization than claimed. The proposed Cross-AUC is a reliable and practical metric for evaluating generalization, highlighting that higher combined AUC values correlate with greater prediction polarization and more stable score alignment across domains.

Approach

The authors propose Cross-AUC, a new evaluation metric that combines the mean AUC across individual datasets with a measure of prediction polarization. This polarization, defined as the Wasserstein Distance between class score distributions, quantifies the separability of real and fake predictions and the robustness to threshold instability. This approach aims to more accurately reflect deepfake detector performance in real-world, mixed-domain settings.

Datasets

FaceForensics++ (FF++), Celeb-DF (CDF), Google Deepfake Detection (DFD), WildDeepfake (DFW), Deepfake Detection Challenge (DFDC), Deepfake Detection Challenge Preview (DFDCP), DF40, MagicBrush

Model(s)

UNKNOWN

Author countries

Luxembourg, Tunisia

← Previous