Honest Fusion Evaluation for Cross-Domain Deepfake Detection: Leakage Audits and Tail Robustness
Authors: Luke Collins, Chang-Tsun Li, Duc Thanh Nguyen
Published: 2026-11-11 00:10:06+00:00
Comment: Accepted at WACV 2027. 19 pages including supplementary material; 6 figures, 10 tables. Code: https://github.com/lukegcollins/WACV27-HFEFCDD
AI Summary
This paper re-evaluates cross-domain video deepfake detection methods, specifically focusing on score averaging, per-video routing, and feature fusion of frozen foundation-model experts. The authors find that previously reported gains from expressive combiners (routing and feature fusion) are largely artifacts of evaluation leakage. They propose honest evaluation protocols and demonstrate that while fusion improves tail robustness for hard videos, global calibrated score fusion (INVB) consistently performs best under honest evaluation for ranking metrics.
Abstract
Cross-domain video deepfake detection increasingly uses frozen foundation-model experts, combined by calibrated score averaging, a per-video router picking one expert, or a learned head over concatenated features. Expressive routing and feature fusion are often reported to beat plain averaging. Across a 22-dataset frozen-expert testbed, we find those gains are largely evaluation artifacts. A corner theorem locates the gap routing aims to close: because AUC depends only on score ordering, the best any score-space router can do is an oracle reading the test label to pick each video's expert, a ceiling no realisable, label-blind router reaches. The theorem does not order real routers against a calibrated global average; the case against routing is empirical, and no router we test beats that average. Feature fusion reads raw features, falls outside the theorem, and is audited empirically. Reported gains trace to leakage: holding out one dataset leaks siblings sharing identities and reals (over-crediting routing), while holding out one manipulation keeps that dataset's reals in training (over-crediting feature fusion). A $2\\times 2$ over fold and evaluation granularity pins the $+2.6$pp feature-fusion win to the fold alone: 12/12 significant under leaky folds vs. 0/12 under honest family-held-out folds, where both gains vanish. Fusion still pays off in the tail of hard videos and families: as experts are added, videos every expert misclassifies drop from 15.2% to 2.3%, and global calibrated fusion (InvB) beats the best single expert on worst-family AUC, selective risk (AURC), and low-false-positive recall, stably across protocols. Practitioners should calibrate and combine globally, hold out whole dataset families, and measure tail performance. For score-space routing under ranking metrics, honest evaluation and expert diversity matter more than combiner complexity.