Honest Fusion Evaluation for Cross-Domain Deepfake Detection: Leakage Audits and Tail Robustness

Authors: Luke Collins, Chang-Tsun Li, Duc Thanh Nguyen

Published: 2026-11-11 00:10:06+00:00

Comment: Accepted at WACV 2027. 19 pages including supplementary material; 6 figures, 10 tables. Code: https://github.com/lukegcollins/WACV27-HFEFCDD

AI Summary

This paper re-evaluates cross-domain video deepfake detection methods, specifically focusing on score averaging, per-video routing, and feature fusion of frozen foundation-model experts. The authors find that previously reported gains from expressive combiners (routing and feature fusion) are largely artifacts of evaluation leakage. They propose honest evaluation protocols and demonstrate that while fusion improves tail robustness for hard videos, global calibrated score fusion (INVB) consistently performs best under honest evaluation for ranking metrics.

Abstract

Cross-domain video deepfake detection increasingly uses frozen foundation-model experts, combined by calibrated score averaging, a per-video router picking one expert, or a learned head over concatenated features. Expressive routing and feature fusion are often reported to beat plain averaging. Across a 22-dataset frozen-expert testbed, we find those gains are largely evaluation artifacts. A corner theorem locates the gap routing aims to close: because AUC depends only on score ordering, the best any score-space router can do is an oracle reading the test label to pick each video's expert, a ceiling no realisable, label-blind router reaches. The theorem does not order real routers against a calibrated global average; the case against routing is empirical, and no router we test beats that average. Feature fusion reads raw features, falls outside the theorem, and is audited empirically. Reported gains trace to leakage: holding out one dataset leaks siblings sharing identities and reals (over-crediting routing), while holding out one manipulation keeps that dataset's reals in training (over-crediting feature fusion). A $2\\times 2$ over fold and evaluation granularity pins the $+2.6$pp feature-fusion win to the fold alone: 12/12 significant under leaky folds vs. 0/12 under honest family-held-out folds, where both gains vanish. Fusion still pays off in the tail of hard videos and families: as experts are added, videos every expert misclassifies drop from 15.2% to 2.3%, and global calibrated fusion (InvB) beats the best single expert on worst-family AUC, selective risk (AURC), and low-false-positive recall, stably across protocols. Practitioners should calibrate and combine globally, hold out whole dataset families, and measure tail performance. For score-space routing under ranking metrics, honest evaluation and expert diversity matter more than combiner complexity.


Key findings
Reported gains from expressive combiners (routing and feature fusion) are largely evaluation artifacts due to leakage in standard protocols. Under honest evaluation (LOFO), no tested score-space router consistently beats global calibrated fusion (INVB) for ranking metrics. Fusion and expert diversity significantly improve tail robustness by reducing the 'ALL-WRONG' rate (videos misclassified by all experts) and boosting performance on worst-family AUC, selective risk, and low-false-positive recall.
Approach
The authors analyze and compare different expert combination strategies for deepfake detection using a testbed of 22 datasets and four frozen foundation-model experts. They introduce a 'corner theorem' to bound the AUC of score-space routing, and empirically audit routing and feature fusion for evaluation leakage. They propose and evaluate using 'leave-one-dataset-family-out' (LOFO) as an honest evaluation protocol, which holds out entire dataset families to prevent leakage of shared identities and reals.
Datasets
FaceForensics++, DeepFake Detection Challenge (DFDC), Google/Jigsaw DeepFake Detection, DFDM, FakeAVCeleb, KoDF, Celeb-DF series (v1/v2/v3), DeepSpeak series (v1/v2), PolyGlotFake, IDForge, DeeperForensics-1.0, UADFV, Face Forensics in the Wild
Model(s)
DINOv3 (spatial expert), V-JEPA 2.1 (temporal expert), FREQ-DCT (spectral CNN), FSFM (face self-supervised ViT-L), CLIP/X-CLIP (fifth expert for tail extension)
Author countries
Australia