Segmentation-Guided Spatial Indexing for Generalizable and Explainable Deepfake Detection

Authors: Izaldein Al-Zyoud, Abdulmotaleb El Saddik

Published: 2026-05-25 17:07:00+00:00

AI Summary

This paper introduces segmentation-guided spatial indexing for generalizable and explainable deepfake detection, reversing the typical design order by first selecting semantically meaningful patch tokens before pooling. A frozen FaRL parser assigns semantic labels to DINOv3 ViT-L/16 patch tokens, discarding non-target tokens, and a linear probe then classifies the retained region. This approach leverages DINOv3's spatial consistency to provide a purer regional subspace, enhancing both detection accuracy and interpretability.

Abstract

We introduce segmentation-guided spatial indexing for generalizable and explainable deepfake detection. The key idea reverses the standard design order: rather than pooling all facial tokens and classifying afterward, we first select semantically meaningful patch tokens, then pool only those. A frozen FaRL parser assigns each DINOv3 ViT-L/16 patch token a semantic label; non-target tokens are discarded; a linear probe classifies the retained region. This spatial indexing exploits DINOv3's patch-level spatial consistency, the same property that enables emergent segmentation, to present the probe with a purer regional subspace where manipulation-relevant evidence is less diluted by whole-face cues. Region attribution is structural: when the mouth model predicts fake, the decision used only mouth tokens, not an overlaid saliency map. On Celeb-DF v2, the mouth-indexed probe achieves AUC 0.905, outperforming LipForensics (+8.1 pp) and Xception (+16.9 pp), with no DINOv3 or FaRL fine-tuning and no target-domain data. Ablations isolate the mechanism: replacing regional selection with DINOv3's CLS token drops Celeb-DF v2 AUC by 26.4 pp; replacing DINOv3 with FaRL features drops it by 20.9 pp. Both DINOv3 representation and the spatial index are independently necessary; neither alone approaches the full system.

Key findings

The mouth-indexed probe achieved AUC 0.905 on Celeb-DF v2 and 0.930 on DFD, outperforming LipForensics (+8.1 pp) and Xception (+16.9 pp) without fine-tuning DINOv3 or FaRL. Ablations showed that replacing regional selection with DINOv3's CLS token dropped Celeb-DF v2 AUC by 26.4 pp, confirming the necessity of the segmentation-guided spatial indexing. The 'filter-before-pool' design is crucial for strong cross-dataset generalization and provides structural interpretability by restricting decisions to specific semantic regions.

Approach

The method processes facial regions by first segmenting them using a frozen FaRL parser to assign semantic labels to DINOv3 ViT-L/16 patch tokens. It then discards non-target tokens, mean-pools only the semantically selected regional tokens, and feeds this regional descriptor to a linear probe for classification. This 'filter-before-pool' design ensures the classifier operates on manipulation-relevant regional evidence, built for structural interpretability.

Datasets

FaceForensics++, Celeb-DF v2, DFD

Model(s)

DINOv3 ViT-L/16, FaRL ViT-B/16+UperNet parser, RetinaFace (for preprocessing), linear probe (logistic regression)

Author countries

Canada

← Previous