Beyond Frame-Level Accuracy: Analyzing Video-Level Stability in Transformer-Based Deepfake Forensics

Authors: Syed Azmul Hasan, Mowmita Pervin Hera, Afroza Lona Mim, M M Jihad Hossain, Hazrat Ali, Mohammad Farhad Bulbul

Published: 2026-11-06 21:37:23+00:00

Comment: Paper submitted to international journal of intelligent systems, Wiley publishers, accepted version of open-access paper

Journal Ref: International Journal of Intelligent Systems, 2026, 6801006, 13 pages, 2026

AI Summary

This research provides a controlled comparative study of CNNs (ResNet-18, EfficientNet-B0) and a Vision Transformer (ViT-B/16) for deepfake detection, focusing on video-level stability rather than just frame-level accuracy. The study finds that while CNNs are competitive at the frame level, ViT-B/16 offers more stable video-level performance within a given data domain, suggesting global self-attention improves robustness, but highlights a significant cross-dataset generalization gap.

Abstract

Deepfake videos generated by advanced generative models pose serious threats to digital security, misinformation control, and media forensics. Although numerous detection methods have been proposed, their reported performances are often based on heterogeneous preprocessing pipelines, inconsistent data splits, and model-specific training protocols, hindering fair architectural comparison. This work presents a controlled and reproducible comparative study of convolutional neural networks (ResNet-18 and EfficientNet-B0) and a transformer-based architecture (ViT-B/16) within a unified, face-centric deepfake detection framework. All models are trained using identical preprocessing, augmentation, optimization settings, and data splits on Celeb-DF v2 and FaceForensics++, and results are validated across three independent random seeds with bootstrapped confidence intervals. Experimental results show that while CNN-based models perform competitively at the frame level, their performance degrades after video-level aggregation. In contrast, ViT-B/16 demonstrates more stable video-level performance, achieving 92.8\\% [95\\% CI: 91.7--94.0] video-level accuracy and an AUC of 0.950 [95\\% CI: 0.934--0.964] on Celeb-DF v2 (3-seed bootstrapped). Cross-dataset evaluation across the same three random seeds reveals a large, asymmetric generalization gap (AUC 0.79 FF++$\\rightarrow$Celeb-DF vs. 0.64 Celeb-DF$\\rightarrow$FF++), indicating that in-domain video-level stability does not transfer to unseen manipulation domains. These findings indicate that global self-attention improves robustness and video-level stability for deepfake detection within a given data domain, while highlighting the need for further work on cross-domain generalization.


Key findings
ViT-B/16 demonstrates more stable video-level performance compared to CNN-based models, achieving 92.8% video-level accuracy and an AUC of 0.950 on Celeb-DF v2. However, a large and asymmetric cross-dataset generalization gap was observed, indicating that in-domain video-level stability does not transfer to unseen manipulation domains. Global self-attention improves robustness and video-level stability within a given data domain, but further work is needed on cross-domain generalization.
Approach
The authors propose a unified, face-centric deepfake detection framework where CNNs (ResNet-18, EfficientNet-B0) and a Vision Transformer (ViT-B/16) are trained and evaluated under identical preprocessing, augmentation, optimization settings, and data splits. Frame-level predictions are aggregated using mean pooling to derive video-level scores, and results are validated across three independent random seeds with bootstrapped confidence intervals.
Datasets
Celeb-DF v2, FaceForensics++
Model(s)
ResNet-18, EfficientNet-B0, ViT-B/16
Author countries
Bangladesh, United Kingdom