MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection

Authors: Ahmed Y. Radwan, Shaina Raza

Published: 2026-11-09 14:48:33+00:00

AI Summary

This paper introduces Multiscale Appearance-Residual Streams (MARS), a two-path video deepfake detector that combines multiscale feature differences with adapted appearance representations. MARS addresses the challenge of generalizing deepfake detection across diverse forgery types, including facial manipulation and fully generated videos, by jointly interpreting spatial and temporal feature differences with appearance features. It achieves state-of-the-art performance, demonstrating strong generalization capabilities to unseen forgery categories.

Abstract

Generalizing deepfake detection across facial manipulation and fully generated video remains challenging due to forgery traces varying with the manipulation method and scene content. Explicit spatial and temporal feature differences offer a way to capture this variation, but also respond to natural texture and motion, motivating their joint interpretation with appearance features. We investigate whether combining multiscale feature differences with adapted appearance representations improves detection across both forgery categories. We introduce Multiscale Appearance-Residual Streams (MARS), a two-path video detector. The residual path applies fixed spatial and temporal difference-of-Gaussians operators to frozen intermediate features and models the resulting differences alongside appearance context across frames. The semantic path adapts the encoder's normalization parameters for frame classification, and a clip-dependent gate combines their predictions. Under a shared protocol across seven datasets, MARS achieves the highest average AUROC of 91.43\\%, outperforming seven compared detectors. When trained on facial manipulations alone, MARS achieves 93.99\\% AUROC on GenVideo and 75.50\\% on DF26, demonstrating transfer to generated videos without generated-video training data. MARS also provides signed patch and frame contributions, whose faithfulness we evaluate through controlled spatial and temporal removal against random controls. MARS is publicly available for research: Project page: https://vectorinstitute.github.io/mars/ Github: https://github.com/VectorInstitute/mars


Key findings
MARS achieved the highest average AUROC of 91.43% across seven datasets, outperforming seven other detectors. It demonstrated strong transferability, achieving 93.99% AUROC on GenVideo and 75.50% on DF26 when trained solely on facial manipulations. Ablation studies confirmed the complementarity of the two paths, with spatial and temporal difference-of-Gaussians operators providing significant contributions, and highlighted the importance of adapting pretrained representations through normalization for generated video detection.
Approach
MARS operates with two complementary paths: a residual path and a semantic path. The residual path applies fixed spatial and temporal difference-of-Gaussians operators to frozen intermediate features, modeling local changes across space and time. The semantic path adapts the encoder's normalization parameters for frame classification, focusing on the global content of each frame. A clip-dependent gate then combines the predictions from both paths.
Datasets
FF++ c23, GTA-V, CelebDF-V2, DeepFakeTIMIT, HiFiFace, UADFV, GenVideo, DF26
Model(s)
PE-Core-L/14 (Perception Encoder), Temporal Transformer
Author countries
Canada