MoSTE: Mixture of Spatio-Temporal Experts for Deepfake Detection in Videos

Authors: Charbel Yahchouchi, Nabyl Quignon, Noemi Roggero, Laurent Saroul, Antitza Dantcheva

Published: 2026-11-09 14:02:42+00:00

Comment: Accepted to IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2027

AI Summary

This paper introduces MoSTE (Mixture Of Spatio-Temporal Experts), an adaptation framework designed to improve the generalizability of deepfake video detection. MoSTE addresses the challenge of diverse forgery artifacts by decomposing adaptation into parallel spatial and temporal expert pools and dynamically routing tokens to the most relevant experts, enabling specialization without explicit manipulation labels. This approach sets a new state-of-the-art in average generalization across various cross-dataset and cross-manipulation benchmarks.

Abstract

Generalizable deepfake video detection remains challenging because forgery artifacts vary substantially across manipulation types and often combine diverse spatial and temporal inconsistencies. Existing parameter-efficient adaptation methods typically rely on a single shared representation, rendering them prone to manipulation-specific bias and limiting transfer to unseen forgeries. To address this challenge, we propose Mixture Of Spatio-Temporal Experts (MoSTE), an adaptation framework that enables conditional specialization of forensic representations. Instead of forcing one adapter to model all forgery patterns, MoSTE decomposes adaptation into parallel spatial and temporal expert pools and dynamically routes tokens to the most relevant experts through an unsupervised token-level routing mechanism. This allows experts to naturally specialize in complementary forgery cues without requiring explicit manipulation labels or multi-stage training, while effectively transferring representations from CLIP to the video domain. Extensive experiments across cross-dataset and cross-manipulation benchmarks demonstrate that MoSTE sets a new state of the art in average generalization while preserving strong performance across diverse manipulation families. Additional analysis of expert routing and attention behavior reveals the emergence of expert diversity, suggesting that MoSTE learns complementary spatio-temporal forensic features. Code is available at https://github.com/charbelyah/MoSTE


Key findings
MoSTE achieves superior average generalization across various deepfake detection benchmarks, setting a new state of the art in cross-dataset and cross-manipulation evaluations. It demonstrates robust performance against common post-processing perturbations like compression and noise. The architecture's effectiveness stems from the conditional spatio-temporal expert routing, which enables specialization in diverse forgery cues without explicit supervision, rather than merely increased model capacity.
Approach
MoSTE adapts a frozen CLIP-ViT backbone by inserting adapters that feature a mixture of spatio-temporal experts. Instead of a single adapter, it uses parallel spatial and temporal expert pools, each processing tokens based on intra-frame or inter-frame inconsistencies. An unsupervised token-level routing mechanism, aided by an auxiliary-loss-free load balancing strategy, dynamically assigns tokens to the most relevant experts, allowing them to specialize in complementary forgery cues.
Datasets
Celeb-DF-v2 (CDF-v2), Celeb-DF-v3 (CDF-v3), DeepFakeDetection (DFD), DeepFake Detection Challenge (DFDC) dataset, DFDCP, FaceForensics++ (FF++), DF40
Model(s)
CLIP-ViT-L/14 (as frozen backbone)
Author countries
UNKNOWN