Unifying Semantic Priors and High-Frequency Traces: Enhancing V-JEPA with Mixture-of-Experts for Robust Synthetic Image Forensics

Authors: Simone Teglia, Irene Amerini

Published: 2026-09-15 07:46:01+00:00

Comment: 10 pages, 2 figures. Code available at https://github.com/ALCOR-Lab-DIAG/MoE-JEPA

AI Summary

This paper introduces MoE-JEPA, a novel dual-stream architecture for robust deepfake detection that pioneers the use of Joint-Embedding Predictive Architecture (JEPA) models. By enhancing a V-JEPA 2 backbone with a Residual Mixture-of-Experts (MoE) mechanism and a noise stream, the model leverages both semantic understanding and high-frequency forensic traces. MoE-JEPA achieves state-of-the-art performance on the SID-Set benchmark, demonstrating superior accuracy and resilience to image degradations.

Abstract

The unchecked proliferation of manipulated images on social media platforms has increased the spread of misinformation, posing a severe threat to public trust and information integrity. Modern deepfake detectors typically rely on Vision Transformers (ViTs) to capture the low-level inconsistencies that characterize fully synthetic or locally tampered images. However, the global understanding of such foundation models is not enough to discriminate alone between real and fake multimedia content, especially in challenging scenarios where images are compressed or transmitted through social media. In this paper we pioneer the application of Joint-Embedding Predictive Architecture (JEPA) models to deepfake detection, taking advantage of the generalized representation of visual reality that such World Models have exhibited. We hypothesize, and empirically demonstrate, that the intrinsic world understanding of JEPA models can be used as a strong prior for a deepfake detector. To fully exploit JEPA capabilities, we propose MoE-JEPA, a dual-stream architecture for deepfake detection. By enhancing a V-JEPA 2 backbone with a Residual Mixture-of-Experts (MoE) mechanism, along with a noise stream branch, our model dynamically internalizes forensic knowledge. Furthermore, a Gated Attention Multiple Instance Learning (MIL) module is employed to ensure precise spatial semantic understanding. Evaluated on the SID-Set benchmark, comprising 300K AI-generated, tampered and authentic images, MoE-JEPA establishes a new state-of-the-art with an accuracy of 95.54%, successfully outperforming vastly larger models.

Key findings

MoE-JEPA achieved a new state-of-the-art accuracy of 95.54% on the SID-Set benchmark, outperforming larger models. The dual-stream approach, integrating semantic priors from V-JEPA with high-frequency noise traces, proved crucial for robust detection across various image types and degradations. Ablation studies confirmed the significant contributions of MoE layers, Gated Attention MIL pooling, and the noise stream to the model's overall performance and robustness.

Approach

MoE-JEPA employs a dual-stream architecture: a semantic branch with a V-JEPA 2 backbone augmented by a Residual Mixture-of-Experts and a Gated Attention Multiple Instance Learning module for global understanding and localized tampering detection, and a parallel noise branch using a constrained BayarConv layer to extract high-frequency artifacts. These streams are combined using an Adaptive Gated Residual Fusion mechanism to detect both semantic inconsistencies and subtle generative traces.

Datasets

SID-Set, RRDataset, GenImage-SDv1.4, ImageNet

Model(s)

V-JEPA 2, Residual Mixture-of-Experts (MoE), Gated Attention Multiple Instance Learning (MIL), BayarConv

Author countries

Italy

← Previous