Inconsistency-aware Multimodal Schrödinger Bridge for Deepfake Localization

Authors: Jiayu Xiong, Jing Wang, Qi Zhang, Wanlong Wang, Jun Xue

Published: 2026-05-22 00:17:16+00:00

Comment: Accepted by CVPR2026

AI Summary

IaMSB is an inconsistency-aware multimodal Schrödinger Bridge framework for audio-visual deepfake localization, designed to overcome noise propagation and precision degradation from symmetric fusion in single-sided or asynchronous forgeries. It jointly estimates cross-modal consistency, selects information, and schedules bridge steps to produce refined, time-aligned interval-level outputs. The method significantly improves strict-IoU boundary precision, particularly for challenging forgery types.

Abstract

Audio-visual deepfake localization demands interval-level outputs that serve as temporal evidence. Despite recent progress, symmetric fusion under single-sided or asynchronous forgeries propagates cross-modal noise, degrading high-precision localization. We present IaMSB, an inconsistency-aware multimodal Schrödinger Bridge (SB) that jointly estimates cross-modal consistency and performs interval-level localization. Unlike diffusion models, SB minimizes path-distribution discrepancy and yields consistency scores without explicit noise injection or denoising. With the Schrödinger Bridge (SB), IaMSB unifies consistency estimation, cross-modal information selection, and bridge-step scheduling in one framework. Specifically, a lightweight coarse bridge first proposes candidate intervals and estimates cross-modal consistency; these statistics select cross-modal witness signals and allocate bridge steps asymmetrically across modalities. A refinement bridge then performs step-tuned fusion and outputs refined, time-aligned intervals. IaMSB anticipates single-sided and asynchronous forgeries and, using bottlenecked cross-modal interaction with step allocation, suppresses noise transfer, avoids unnecessary iterations. Across benchmarks, IaMSB stabilizes strict-IoU boundary precision, raising AP@0.95 by 3%~10%, and yields improved high-precision localization, particularly for single-sided forgeries.

Key findings

IaMSB significantly improves strict-IoU boundary precision, raising AP@0.95 by 3%~10% across multiple benchmarks, outperforming existing methods. The method demonstrates robust cross-modal consistency and superior high-precision localization, particularly for single-sided forgeries. This is achieved by adaptive compute allocation and bottlenecked cross-modal interaction, which effectively suppresses noise transfer and focuses refinement where inconsistencies are higher.

Approach

The approach utilizes a cascaded Schrödinger Bridge (SB) architecture with three stages: a Coarse SB (CSB) for initial interval proposals, a Witness SB (WSB) that employs entropy-regularized optimal transport to estimate cross-modal consistency and asymmetrically allocate computational steps, and a Refinement SB (RSB) to perform step-tuned fusion and refine the selected queries into precise, time-aligned event intervals.

Datasets

LAV-DF, AV-Deepfake1M, TVIL

Model(s)

Schrödinger Bridge (SB) framework (Coarse SB, Witness SB, Refinement SB), Multi-Head Attention, SwiGLU Feed-Forward Networks. Encoders used are ViT-S initialized from VideoMAE (visual) and WavLM (audio).

Author countries

China

← Previous