LoCC: Detection and Localization of Lip-Syncing Deepfakes via Counterfactual Frame Consistency

Authors: Soumyya Kanti Datta, Shan Jia, Siwei Lyu

Published: 2026-06-22 02:19:39+00:00

Comment: Accepted at the IEEE International Conference on Multimedia and Expo (ICME) 2026

AI Summary

LoCC is a novel framework designed to detect and localize lip-syncing deepfakes by analyzing counterfactual frame consistency in the mouth region at segment and frame levels. It evaluates whether each frame aligns with a counterfactual estimate generated from temporal neighbors, leveraging the localized inconsistencies introduced by deepfakes. Utilizing a teacher-student learning paradigm, LoCC effectively captures these discrepancies, outperforming state-of-the-art methods on multiple benchmark datasets.

Abstract

Lip-syncing deepfakes are among the most challenging forms of manipulated media because their artifacts are localized almost exclusively to the mouth region and evolve dynamically over time. Detecting such deepfakes requires precise temporal and spatial modeling of lip motion. In this paper, we propose LoCC, a novel detection framework that performs fine-grained detection and localization of lip-syncing deepfakes at both segment and frame levels. Unlike prior approaches that analyze videos holistically, our method evaluates whether each frame aligns with a counterfactual estimate generated from its temporal neighbors. Real videos exhibit strong and stable consistency, whereas lip-sync deepfakes introduce localized inconsistencies. Following a teacher-student learning paradigm, our model effectively captures these frame-level discrepancies and achieves superior performance over state-of-the-art methods on multiple benchmark lip-syncing deepfake datasets, including LAV-DF, AVDF1M, FakeAVCeleb, and KODF, and generalizes well across compression levels and datasets.

Key findings

LoCC achieves superior performance in detecting and localizing lip-syncing deepfakes, significantly outperforming most baseline methods on FakeAVCeleb and AVDF1M, and demonstrating strong cross-dataset generalization on KODF. The framework effectively identifies localized inconsistencies in mouth motion, and ablation studies confirm the crucial contribution of each component (reconstruction, teacher, student, and Diffusion Inconsistency Loss) to its effectiveness.

Approach

LoCC employs a three-stage framework: a diffusion-based reconstruction model generates counterfactual middle frames from real mouth frames; a segment-level teacher network models inconsistencies over short segments using a Diffusion Inconsistency Loss; and a frame-level student network distills the teacher's knowledge for efficient per-frame prediction. This approach focuses on localized temporal and spatial inconsistencies in mouth motion.

Datasets

FakeAVCeleb, DeepSpeak v2.0, LavDF, AVDF1M, KODF

Model(s)

Diffusion-based reconstruction model, 3D Convolutional Network, Transformer Encoder, 2D Convolutional Neural Network, Multi-layer Perceptron (for student module)

Author countries

United States

← Previous