DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization

Authors: Christos Koutlis, Symeon Papadopoulos

Published: 2024-11-15 13:47:33+00:00

AI Summary

This paper introduces DiMoDif, an audio-visual deepfake detection and localization framework that leverages inter-modality differences in machine perception of speech. It identifies frame-level cross-modal incongruities to detect and temporally localize deepfake forgeries. DiMoDif significantly outperforms state-of-the-art methods on challenging benchmarks for both deepfake detection and temporal forgery localization.

Abstract

Deepfake technology has rapidly advanced and poses significant threats to information integrity and trust in online multimedia. While significant progress has been made in detecting deepfakes, the simultaneous manipulation of audio and visual modalities, sometimes at small parts or in subtle ways, presents highly challenging detection scenarios. To address these challenges, we present DiMoDif, an audio-visual deepfake detection framework that leverages the inter-modality differences in machine perception of speech, based on the assumption that in real samples -- in contrast to deepfakes -- visual and audio signals coincide in terms of information. DiMoDif leverages features from deep networks that specialize in visual and audio speech recognition to spot frame-level cross-modal incongruities, and in that way to temporally localize the deepfake forgery. To this end, we devise a hierarchical cross-modal fusion network, integrating adaptive temporal alignment modules and a learned discrepancy mapping layer to explicitly model the subtle differences between visual and audio representations. Then, the detection model is optimized through a composite loss function accounting for frame-level detections and fake intervals localization. DiMoDif outperforms the state-of-the-art on the Deepfake Detection task by 30.5 AUC on the highly challenging AV-Deepfake1M, while it performs exceptionally on FakeAVCeleb and LAV-DF. On the Temporal Forgery Localization task, it outperforms the state-of-the-art by 47.88 AP@0.75 on AV-Deepfake1M, and performs on-par on LAV-DF. Code available at https://github.com/mever-team/dimodif.

Key findings

DiMoDif achieves state-of-the-art performance, outperforming previous methods by 30.5 AUC for deepfake detection and 47.88 AP@0.75 for temporal forgery localization on the challenging AV-Deepfake1M dataset. It also performs exceptionally well on FakeAVCeleb and LAV-DF, demonstrating robust detection capabilities even for subtle or partial manipulations and high interpretability due to its cross-modal similarity analysis.

Approach

The approach, DiMoDif, extracts speech features from visual and audio streams using pre-trained visual and audio speech recognition (VSR/ASR) models. A hierarchical cross-modal fusion network, incorporating adaptive temporal alignment modules and a learned discrepancy mapping layer, then models subtle differences between these representations to identify frame-level inconsistencies. The model is optimized using a composite loss function for both detection and localization tasks.

Datasets

FakeAVCeleb, LAV-DF, AV-Deepfake1M, VoxCeleb2 (for balancing FakeAVCeleb)

Model(s)

DiMoDif framework; components include VSR/ASR models (specifically Ma 2022 [46]), a Transformer encoder with local cross-modal attention and feature pyramids (for the Modality-information Differentiation module), and feed-forward/1-D convolutional heads for classification/regression.

Author countries

Greece

← Previous