MIS-AVoiDD: Modality Invariant and Specific Representation for Audio-Visual Deepfake Detection

View on arXiv ← Back to list

Authors: Vinaya Sree Katamneni, Ajita Rattani

Published: 2023-10-03 17:43:24+00:00

AI Summary

This paper introduces MIS-AVoiDD, a novel multimodal deepfake detector that uses modality-invariant and -specific representations to improve audio-visual fusion for deepfake detection. Experimental results on FakeAVCeleb and KoDF datasets show significant accuracy improvements over state-of-the-art unimodal and multimodal methods.

Abstract

Deepfakes are synthetic media generated using deep generative algorithms and have posed a severe societal and political threat. Apart from facial manipulation and synthetic voice, recently, a novel kind of deepfakes has emerged with either audio or visual modalities manipulated. In this regard, a new generation of multimodal audio-visual deepfake detectors is being investigated to collectively focus on audio and visual data for multimodal manipulation detection. Existing multimodal (audio-visual) deepfake detectors are often based on the fusion of the audio and visual streams from the video. Existing studies suggest that these multimodal detectors often obtain equivalent performances with unimodal audio and visual deepfake detectors. We conjecture that the heterogeneous nature of the audio and visual signals creates distributional modality gaps and poses a significant challenge to effective fusion and efficient performance. In this paper, we tackle the problem at the representation level to aid the fusion of audio and visual streams for multimodal deepfake detection. Specifically, we propose the joint use of modality (audio and visual) invariant and specific representations. This ensures that the common patterns and patterns specific to each modality representing pristine or fake content are preserved and fused for multimodal deepfake manipulation detection. Our experimental results on FakeAVCeleb and KoDF audio-visual deepfake datasets suggest the enhanced accuracy of our proposed method over SOTA unimodal and multimodal audio-visual deepfake detectors by $17.8$% and $18.4$%, respectively. Thus, obtaining state-of-the-art performance.

Key findings

MIS-AVoiDD outperforms state-of-the-art unimodal and multimodal deepfake detectors on both FakeAVCeleb and KoDF datasets. The improvement is attributed to the effective fusion of modality-invariant and -specific representations. Ablation studies confirm the importance of all components of the proposed method.

Approach

MIS-AVoiDD learns both modality-invariant and -specific representations for audio and visual streams. These representations are fused using a transformer-based self-attention mechanism, followed by a final classification layer for deepfake detection. The model is trained with a loss function that incorporates modality invariance, orthogonality, similarity, and classification losses.

Datasets

FakeAVCeleb, KoDF

Model(s)

MTCNN for face detection, Xception for visual feature extraction, MFCC for audio feature extraction, a custom designed model with transformer-based self-attention and a feedforward neural network for feature fusion and classification.

Author countries

USA

← Previous