From Spectrograms to Shallow Speech SSL: Audio-Frontend Design for Generator-Disjoint and Low-Latency Audio--Visual Deepfake Detection

Authors: Qiwu Wen, Jahangir Alam, Eric Charton

Published: 2026-10-16 19:07:55+00:00

AI Summary

This paper investigates audio-frontend design for audio-visual deepfake detection, specifically within a fixed MRDF-based pipeline. It compares spectrograms with self-supervised learning (SSL) representations like wav2vec 2.0 and WavLM, finding that shallow execution of SSL models with learned non-uniform aggregation significantly improves detection performance and reduces latency.

Abstract

Video-mediated conversational systems rely on synchronized speech and facial video, making audio--visual input integrity important for reliable perception. We study audio-frontend design in a fixed MRDF (Modality-Regularization-based DeepFake)-based pipeline with AV-HuBERT visual processing and an MSCT (multi-scale cross-modal transformer) fusion encoder. We compare spectrogram-style log-filterbank features with frozen wav2vec~2.0 and WavLM representations, dual-SSL fusion, executed depth, and shallow-layer aggregation. Across three training runs, full wav2vec~2.0 improves mean macro AUROC from $84.19\\%$ to $87.19\\%$ on generator-disjoint MMDF. Six-layer execution further improves detection, and learned non-uniform aggregation adds $2.53$ percentage points, reaching $90.61\\%$. Shallow execution reduces measured inference latency relative to full-depth wav2vec~2.0 across the tested input durations; first-second evaluation reaches $86.85\\%$ macro AUROC. In a single-run comparison, dual-SSL concatenation outperforms either single stream, while one-way cross-attention yields lower detection performance and higher latency than concatenation. Layer-weight analysis shows greater emphasis on early speech layers, while removing the selector's visual input has little effect on detection performance.

Key findings

Full wav2vec 2.0 significantly improves mean macro AUROC over spectrograms. Shallow execution (e.g., six layers) of wav2vec 2.0 further enhances detection while reducing inference latency. Learned non-uniform aggregation of shallow layers yields the best performance, reaching 90.61% macro AUROC, with early speech layers receiving greater emphasis.

Approach

The authors integrate different audio frontends (spectrograms, frozen wav2vec 2.0, WavLM) into a fixed Modality-Regularization-based DeepFake (MRDF) pipeline with AV-HuBERT visual processing and a multi-scale cross-modal transformer (MSCT) fusion encoder. They explore varying execution depths of the SSL models and implement learned non-uniform aggregation of shallow layers to optimize for both detection accuracy and inference latency.

Datasets

MMDF (Multi-Modal DeepFake)

Model(s)

wav2vec 2.0 (W2V2), WavLM, AV-HuBERT, MSCT (multi-scale cross-modal transformer)

Author countries

UNKNOWN

← Previous