What Do Deepfake Audio Detectors Hear? From Self-Supervised Models to Audio LLMs

Authors: Wanying Ge, Xin Wang, Junichi Yamagishi

Published: 2026-10-13 02:40:30+00:00

Comment: Accepted to SLT 2026

AI Summary

This research investigates how deepfake audio detectors perceive mixed audio recordings containing both speech and background events, which can independently be real or fake. The study reveals that current detectors are better at identifying fake events than fake speech, primarily due to poor generalization to unseen fake speech domains. Furthermore, the front-end encoder significantly influences where a countermeasure focuses its frequency evidence.

Abstract

Real-world audio recordings for deepfake detection rarely consist solely of speech; they often are mixtures of speech and background events that themselves may also be synthetic. When processing these mixtures, it remains unclear whether detectors attend to the speech or the background event, and where their spectral evidence lies. We study this on the CompSpoof V2 database, whose mixtures pair independently real-or-fake speech and event components. Our results show that all systems detect fake events more accurately than fake speech. However, this performance gap is largely driven by poor generalization to an unseen fake speech domain, rather than a true difference in difficulty. Furthermore, sub-band probing reveals that the front-end encoder dictates where a countermeasure focuses its frequency evidence. To enable reliable deployment in real-world scenarios, future countermeasures should address out-of-domain generalization failures and carefully select front-end encoders aligned with their target acoustic evidence. Our source code and checkpoints are released at https://github.com/nii-yamagishilab/AntiDeepfake-AudioLLM.

Key findings

All tested systems detect fake events more accurately than fake speech, but this performance gap is largely attributed to poor generalization to unseen fake speech domains. Sub-band probing demonstrates that the front-end encoder dictates the frequency areas where countermeasures concentrate their evidence. Future deepfake audio detection systems should prioritize addressing out-of-domain generalization failures and carefully select front-end encoders aligned with target acoustic evidence for reliable real-world deployment.

Approach

The authors analyze the behavior of two families of deepfake audio detectors (Standard CMs and ALLM-based CMs) on the CompSpoof V2 database, which features independently real or fake speech and event components. They evaluate component sensitivity by comparing detection rates for fake speech versus fake events and use sub-band probing to determine which frequency bands carry the most significant evidence for detection.

Datasets

CompSpoof V2 database, SoulX-Podcast, MLAAD, ASVspoof 5

Model(s)

XLS-R, Whisper encoder, Qwen2-Audio (disc), Qwen2-Audio (gen)

Author countries

Japan

← Previous