Textual Acoustic Grounding for Generalizable LLM-Based Deepfake Voice Detection

View on arXiv ← Back to list

Authors: Yassine El Kheir, Xin Wang, Wanqing Ge, Tim Polzehl, Sebastian Moeller, Junichi Yamagishi

Published: 2026-08-31 11:29:51+00:00

Comment: 6pages + 1ref, SLT Submission

AI Summary

This research addresses the poor generalization of deepfake voice detection across unseen domains by enhancing Audio Large Language Models (ALLMs). They propose a cross-modal prompting strategy that injects linguistic-knowledge-driven acoustic features as structured text tokens, demonstrating that fine-tuning the LLM alone risks out-of-domain overfitting and a frozen LLM is a stronger baseline. This textual grounding significantly improves performance, especially on out-of-domain benchmarks.

Abstract

Deepfake voice detection suffers from poor generalization across unseen domains. While Audio Large Language Models (ALLMs) show promise, the modality gap between continuous audio embeddings which capture the subtle acoustic details necessary for deepfake detection and the semantic space of LLMs remains a critical, underexplored bottleneck. We address this by benchmarking diverse audio encoders integrated with Qwen LLMs (0.5B to 7B parameters). First, we demonstrate that fine-tuning the LLM alone risks out-of-domain overfitting, making a frozen LLM a stronger, resource-efficient baseline. Second, to explicitly bridge the modality gap, we introduce a cross-modal prompting strategy that injects linguistic-knowledge-driven acoustic features (via openSMILE) as structured text tokens. This explicit textual grounding not only enhances the frozen baseline but also makes LLM fine-tuning more effective. Ultimately, our approach demonstrates state-of-the-art resilience on the out-of-domain ITW and MLAAD benchmarks, yielding over \\textbf{16.2\\%} absolute improvement in Macro-F1 over existing ALLM baselines while maintaining competitive in-domain performance. All models reported in this work are \\href{https://huggingface.co/01Yassine/AudioLLM-Deepfake-Detection}{publicly available}.

Key findings
Fine-tuning the audio encoder is crucial for out-of-domain generalization, while fine-tuning the LLM alone can lead to overfitting. Explicitly injecting openSMILE features as text tokens significantly enhances the performance of ALLMs, particularly improving out-of-domain robustness and making LoRA-based LLM adaptation more effective. This approach achieves state-of-the-art results on unseen domains, with over 16.2% absolute F1 improvement over existing ALLM baselines.
Approach
The authors benchmark diverse audio encoders with Qwen LLMs (0.5B to 7B parameters) and show that a frozen LLM with fine-tuned audio encoders is a robust baseline. To bridge the modality gap, they introduce a cross-modal prompting strategy using openSMILE features, converted into structured text tokens, which enhances both frozen and fine-tuned LLM performance by providing explicit acoustic grounding.
Datasets
ASVspoof 2019 (LA), ASVspoof 2021 (LA21), In-the-Wild (ITW), MLAAD v9 (ML)
Model(s)
Qwen2.5-Instruct (0.5B, 3B, 7B), Qwen2.5-Omni-3B, Whisper, Wav2Vec2, EAT, SpeechTokenizer, DAC
Author countries
Germany, Japan
← Previous
Next →