Large Audio Language Models for Spoofing-Aware Speaker Verification

Authors: Sofya Savelyeva, Mariia Perunova, Evgeny Kushnir, Artem Dvirniak, Dmitrii Korzh, Oleg Y. Rogov

Published: 2026-07-16 09:38:45+00:00

AI Summary

This research explores the application of Large Audio Language Models (LALMs) for Spoofing-Aware Speaker Verification (SASV), a task combining speaker verification with deepfake detection. The study evaluates LALMs under various conditions, including zero-shot prompting, supervised adaptation, and novel reasoning-oriented training and reinforcement learning optimizations. The findings demonstrate that while LALMs are not inherently suited for SASV in a zero-shot setting, task-specific adaptations effectively bridge this performance gap, yielding competitive results against conventional pipelines and highlighting their potential as an auditable foundation for unified SASV.

Abstract

Recent advances in text-to-speech and voice cloning make high-quality spoofing inexpensive and scalable, threatening voice authentication systems, especially automatic speaker verification (ASV). Existing defenses mainly address this threat through binary countermeasures (CMs) for deepfake detection or spoofing-aware speaker verification (SASV), where current systems are dominated by modular ASV-CM fusion and cascaded pipelines. Although large audio language models (LALMs) have shown promise on related audio tasks, including CM and ASV, their use for SASV remains unexplored, despite their capacity to produce natural-language rationales for auditing and robustness beyond discriminative predictions. This work systematically evaluates LALMs for SASV against conventional pipelines under zero-shot prompting, supervised adaptation, reasoning-oriented training, and reinforcement-learning-based optimization. Our results show that pretrained LALMs are near chance in the zero-shot setting, confirming that they are not natively suited to SASV, but that task-specific adaptation closes this gap. We further find that competitive SASV performance can be achieved through several distinct routes. These findings position LALMs as a promising and auditable foundation for unified SASV, while clarifying where conventional cascade systems still lead.

Key findings

Pretrained LALMs perform poorly in zero-shot SASV, but task-specific adaptation using LoRA fine-tuning significantly improves performance, making them competitive with conventional SASV systems. A multi-objective loss function combining decision-level, speaker-discriminative (AAM), and spoof-detection (BCE) supervisions, along with hard-sample mining, effectively balances ASV and CM tasks, achieving strong overall accuracy and min a-DCF. While explicit reasoning supervision does not consistently outperform hard-label adaptation in terms of accuracy, it offers valuable interpretable rationales, positioning LALMs as a promising, auditable framework for unified SASV.

Approach

The authors formulate SASV as a three-way decision problem (target, non-target, spoof) and adapt pretrained LALMs (SALMONN-7B and Qwen2-Audio-7B) using LoRA fine-tuning. They employ a composite objective function combining cross-entropy for SASV decisions, Additive Angular Margin (AAM) loss for speaker separability, and binary cross-entropy for spoof detection, along with hard-sample mining. Reasoning-oriented training (Chain-of-Thought SFT and GRPO) is also explored to generate interpretable rationales and improve robustness.

Datasets

ASVspoof 5 dataset for training and evaluation. Bona fide utterances from VoxCeleb are used to enrich the genuine side of the training data.

Model(s)

SALMONN-7B, Qwen2-Audio-7B, ECAPA2, WavLM, Wav2Vec2-AASIST, Whisper, BEATS, Vicuna LLM backbone

Author countries

UNKNOWN

← Previous