Supervised Post-training of Speech Foundation Models for Robust Adaptation in Speech Deepfake Detection

View on arXiv ← Back to list

Authors: Zihan Pan, Sailor Hardik, Jinyang Wu

Published: 2026-06-24 02:49:17+00:00

AI Summary

This paper proposes a mix-frame post-training strategy for speech foundation models to improve deepfake detection by generating localized spoof-oriented perturbations and using frame-level supervision. This approach encourages the model to learn local inconsistencies crucial for robust spoof detection, overcoming limitations of direct fine-tuning of self-supervised pre-trained models. The method achieves state-of-the-art performance on ASVspoof5 and strong, balanced robustness across distinct distortion conditions on ASVspoof2021 LA/DF.

Abstract

Large speech foundation models have shown strong potential for speech deepfake detection, but direct fine-tuning is limited by a mismatch between self-supervised pre-training objectives and spoof-specific artifacts. To address this, we propose a mix-frame post-training strategy to create localized spoof-oriented perturbations and use frame-level supervision to encourage the SSL model to learn local inconsistencies that are critical for robust spoof detection. On ASVspoof5, we achieve state-of-the-art EER 4.50% for a single model without data augmentation. On ASVspoof2021 LA/DF, it further achieves only 0.16\\% absolute EER gap between LA and DF, indicating strong and balanced robustness across distinct distortion conditions. These results show that supervised post-training provides an effective and practical way to adapt speech foundation models for robust deepfake detection.

Key findings
The proposed supervised post-training strategy significantly improves robustness, especially in low-resource adaptation and out-of-domain evaluation scenarios. The method achieved a state-of-the-art EER of 4.50% on ASVspoof5 for a single model without data augmentation. Furthermore, it demonstrated strong and balanced performance on ASVspoof2021 LA/DF, with only a 0.16% absolute EER gap between LA and DF, indicating superior cross-condition stability compared to other competitive approaches.
Approach
The approach involves a three-stage process: first, 'mix-frame perturbation generation' creates localized spoof-oriented perturbations by splicing segments from an opposite-class utterance into a base utterance, generating frame-level labels. Second, 'supervised post-training' uses a lightweight frame classifier on top of the SSL model to learn spoof-discriminative local inconsistencies using these frame-level labels and LoRA adapters. Finally, 'fine-tuning' adapts the post-trained SSL model with an utterance-level task head and LoRA adapters for the final deepfake detection task.
Datasets
ASVspoof 2019 LA (ASV19LA), ASVspoof 2021 LA (ASV21LA), ASVspoof 2021 DeepFake (ASV21DF), ASVspoof 5 (ASV5)
Model(s)
WavLM Large (backbone encoder), BiLSTM-based classifier, ECAPA-TDNN, Nes2Net, Low-Rank Adaptation (LoRA)
Author countries
Singapore
← Previous
Next →