Progressive Layer-wise Supervision: Deep-to-Shallow Supervision Annealing for Efficient and Robust Speech Deepfake Detection

Authors: Hoan My Tran, Xin Wang, Xuechen Liu

Published: 2026-11-16 06:52:25+00:00

Comment: neurips 2026

AI Summary

This paper addresses the 'shallow-layer dormancy' in speech foundation models fine-tuned for deepfake detection, where only the deepest layers contribute meaningfully. The authors propose Progressive Layer-wise Supervision (PLS), a training framework that uses an exponentially annealed power weighting for layer-wise loss contributions, enabling earlier layers to become discriminative. PLS achieves state-of-the-art performance with reduced inference parameters and improved out-of-domain generalization.

Abstract

Speech foundation models pretrained via self-supervised learning provide powerful representations for deepfake detection, yet we find that standard fine-tuning exploits only the deepest transformer layers-leaving the majority of the network's capacity discriminatively inert. Layer-wise probing of XLS-R-128 reveals that the top two-thirds of the transformer stack contribute negligibly to anti-spoofing performance after standard fine-tuning, a phenomenon we term shallow-layer dormancy. We connect this observation to the representational redundancy of pretrained transformers: adjacent layers encode highly similar information, and fine-tuning without explicit intermediate supervision reinforces rather than breaks this redundancy. To address this, we propose Progressive Layer-wise Supervision (PLS), a training framework that attaches a shared linear classifier to every transformer layer and schedules layer-wise loss contributions via an exponentially annealed power weighting. Early training concentrates supervision on deep layers to preserve the pretrained representational hierarchy; later training progressively redistributes supervision to shallower layers, converting dormant earlier representations into discriminative ones. Competitive detection (<9.25% average EER) is achievable on XLS-R-128 at layer 7, which reduces the parameter count from 315M to 101M and exemplifies an early exit mechanism. Experiments across twelve benchmarks spanning in-domain, and out-of-domain deepfake scenarios show PLS achieves a new state-of-the-art with 239M parameters, outperforming prior methods with larger and more complex architectures.


Key findings
PLS significantly improves out-of-domain generalization, achieving a new state-of-the-art average EER of 6.03% with 239M parameters, outperforming prior methods with larger architectures. It also enables efficient early-exit inference, with competitive detection at layer 7 of XLS-R-128 (101M parameters), representing a 70% reduction in inference parameters. The annealing schedule effectively activates dormant shallow layers, distributing discriminative power across the network and increasing robustness to unseen spoofing scenarios.
Approach
PLS tackles shallow-layer dormancy by applying a shared linear classifier to every transformer layer and scheduling layer-wise loss contributions using an exponentially annealed power weighting. Early in training, supervision is concentrated on deep layers to stabilize representations, while later it progressively shifts to shallower layers to convert dormant representations into discriminative ones. An alternating epoch strategy ensures the primary classification objective remains anchored.
Datasets
ASVspoof 2019 logical access (19LA), ASVspoof5 (25DF), In-the-Wild (ITW), Fake-or-Real (FoR), SpoofCeleb (SC), LibriSeVoc (LSV), DFADD, DIFSSD, DECRO, MLAAD-EN, SpeechFake
Model(s)
XLS-R-128 (300M, 400M, 1B, 2B), mHuBERT-147 (base), XLS-R-300M (base, large, XL)
Author countries
France, Japan, China