Probing-Guided Layer Selection from Self-Supervised Speech Models for Generalizable Audio Deepfake Detection

Authors: Marjan Beheshti, Majid Rostami, Bo Chen

Published: 2026-06-29 18:19:37+00:00

Comment: Submitted to Computer Speech & Language

AI Summary

This paper introduces a two-stage, model-agnostic methodology for generalizable audio deepfake detection using self-supervised speech models. The approach first identifies informative layers within a frozen backbone using lightweight XGBoost probes and then fuses the selected layers with a compact neural classifier. This method achieves significant performance improvements and better cross-domain generalization compared to existing techniques.

Abstract

Audio deepfake detection systems often fail to generalize across domains because they rely on features tied to specific attacks or recording conditions. Self-supervised speech models offer rich multi-layer representations, yet existing approaches either use a single layer or fuse all layers indiscriminately, and only reveal layer importance after training. We propose a model-agnostic, two-stage methodology that identifies informative depth zones before any task-specific model is trained. In the first stage, lightweight XGBoost probes evaluate each transformer layer's cross-domain discriminative power, producing a layer ranking. In the second stage, a compact neural classifier fuses only the selected layers through per-layer attention pooling and a shared bottleneck projection, while the backbone remains frozen. Applied across three backbones, the probing reveals two key findings. First, informative layers cluster in depth zones rather than at uniquely optimal positions: within-zone substitutions fall within multi-seed noise, while zone violations degrade performance by up to 5x. Second, the probing produces backbone-specific selections rather than a fixed layer recipe. On XLS-R-300M, four probing-selected layers with 1.34M trainable parameters achieve 4.94 +/- 0.32% equal error rate on In-The-Wild and 5.07% cross-domain average over four shared datasets, a 28% relative improvement over the best prior frozen-backbone result (Xiao and Vu, 2025) using all 25 layers with identical training data.

Key findings

Informative layers for audio deepfake detection cluster in depth zones rather than uniquely optimal positions, and within-zone substitutions perform similarly, while zone violations degrade performance significantly. The probing methodology produces backbone-specific layer selections, confirming that a fixed layer recipe does not generalize across different self-supervised models. The approach achieved a 28% relative improvement in EER on In-The-Wild compared to prior frozen-backbone methods, using only four selected layers with 1.34M trainable parameters.

Approach

The proposed method consists of two stages. First, lightweight XGBoost probes evaluate the cross-domain discriminative power of each transformer layer from a frozen self-supervised speech model, resulting in a layer ranking and selection of a compact subset of layers. Second, a compact neural classifier fuses only these selected layers through per-layer attention pooling and a shared bottleneck projection, with the backbone remaining frozen.

Datasets

ASVspoof 2019 LA Train, ASVspoof 2019 LA Dev, ASVspoof 2019 LA Eval, In-The-Wild, ASVspoof 2021 DF, FakeAVCeleb, WaveFake, ASVspoof5 Eval, ASVspoof5 Dev

Model(s)

XLS-R-300M, WavLM Large, XLSR-53 (as backbones); XGBoost (for probing); a compact neural classifier (MLP for feature fusion and classification)

Author countries

USA

← Previous