Phonetically Explainable Speech Deepfake Detection

Authors: Manasi Chhibber, Jagabandhu Mishra, Tomi H. Kinnunen

Published: 2026-06-13 20:13:28+00:00

AI Summary

This paper proposes a phoneme-guided cross-attention framework for interpretable speech deepfake detection, moving beyond opaque classification by explicitly decomposing the spoofing posterior. It factorizes the detection objective into phone-conditional acoustic evidence modulated by continuous phonetic presence weights, providing phonetic-explainability-by-design. The framework reveals that discriminative power concentrates on articulatory categories that generative models struggle to faithfully reproduce, such as stops and fricatives, while achieving competitive detection performance.

Abstract

Speech deepfake detection is predominantly treated as an opaque classification task where all temporal frames are aggregated equally. This ignores that different phonetic categories carry vastly different amounts of discriminative information. To address this, we propose a phoneme-guided cross-attention framework that transforms detection into an interpretable, phonetically grounded process. We factorize the spoofing posterior $P(\\text{spoofed}\\mid X, W)$, conditioned on the acoustic representation $X$ and the phonetic posteriorgram $W$. The resulting factorization can be written as $P(\\text{spoofed} \\mid X, W) = \\sum_{i=1}^{M} w_i \\cdot P(\\text{spoofed} \\mid X, Z = z_i)$, where $M$ denotes the number of phonetic classes, $P(\\text{spoofed} \\mid X, Z = z_i)$ is the spoofing probability for the $i$-th phonetic class $z_i$ conditioned on $X$, and each $w_i$ is the prevalence of phonetic class $z_i$ in the utterance. Our transformer-based architecture instantiates this through a cross-attention block in which phonetic queries selectively probe information in acoustic keys and values, with softmax-normalized pooling supplying explicit phone-presence weights. Unlike prior approaches that rely heavily on post-hoc explainability methods, our framework offers phonetic-explainability-by-design. We evaluate the framework on an LJSpeech-derived corpus, ASVspoof 2019 LA, and ASVspoof 5 Track 1. Per-phone importance rankings reveal that discriminative power concentrates on articulatory categories that generative models struggle to reproduce faithfully. Stops, fricatives, affricates, nasals, and silence-boundary closures rank most discriminative, while periodic vowels and semivowels rank lower. Beyond competitive performance, our model provides structural interpretability, yielding an inspectable per-articulatory category breakdown of the final verdict.

Key findings

The framework consistently found that discriminative power for deepfake detection concentrates on articulatory categories like stops, fricatives, affricates, nasals, and silence-boundary closures, which generative models struggle to reproduce faithfully. Conversely, periodic vowels and semivowels, which are well-synthesized, contribute less discriminative information. The model provides structural interpretability, offering a per-articulatory category breakdown of the detection decision while maintaining competitive performance across diverse datasets.

Approach

The approach utilizes a phoneme-guided cross-attention framework that probabilistically factorizes the spoofing posterior into a weighted sum of phone-conditional spoofing probabilities and phonetic presence weights. It fuses self-supervised acoustic representations (XLS-R) with phonetic posteriorgrams (PPGs) via a cross-attention mechanism where phonetic queries probe acoustic keys and values. A learned weighted pooling mechanism then explicitly assigns per-phone weights to aggregate this evidence for the final binary classification.

Datasets

LJSpeech-derived corpus, ASVspoof 2019 LA, ASVspoof 5 Track 1

Model(s)

Phoneme-guided cross-attention framework, XLS-R (300M, Multilingual) for acoustic features, Wav2Vec 2.0 (Large, TIMIT fine-tuned) for phonetic posteriorgram extraction, Transformer-based architecture, Multi-layer perceptron (MLP)

Author countries

Finland

← Previous