Phoneme-Level Deepfake Detection Across Emotional Conditions Using Self-Supervised Embeddings

Authors: Vamshi Nallaguntla, Shruti Kshirsagar, Anderson R. Avila

Published: 2026-05-04 18:49:29+00:00

Comment: 6 pages, 2 figures, submitted to IEEE SMC 2026

AI Summary

This work proposes a phoneme-level framework for detecting emotionally manipulated synthetic speech, addressing the limitations of utterance-level approaches in emotionally conditioned settings. It analyzes real and emotional voice conversion (EVC)-generated speech using self-supervised WavLM embeddings and phoneme-aligned data. The study demonstrates that phoneme-level analysis is an effective and interpretable method for identifying emotionally manipulated deepfakes.

Abstract

Recent advances in emotional voice conversion (EVC) have enabled the generation of expressive synthetic speech, raising new concerns in audio deepfake detection. Existing approaches treat speech as a homogeneous signal and largely overlook its internal phonetic structure, limiting their interpretability in emotionally conditioned settings. In this work, we propose a phoneme-level framework to analyze emotionally manipulated synthetic speech using real and EVC-generated speech under matched emotional conditions with shared transcripts, phoneme-aligned TextGrids, and WavLM-based embeddings. Our results show that phoneme behavior varies across categories, with complex vowels and fricatives exhibiting higher divergence while simpler phonemes remain more stable. Phonemes with larger distributional differences are also found to be more easily detected, consistently across multiple emotions and synthesis systems. These findings demonstrate that phoneme-level analysis is an effective and interpretable approach for detecting emotionally manipulated synthetic speech.

Key findings

The study found that complex vowels and fricatives exhibit higher distributional divergence and classification accuracy, indicating greater sensitivity to EVC artifacts. Simpler phonemes, like monophthongs and plosives, show more stability. A generally positive correlation was observed between Kullback–Leibler divergence and classification performance, suggesting that phonemes with larger distributional differences are more easily detectable.

Approach

The authors propose a phoneme-level analysis framework where real and synthetic emotional speech are segmented into phonemes using the Montreal Forced Aligner. WavLM embeddings are extracted for each phoneme segment, and distributional differences are quantified using symmetric Kullback–Leibler divergence. Phoneme-wise binary classification is then performed using an RBF-kernel Support Vector Machine.

Datasets

EmoFake dataset, Emotional Speech Dataset (ESD)

Model(s)

WavLM (for feature extraction), Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel (for classification)

Author countries

USA, Canada

← Previous