Too Emotional to Be Fake? Intra-Utterance Emotion Shifts Expose Failure Modes in Speech Deepfake Detectors

View on arXiv ← Back to list

Authors: Jingyi Chen, Surya Koppisetti

Published: 2026-10-16 02:19:40+00:00

Comment: Preprint. Under review

AI Summary

This research introduces EmoShift, a benchmark designed to assess the impact of intra-utterance emotion shifts on speech deepfake detection. It compares emotion-stable and emotion-changing speech, revealing that emotion changes can make synthetic speech appear more genuine, increasing spoof miss rates, while also raising false alarms for bona fide emotion-changing speech. These findings highlight the importance of considering emotion dynamics in deepfake detection.

Abstract

Expressive speech synthesis can now change emotion within a single utterance, yet its impact on deepfake detection remains unclear. We introduce EmoShift, a benchmark comparing emotion-stable (AA) and emotion-changing (AB) speech to examine these effects while accounting for static emotion identity and acoustic confounds. It comprises 7,240 bona fide utterances from four source datasets and 9,720 synthetic utterances generated using four expressive TTS systems. Across seven detectors, we find that emotion changes can make synthetic speech appear more genuine, increasing spoof miss rates by up to 25.3 percentage points on OMG-Emotion at fixed thresholds. Conversely, emotion-changing bona fide speech increases false-alarm rates by up to 24.4 percentage points on IEMOCAP. Further analyses reveal detector-specific sensitivities to emotion trajectories, not fully explained by static emotion identity or splicing artifacts. These findings underscore the importance of emotion dynamics in deepfake detection.

Key findings
Emotion changes within synthetic speech can significantly reduce its detectability, increasing spoof miss rates by up to 25.3 percentage points on OMG-Emotion. Conversely, emotion-changing bona fide speech can lead to an increase in false-alarm rates by up to 24.4 percentage points on IEMOCAP. The study also found detector-specific sensitivities to emotion trajectories that are not fully explained by static emotion identity or splicing artifacts, underscoring the need for deepfake detectors robust to emotion dynamics.
Approach
The authors developed EmoShift, a benchmark containing both bona fide and synthetic speech with either stable or changing emotions within an utterance. They evaluated seven deepfake detectors on this benchmark, comparing detector scores and error rates between emotion-stable and emotion-changing speech, while controlling for static emotion identity, linguistic content, splicing artifacts, and duration.
Datasets
EmoShift (custom benchmark, comprising ESD, OMG-Emotion, IEMOCAP, MSP-Podcast), ASVspoof 2019 LA
Model(s)
RawNet2, AASIST, XLS-R-SLS, XLS-R-Mamba, ProsDD, SLIM, Alethia
Author countries
UNKNOWN
← Previous
Next →