Evaluating Pre-trained Speech Encoders for Spontaneous Speech Detection and Out of Domain Synthetic Speech Generalisation in Indic Languages

Authors: Varun Rai, Pavan Kumar J, Sujith Pulikodan, Nihar Desai

Published: 2026-08-12 19:09:10+00:00

AI Summary

This paper evaluates five frozen transformer encoders for distinguishing spontaneous from scripted speech and natural from synthetic speech across 22 Indic languages. It introduces language isolation probing and centroid proximity analysis to explain encoder behavior and deepfake generalization failures. The study finds an encoder-dependent trade-off between language discriminability and spontaneity detection, and that out-of-domain generalization for deepfake detection is predicted by the proximity of training system embeddings to unseen TTS embeddings.

Abstract

Transformer-based models have shown strong accuracy in distinguishing spontaneous from scripted speech and natural from synthetic speech, but these results are established on a narrow set of well-resourced language benchmarks and have not been extended across Indic languages, nor has embedding geometry been used to explain encoder behaviour or deepfake generalisation failure. We address these gaps by evaluating five frozen transformer encoders, AST, Vaani-FastConformer, Wav2vec2, Whisper and BEATs, across 22 Indic languages, and by conducting a multi-system TTS generalisation experiment across four TTS models. Beyond accuracy, we present language isolation probing and centroid proximity analysis. Probing reveals an encoder-dependent trade-off between language-discriminability and spontaneity detection. Centroid analysis shows that out-of-domain generalisation is predicted by a training system's proximity to unseen TTS embeddings, not its distance from natural speech, a finding with direct implications for training data selection in real-world deepfake detectors.

Key findings

There is an encoder-dependent trade-off between language-discriminability and spontaneity detection, with Whisper and Vaani showing high accuracy decoupled from language-specific structure. Training diversity is crucial for out-of-domain deepfake generalization, improving recall from 7% to 51%. Generalization is best predicted by the proximity of training TTS embeddings to unseen TTS embeddings, rather than their distance from natural speech.

Approach

The authors evaluate pre-trained speech encoders on two tasks: spontaneous vs. scripted speech detection and natural vs. synthetic speech detection. They train lightweight DNN classifiers on frozen embeddings from various transformer models. Additionally, they perform language isolation probing and centroid proximity analysis to understand embedding space characteristics and their impact on performance and generalization.

Datasets

IndicVoices, IEMOCAP, Indic F5, Indic VITS, OmniVoice, Meta M4, freevc24, xttsv2, IndicSynth

Model(s)

AST, Vaani-FastConformer, Wav2vec2, Whisper, BEATs

Author countries

India

← Previous