Linguistically Augmented Audio Speech Data (LinguAS)

Authors: Ashley R. Keaton, Zahra Khanjani, Christine Mallinson, Vandana P. Janeja

Published: 2026-06-08 23:26:39+00:00

AI Summary

This paper introduces Linguistically Augmented Audio Speech Data (LinguAS), a novel dataset of genuine and deepfaked audio samples annotated with five Expert-Defined Linguistic Features (EDLFs). LinguAS aims to enhance audio deepfake detection by providing linguistic cues at larger timescales, demonstrating that models augmented with these features achieve significantly improved performance over traditional acoustic-feature-based and self-supervised learning baselines. The dataset also includes metadata on speaker gender and generator, emphasizing real human language traits for more robust model inference.

Abstract

Maliciously-created fake speech, including deepfaked and spoofed audio, is proliferating at an alarming rate, and detection models are racing to stay ahead of the curve. Yet, most detection models are trained to make inference on frame-level audio features alone without leveraging valuable linguistic cues at larger timescales. To address this gap, we present Linguistically Augmented Audio Speech Data (LinguAS), a dataset of genuine and deepfaked audio samples annotated with five strategically-chosen, Expert-Defined Linguistic Features (EDLFs) that occur frequently in spoken English and are characteristic of natural human speech. LinguAS contains over 800 audio samples, each of which are annotated with EDLFs. The dataset has a balanced number of four spoofed audio attack types and a proportionate number of genuine speech samples. We also include metadata on speaker gender and the generator/source for each spoofed audio sample, offering more granularity for model training. We found that models trained on data augmented with EDLFs had improved model performance significantly beyond the ASVspoof 2021 deep learning baselines and SSL models like HuBert and XLSR. LinguAS's augmented linguistic, gender, and generator metadata provide audio deepfake researchers with a dataset that emphasizes real human language traits to improve model inference of faked speech. Data and code are publicly available.

Key findings

Models trained on data augmented with Expert-Defined Linguistic Features (EDLFs) significantly improved deepfake detection performance beyond ASVspoof 2021 deep learning baselines and SSL models like HuBert and XLSR. Ensemble models combining EDLF classification with existing deep learning models consistently showed increased prediction accuracy and lower Equal Error Rates. The 'Audio Quality Anomaly' EDLF was identified as particularly essential for optimal model performance, highlighting the value of human-interpretable linguistic features.

Approach

The authors developed the LinguAS dataset, which comprises over 800 genuine and deepfaked audio samples, balanced across four spoofed audio attack types and speaker gender. The core innovation is the augmentation of these samples with five Expert-Defined Linguistic Features (EDLFs) that are characteristic of natural human speech. They then validate the effectiveness of these linguistic features by training various machine learning and deep learning models, including ensemble methods, to demonstrate improved deepfake detection performance.

Datasets

LinguAS (newly introduced), LJSpeech, ASSEM-VC, ASVspoof 2021, ASVspoof 2017, FakeOrReal (FoR), Google TTS, MelGan, WaveNet, Descript, ResembleAI, PPG, Mellotron, Cotatron, YouTube videos of public figures.

Model(s)

Logistic Regression, Multilayer Perceptron, Support Vector Machine, Random Forest, XGBoost, LFCC Gaussian Mixture Model (GMM), LFCC Light Convolutional Neural Network (LCNN), RawNet2, VGGish, HuBert, XLSR.

Author countries

USA

← Previous