Ethical and Technical Limits of Deepfake Speech Datasets

Authors: Vojtěch Staněk, Eva Trnovská, Kamil Malinka, Anton Firc

Published: 2026-06-09 14:20:55+00:00

Comment: Accepted to Interspeech 2026

AI Summary

This paper presents a comprehensive dataset-level audit of 39 deepfake speech datasets to assess their suitability for robust and fair deepfake speech detection. The audit reveals significant limitations primarily due to a lack of demographic metadata for fairness assessment and substantial overlap in underlying bona fide speech sources, which undermines cross-dataset evaluation. The authors advocate for improved dataset documentation and practices to enable more reliable and non-discriminatory deepfake speech detection.

Abstract

Claims about the robustness and fairness of deepfake speech detectors are only as credible as the datasets used to train and evaluate those systems. We present a dataset-level audit of the deepfake speech landscape. We compile and analyze 39 deepfake speech datasets, examining key attributes including accessibility, documentation, demographic and language coverage, dataset scale, and the underlying bona fide speech sources. Our audit reveals two important takeaways. Firstly, fairness assessment is largely infeasible because most datasets lack demographic metadata, and only a few contain gender or language labels. This prevents any meaningful subgroup analysis and leaves other demographic attributes unaddressed. Secondly, we identify substantial overlap in underlying bona fide source corpora across datasets, which can undermine cross-dataset evaluation and lead to overstated generalization claims.

Key findings

The audit found that fairness assessment is largely infeasible due to the pervasive lack of demographic metadata in most datasets, with only a few providing gender or language labels. Additionally, a substantial overlap exists in underlying bona fide source corpora across datasets, potentially leading to inflated generalization claims and biased cross-dataset evaluations. Incomplete synthesis documentation, restricted access, and unclear licensing also hinder reproducibility and practical adoption.

Approach

The authors conducted an audit of 39 deepfake speech datasets, analyzing key attributes such as accessibility, documentation, demographic and language coverage, dataset scale, and underlying bona fide speech sources. They mapped overlaps in the source corpora and assessed the feasibility of fairness evaluation based on available metadata.

Datasets

39 deepfake speech datasets (e.g., VCC 2016, ASVspoof 2019 LA, Half-Truth, MLAAD, ASVspoof 5, SCDF).

Model(s)

UNKNOWN

Author countries

Czech Republic

← Previous