SingFake: Singing Voice Deepfake Detection

Authors: Yongyi Zang, You Zhang, Mojtaba Heydari, Zhiyao Duan

Published: 2023-09-14 08:49:05+00:00

Comment: Accepted at ICASSP 2024

AI Summary

This paper introduces the Singing Voice Deepfake Detection (SVDD) task and presents SingFake, the first curated in-the-wild dataset for singing voice deepfakes. It evaluates state-of-the-art speech countermeasure systems, demonstrating their significant performance degradation on singing voices when trained on speech. However, retraining these systems on SingFake leads to substantial improvements, though challenges related to unseen singers, languages, and musical contexts remain.

Abstract

The rise of singing voice synthesis presents critical challenges to artists and industry stakeholders over unauthorized voice usage. Unlike synthesized speech, synthesized singing voices are typically released in songs containing strong background music that may hide synthesis artifacts. Additionally, singing voices present different acoustic and linguistic characteristics from speech utterances. These unique properties make singing voice deepfake detection a relevant but significantly different problem from synthetic speech detection. In this work, we propose the singing voice deepfake detection task. We first present SingFake, the first curated in-the-wild dataset consisting of 28.93 hours of bonafide and 29.40 hours of deepfake song clips in five languages from 40 singers. We provide a train/validation/test split where the test sets include various scenarios. We then use SingFake to evaluate four state-of-the-art speech countermeasure systems trained on speech utterances. We find these systems lag significantly behind their performance on speech test data. When trained on SingFake, either using separated vocal tracks or song mixtures, these systems show substantial improvement. However, our evaluations also identify challenges associated with unseen singers, communication codecs, languages, and musical contexts, calling for dedicated research into singing voice deepfake detection. The SingFake dataset and related resources are available at https://www.singfake.org/.

Key findings

Speech deepfake detection systems trained on speech data show severe performance degradation (EER near 50%) when evaluated on singing voice deepfakes. Retraining these systems on the SingFake dataset leads to substantial performance improvements, with EERs dropping significantly (e.g., to ~8-11% for best models). While showing robustness to unseen communication codecs, the systems still struggle with generalization to unseen singers, languages, and diverse musical contexts, indicating a need for specialized SVDD research.

Approach

The authors curate SingFake, a novel in-the-wild dataset of bonafide and deepfake singing voice clips, featuring both song mixtures and separated vocal tracks. They evaluate four state-of-the-art speech deepfake detection systems, first using models pre-trained on speech utterances and then after retraining them on the SingFake dataset, to assess their performance on the SVDD task across various challenging scenarios.

Datasets

SingFake, ASVspoof2019LA

Model(s)

AASIST, Spectrogram+ResNet18, LFCC+ResNet18, Wav2Vec2+AASIST

Author countries

USA

← Previous