RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models

Authors: Ruinan Jin, Xinting Liao, Hanlin Yu, Deval Pandya, Xiaoxiao Li

Published: 2026-01-31 01:38:55+00:00

Comment: 65 pages, 10 figures

AI Summary

This paper introduces RVCBench, a comprehensive dataset and benchmark designed to evaluate the robustness of modern voice cloning (zero-shot text-to-speech) models under realistic deployment conditions. It features 18 robustness evaluations across 10 tasks, 225 speakers, and 14,370 utterances, covering input sensitivity, generation stability, output resilience, and perturbation robustness. The evaluation of 18 representative open-source voice cloning models reveals systematic vulnerabilities across various realistic scenarios.

Abstract

Modern voice cloning, also known as zero-shot text-to-speech (TTS), can synthesize speech that closely matches a target speaker from only seconds of reference audio, enabling applications such as personalized speech interfaces and dubbing. In practice, these systems often face noisy reference audio, imperfect text prompts, multilingual and long-form generation, post-processing, and adversarial perturbations, all of which can weaken robustness. Despite rapid progress in codec-token language models and diffusion-based TTS, robustness under realistic deployment shifts remains underexplored. This paper introduces RVCBench, a comprehensive dataset and benchmark for evaluating robustness in voice cloning. RVCBench provides task-aligned tests covering controlled text-audio pairing, multilingual and long-form scenarios, expressive prompts, post-processing conditions, and passive or proactive audio perturbations. Across 18 robustness evaluations, 225 speakers, and 14,370 utterances, RVCBench supports unified evaluation of input sensitivity, generation stability, output resilience, perturbation robustness, speaker similarity, and deepfake detectability. We evaluate 18 representative open-source voice cloning models and reveal systematic vulnerabilities in content consistency, speaker similarity, long-form stability, post-processing resilience, adversarial robustness, and detector-facing separability. We release the code and dataset to support reproducible evaluation and future research on robust voice cloning, speech synthesis, and audio generation. Code: https://github.com/Nanboy-Ronan/RVCBench. Dataset: https://huggingface.co/datasets/Nanboy/RVCBench.

Key findings

The study reveals systematic vulnerabilities in modern voice cloning models, showing significant degradation under realistic deployment shifts. These include poor content consistency and speaker similarity with varied reference audios and text prompts, instability in long-form and multilingual generation, and weak resilience to post-processing and both passive and proactive audio perturbations. This highlights that robustness is a critical, underexplored aspect distinct from overall generation quality.

Approach

The authors developed RVCBench, a comprehensive dataset and benchmark for evaluating robustness in voice cloning. This benchmark provides task-aligned tests covering controlled text-audio pairing, multilingual and long-form scenarios, expressive prompts, post-processing conditions, and passive or proactive audio perturbations. They then evaluate 18 representative open-source voice cloning models across these 18 robustness evaluations.

Datasets

VCTK, LibriTTS, AISHELL-1, EMIME bilingual English-Mandarin, Common Voice FR, LibriSpeech-Long, VoiceBank+DEMAND, Multispeaker Libri, Robocall, Audio Compression Dataset, Hallucination dataset

Model(s)

SQ-LLM (SpeechLLM-as-Judges), XLSR-SLS, Wav2Vec2-ECAPA, HuBERT-ECAPA, WavLM-ECAPA, TCM-ADD, RawGAT-ST, AASIST, RawNet2

Author countries

Canada

← Previous