BabelFake: A Multilingual Audio-Visual DeepFake Benchmark

Authors: Carlotta Segna, Joel Tschesche, Anna Rohrbach

Published: 2026-10-05 13:42:34+00:00

Comment: 25 pages (8 main paper + ack), 25 pages total, 8 figures, under submission

AI Summary

This paper introduces BabelFake, a multilingual audio-visual DeepFake benchmark featuring 399k clips (1,323 hours) from 496 consenting individuals across five languages. The benchmark utilizes a modular data generation pipeline combining 11 video manipulation methods with 4 voice cloning engines, distinguishing visual-only and joint audio-visual manipulations. Benchmarking state-of-the-art detectors on BabelFake reveals that detection difficulty is highly dependent on the audio-visual generation pairing, with significant performance degradation when authentic audio is preserved.

Abstract

Reliable and practical audio-visual DeepFake detection requires benchmarks that reflect diverse linguistic contexts and modern data synthesis pipelines for visual as well as audio manipulations. However, existing datasets predominantly contain footage of English-speakers, often include outdated manipulation types, or overlook the audio modality. Further, many datasets feature individuals who did not consent to be used in DeepFake creation. We introduce BabelFake, a multilingual audio-visual DeepFake benchmark recorded with consenting participants. BabelFake contains 399k clips (1,323 hours) from 496 individuals spanning five languages (English, German, Italian, French, Spanish). Our modular data generation pipeline pairs 11 modern video manipulation methods with 4 voice cloning engines, distinguishing visual-only (face swapping) and joint audio-visual manipulations (lip synchronization and portrait animation). By benchmarking state-of-the-art detectors, we show that detection difficulty depends on the audio-visual generation pairing, with substantial performance degradation when authentic audio is preserved. Cross-language/demographic evaluation reveals sensitivity varying across detector architectures and training data, while human evaluation reveals that perceived realism and machine-detection difficulty do not necessarily align.

Key findings

Detection difficulty strongly depends on the audio-visual generation pairing, with substantial performance degradation observed when authentic audio is preserved. Multimodal models generally outperform unimodal counterparts, though their advantage varies across audio-video generation combinations. Human perception of realism and machine detection difficulty do not always align.

Approach

BabelFake creates a dataset of diverse DeepFake manipulations by using a modular data generation pipeline. This pipeline pairs 11 modern video manipulation methods (face swapping, lip synchronization, and portrait animation) with 4 voice cloning engines, generating both visual-only and joint audio-visual manipulations. This comprehensive dataset is then used to benchmark existing state-of-the-art DeepFake detectors to evaluate their performance across different languages and manipulation types.

Datasets

BabelFake (newly introduced), DeepSpeak, MAVOS-DD, AV-DeepFake-1M, FakeAVCeleb, LRS3, VoxCeleb2

Model(s)

SpeechForensics, AVH-Align, (B)RAVEn

Author countries

Germany

← Previous