Beyond Speech: Dual-Domain SSL Fusion for Unified All-Type Audio Deepfake Detection

Authors: Cunhang Fan, Junqin Cao, Tian Gao, Zhipeng Xie, Jun Xue, Zhao Lv, Xin Fang

Published: 2026-08-29 03:29:47+00:00

Comment: 8 pages, 5 figures; accepted to the AT-ADD Grand Challenge at ACM Multimedia 2026 (MM '26)

AI Summary

This paper introduces a dual-domain SSL fusion method for unified all-type audio deepfake detection, addressing the limitations of existing speech-centric solutions when audio type is unknown. The approach combines EAT-large and wav2vec 2.0 XLS-R-300M representations, integrating them through layer-wise weighted fusion and token-level fusion, and classifies the fused features with an MLP head. The proposed system achieved 95.58% Macro-F1 on the AT-ADD Track 2 evaluation set, ranking second in the challenge.

Abstract

Unified all-type audio deepfake detection aims to determine whether an input clip is real or fake when its audio type may be speech, environmental sound, singing voice, or music. Existing speech-centric or type-dependent solutions are insufficient for this setting because the test-time audio type is unknown, while the required output is still a single binary decision. To address these issues, this paper proposes a dual-domain SSL fusion method that maps heterogeneous audio into a shared binary authenticity space. EAT-large and wav2vec 2.0 XLS-R-300M are used as complementary SSL feature sources, providing broad acoustic and event-level representations as well as waveform-level, vocal, and speech-sensitive representations. Layer-wise weighted fusion integrates multi-level artifacts from different transformer depths, while token-level fusion forms a unified feature pool without enforcing frame-level alignment between the two SSL streams. The fused tokens are summarized by multi-head attentive statistics pooling and classified with a binary MLP head. With conservative speech refinement applied on top of this unified core detector, the submitted system achieves 95.58% Macro-F1 on the AT-ADD Track 2 evaluation set and ranks second in the challenge.

Key findings

The dual-domain SSL fusion effectively handles diverse audio types (speech, environmental sound, singing voice, music) for deepfake detection, outperforming single-modality baselines. The integration of complementary SSL models through layer-wise and token-level fusion, coupled with conservative speech refinement, significantly improved detection performance. The system achieved 95.58% Macro-F1 on the AT-ADD Track 2 evaluation set, demonstrating that a shared detector can utilize type-sensitive cues without requiring type-dependent inference paths.

Approach

The method fuses features from two self-supervised learning (SSL) models, EAT-large (for broad acoustic representations) and wav2vec 2.0 XLS-R-300M (for speech-sensitive representations). Layer-wise weighted fusion combines representations from different transformer depths, followed by token-level fusion to create a unified feature pool without enforcing frame-level alignment. This fused representation is then passed through multi-head attentive statistics pooling and classified by an MLP head.

Datasets

AT-ADD Track 2 evaluation set, AT-ADD training and development sets

Model(s)

EAT-large, wav2vec 2.0 XLS-R-300M, Whisper-large-v3

Author countries

China

← Previous