MUTE: Multi-Level Alignment Uncoupling Against Talking-Head Exploitation for Voice Protection

Authors: Donghyun Kim, Jin Hong, Seungmin Kim, Dain Kim, Junseok Kwon, Daeseon Choi

Published: 2026-11-09 07:16:57+00:00

Comment: Accepted at NeurIPS 2026. 33 pages, including appendix and checklist. Project and code: https://github.com/jindl465/MUTE

AI Summary

This paper proposes MUTE, an audio protection framework designed to prevent the misuse of talking-head generation models by disrupting audio-visual alignment. MUTE employs a multi-level strategy combining alignment-level disruption of cross-modal attention and representation-level degradation of temporal audio embeddings. It achieves robustness through STFT-domain perturbations in high-energy regions and optionally mitigates TTS-based resynthesis via a speaker-level objective.

Abstract

Talking-head generation models, which synthesize realistic facial animations from audio, are increasingly vulnerable to misuse in multimodal deepfake scenarios. Protecting such systems remains challenging, as talking-head models fundamentally rely on precise audio-visual alignment, while effective audio protection methods remain largely underexplored. In this work, we propose MUTE, an audio protection framework that uncouples the underlying audio-visual alignment through a multi-level strategy. Specifically, MUTE combines (1) Alignment-level Disruption, which directly perturbs cross-modal attention to disrupt structural alignment, and (2) Representation-level Degradation, which perturbs temporal audio embeddings to degrade their representations. To improve robustness, perturbations are constrained in the STFT domain to high-energy regions, making them resistant to post-processing, denoising, and adaptive attacks. An optional speaker-level objective further mitigates potential bypass via TTS-based resynthesis. Extensive experiments demonstrate that MUTE consistently degrades lip synchronization across both white-box and black-box talking-head models while preserving perceptual audio quality. The proposed method remains effective under real-world transformations and can be combined with image-based approaches to provide stronger multimodal protection. The source code and project page are available at https://github.com/jindl465/MUTE.


Key findings
MUTE significantly outperforms baselines in disrupting audio-visual alignment (lowest SyncNet, highest M-LMD) while preserving perceptual audio quality. It demonstrates strong transferability across various black-box talking-head models and maintains effectiveness under real-world transformations such as post-processing, denoising, and adaptive attacks. Furthermore, MUTE can be effectively combined with image-based protection methods for stronger multimodal defense and its optional speaker-level objective mitigates TTS-based resynthesis attacks.
Approach
MUTE introduces adversarial perturbations into audio signals to uncouple audio-visual alignment in talking-head generation. It does this by using a two-phase optimization: first, it disrupts the cross-modal attention map (Alignment-level Disruption) and then degrades temporal audio embeddings (Representation-level Degradation). Perturbations are constrained in the STFT domain to high-energy regions for robustness, and an optional speaker-level objective can be added for TTS-based resynthesis mitigation.
Datasets
TalkingHead-1KH, CelebV-HQ
Model(s)
Hallo (surrogate model), Hallo2, Audio2Head, SadTalker, Ditto (target models), wav2vec2.0 (audio encoder), SyncNet (for alignment evaluation), M-LMD (for lip landmark distance), Whisper (for WER), CAM++, ResNet-based speaker encoder (for speaker-level objective), Demucs, WavePurifier (for denoising evaluation)
Author countries
UNKNOWN