Phantom: A Unified Face-Swap Deepfake Protection Framework with Latent and Spatial Constraints

Authors: Jungkon Kim, Cheolseung Jung, Jong-Min Choi, Juseong Lee

Published: 2026-06-30 14:10:38+00:00

Comment: Accepted to CVPR 2026 (Findings)

AI Summary

This paper introduces Phantom, a unified face-swap deepfake protection framework that jointly constrains perturbations in latent and spatial domains. Phantom adaptively synthesizes identity-shifted yet attribute-preserving targets to guide identity-aware latent optimization and applies masked perturbations to semantically relevant facial regions. The framework significantly improves protection success rates against various face-swapping deepfakes and generalizes to impersonation scenarios while enhancing visual quality.

Abstract

Face-swapping deepfakes pose an escalating threat to personal privacy by enabling unauthorized identity manipulation. While adversarial approaches have demonstrated success against black-box face recognition (FR) models, their applicability to face-swapping scenarios remains underexplored. In particular, reliance on fixed or random targets yields ambiguous latent guidance, and the lack of explicit spatial constraints causes perturbations to spill into identity-irrelevant regions. These issues are further exacerbated by identity-style disentanglement, which suppresses adversarial signals during deepfake generation. In this paper, we present Phantom, a unified face-swap deepfake protection framework that jointly constrains perturbations in latent and spatial domains. Phantom adaptively synthesizes identity-shifted yet attribute-preserving targets to guide identity-aware latent optimization, and applies masked perturbations confined to semantically relevant facial regions. Extensive experiments on state-of-the-art face-swapping deepfakes demonstrate that Phantom improves protection success rates in dodging scenarios by 27.8%, 25.6%, and 16.6% on UniFace, INSwapper, and SimSwap, respectively, while also enhancing visual quality. Furthermore, Phantom generalizes to impersonation scenario, yielding up to 10.2% higher protection while improving perceptual fidelity. These results underscore the effectiveness of jointly leveraging latent and spatial constraints for robust and coherent facial privacy protection.

Key findings

Phantom improves deepfake protection success rates in dodging scenarios by 27.8%, 25.6%, and 16.6% on UniFace, INSwapper, and SimSwap, respectively, while also enhancing visual quality. It also generalizes effectively to impersonation scenarios, yielding up to 10.2% higher protection. The method demonstrates strong robustness against face-swapping deepfakes by overcoming identity-style disentanglement.

Approach

Phantom employs adaptive target synthesis to generate identity-shifted but attribute-preserving target faces, guiding adversarial perturbations along identity-aware directions in latent space. Concurrently, it uses masked adversarial attacks to confine perturbations to semantically relevant facial regions. This dual constraint ensures effective identity obfuscation with minimal visual degradation.

Datasets

CelebA-HQ, LADN

Model(s)

IRSE50, IR152, FaceNet, MobileFace, RetinaFace, UniFace, INSwapper, SimSwap

Author countries

South Korea