Unmasking Puppeteers: Leveraging Biometric Leakage to Disarm Impersonation in AI-based Videoconferencing

View on arXiv ← Back to list

Authors: Danial Samadi Vahdati, Tai Duc Nguyen, Ekta Prashnani, Koki Nagano, David Luebke, Orazio Gallo, Matthew Stamm

Published: 2025-10-03 22:37:03+00:00

AI Summary

AI-based talking-head videoconferencing systems are vulnerable to 'puppeteering' attacks where an attacker hijacks a victim's likeness in real-time, which traditional deepfake detectors cannot address as every frame is synthetic. This paper introduces a biometric leakage defense that leverages identity cues within the transmitted pose-expression latent, using a pose-conditioned, large-margin contrastive encoder to disentangle persistent identity from transient pose and expression. A simple cosine similarity test on the disentangled embedding enables real-time detection of illicit identity swaps, outperforming existing defenses and generalizing well to unseen scenarios.

Abstract

AI-based talking-head videoconferencing systems reduce bandwidth by sending a compact pose-expression latent and re-synthesizing RGB at the receiver, but this latent can be puppeteered, letting an attacker hijack a victim's likeness in real time. Because every frame is synthetic, deepfake and synthetic video detectors fail outright. To address this security problem, we exploit a key observation: the pose-expression latent inherently contains biometric information of the driving identity. Therefore, we introduce the first biometric leakage defense without ever looking at the reconstructed RGB video: a pose-conditioned, large-margin contrastive encoder that isolates persistent identity cues inside the transmitted latent while cancelling transient pose and expression. A simple cosine test on this disentangled embedding flags illicit identity swaps as the video is rendered. Our experiments on multiple talking-head generation models show that our method consistently outperforms existing puppeteering defenses, operates in real-time, and shows strong generalization to out-of-distribution scenarios.

Key findings

The proposed method achieved state-of-the-art puppeteering detection, with AUC consistently above 0.97 across various datasets and generation methods, significantly outperforming prior work and deepfake detectors. It operates in real-time (75 FPS) and demonstrates strong generalization to unseen domains and robustness to facial appearance changes. Ablation studies confirmed the critical contribution of each component, including the EBL space, pose-conditioned contrastive loss, and temporal LSTM fusion.

Approach

The approach exploits biometric leakage in the transmitted pose-expression latent by re-encoding it into an Enhanced Biometric Leakage (EBL) space using a pose-conditioned, large-margin contrastive encoder. This encoder disentangles identity cues from pose and expression variability. Real-time detection is achieved by comparing the EBL embedding of the live stream with a reference portrait via a cosine test, further stabilized by an LSTM for temporal fusion.

Datasets

NVFAIR (NVIDIA VC, CREMA-D, RAVDESS)

Model(s)

UNKNOWN

Author countries

USA

← Previous