Anonymization Prompt Learning for Facial Privacy-Preserving Text-to-Image Generation

Authors: Liang Shi, Jie Zhang, Shiguang Shan

Published: 2024-05-27 07:38:26+00:00

Comment: 15 pages, 8 figures and 5 tables

AI Summary

This paper proposes Anonymization Prompt Learning (APL) to address privacy concerns arising from text-to-image diffusion models' ability to generate identifiable facial images. APL trains a learnable prompt prefix that forces the model to produce anonymized facial identities, even when specific individuals are prompted, without degrading the quality of non-identity-specific image generation. The learned prompt exhibits a plug-and-play property, enabling its transferability across various pretrained text-to-image models for comprehensive privacy and security protection against deepfakes.

Abstract

Text-to-image diffusion models, such as Stable Diffusion, generate highly realistic images from text descriptions. However, the generation of certain content at such high quality raises concerns. A prominent issue is the accurate depiction of identifiable facial images, which could lead to malicious deepfake generation and privacy violations. In this paper, we propose Anonymization Prompt Learning (APL) to address this problem. Specifically, we train a learnable prompt prefix for text-to-image diffusion models, which forces the model to generate anonymized facial identities, even when prompted to produce images of specific individuals. Extensive quantitative and qualitative experiments demonstrate the successful anonymization performance of APL, which anonymizes any specific individuals without compromising the quality of non-identity-specific image generation. Furthermore, we reveal the plug-and-play property of the learned prompt prefix, enabling its effective application across different pretrained text-to-image models for transferrable privacy and security protection against the risks of deepfakes.

Key findings

APL significantly reduces the identity accuracy of generated faces, achieving nearly a tenfold decrease compared to original models, demonstrating effective anonymization even for unseen identities and across various text-to-image models like Stable Diffusion v1.5, v1.4, RealisticVision v5.1, and SDXL. The method successfully preserves the quality of non-identity-specific image generation and text fidelity, outperforming existing concept editing approaches. Furthermore, the learned Anonymization Prompt is plug-and-play and remains effective even when new identities are introduced via personalization methods like Dreambooth, ensuring robust protection against deepfake threats.

Approach

The authors propose Anonymization Prompt Learning (APL), which involves training a learnable soft prompt prefix that is prepended to any text input for text-to-image diffusion models. For identity-specific prompts, this prefix guides the model to generate anonymized faces while preserving general facial attributes. For non-identity-specific prompts, the prefix is regularized to be inactive, ensuring the preservation of the model's original generation quality and text fidelity.

Datasets

SID (curated identity-specific dataset), Stest (unseen identity test set), LAION-2B (for regularization), MS COCO 2014 evaluation set (for quality evaluation), VGGFace-2 (for new identity learning experiments)

Model(s)

UNKNOWN

Author countries

China

← Previous