DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models

View on arXiv ← Back to list

Authors: Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, Duen Horng Chau

Published: 2022-10-26 17:54:20+00:00

Comment: Accepted to ACL 2023 (nominated for best paper, top 1.6% of submissions, oral presentation). 17 pages, 11 figures. The dataset is available at https://huggingface.co/datasets/poloclub/diffusiondb. The code is at https://github.com/poloclub/diffusiondb. The interactive visualization demo is at https://poloclub.github.io/diffusiondb/explorer/

AI Summary

This paper introduces DiffusionDB, the first large-scale text-to-image prompt dataset, comprising 14 million Stable Diffusion-generated images, 1.8 million unique user prompts, and associated hyperparameters. The dataset facilitates research into understanding prompt-model interplay, detecting deepfakes, and designing human-AI interaction tools. The authors analyze prompt characteristics, pinpoint sources of model errors, and identify instances of potentially harmful model usage like misinformation generation.

Abstract

With recent advancements in diffusion models, users can generate high-quality images by writing text prompts in natural language. However, generating images with desired details requires proper prompts, and it is often unclear how a model reacts to different prompts or what the best prompts are. To help researchers tackle these critical challenges, we introduce DiffusionDB, the first large-scale text-to-image prompt dataset totaling 6.5TB, containing 14 million images generated by Stable Diffusion, 1.8 million unique prompts, and hyperparameters specified by real users. We analyze the syntactic and semantic characteristics of prompts. We pinpoint specific hyperparameter values and prompt styles that can lead to model errors and present evidence of potentially harmful model usage, such as the generation of misinformation. The unprecedented scale and diversity of this human-actuated dataset provide exciting research opportunities in understanding the interplay between prompts and generative models, detecting deepfakes, and designing human-AI interaction tools to help users more easily use these models. DiffusionDB is publicly available at: https://poloclub.github.io/diffusiondb.

Key findings
The analysis revealed common prompt patterns, distinct semantic categories for art and photography prompts, and a semantic misalignment between prompts and generated images in CLIP embedding space. They identified that specific hyperparameters (e.g., negative CFG scales, small steps/sizes) and prompt styles (e.g., short, non-English, emoji-only) lead to Stable Diffusion errors. Furthermore, evidence was found of potentially harmful model usage, such as generating misinformation and nonconsensual pornography.
Approach
The authors address the lack of large-scale text-to-image prompt datasets by collecting 14 million images and 1.8 million unique prompts with hyperparameters from the Stable Diffusion Discord server. They process this data, extracting metadata, applying NSFW content detectors, and organizing it into DiffusionDB to enable research into generative models.
Datasets
DiffusionDB (created by the authors, sourced from Stable Diffusion Discord server)
Model(s)
UNKNOWN
Author countries
USA
← Previous
Next →