Exploring the Use of Abusive Generative AI Models on Civitai

Authors: Yiluo Wei, Yiming Zhu, Pan Hui, Gareth Tyson

Published: 2024-07-16 06:18:03+00:00

Comment: Accepted to ACM Multimedia 2024

AI Summary

This paper conducts the first comprehensive empirical study of Civitai, a prominent AI-Generated Content (AIGC) social platform, to investigate the use of generative AI models for creating abusive content. Analyzing a dataset of 87K models and 2M images, the study explores content characteristics, user engagement, and creator network positions to inform platform moderation strategies.

Abstract

The rise of generative AI is transforming the landscape of digital imagery, and exerting a significant influence on online creative communities. This has led to the emergence of AI-Generated Content (AIGC) social platforms, such as Civitai. These distinctive social platforms allow users to build and share their own generative AI models, thereby enhancing the potential for more diverse artistic expression. Designed in the vein of social networks, they also provide artists with the means to showcase their creations (generated from the models), engage in discussions, and obtain feedback, thus nurturing a sense of community. Yet, this openness also raises concerns about the abuse of such platforms, e.g., using models to disseminate deceptive deepfakes or infringe upon copyrights. To explore this, we conduct the first comprehensive empirical study of an AIGC social platform, focusing on its use for generating abusive content. As an exemplar, we construct a comprehensive dataset covering Civitai, the largest available AIGC social platform. Based on this dataset of 87K models and 2M images, we explore the characteristics of content and discuss strategies for moderation to better govern these platforms.

Key findings

The study reveals a prevalence of abusive content, with 16.97% of models and 72.05% of images containing NSFW tags, and deepfakes making up 23.54% of models and 32.98% of images, often correlated with NSFW content and targeting celebrities. NSFW models and images receive significantly more engagement (downloads/views, favorites, tips), and creators of abusive content demonstrate higher centrality in the social network. Non-NSFW models are also frequently repurposed for NSFW content via prompting.

Approach

The authors collected metadata for models, images, and creators from Civitai, augmenting it with thematic categories and person names using ChatGPT, and NSFW scores for prompts using OpenAI's moderation API. They then performed an empirical analysis, investigating content themes, model popularity, user engagement metrics, and creator network centrality to characterize abusive content.

Datasets

Civitai dataset (87,042 generative models, 2,740,149 AI-generated images, 56,779 creators), DiffusionDB (from Stable Diffusion Discord), JourneyDB (from Midjourney).

Model(s)

UNKNOWN

Author countries

Hong Kong, Finland, United Kingdom

← Previous