Effect of AI Performance, Risk Perception, and Trust on Human Dependence in Deepfake Detection AI system

Authors: Yingfan Zhou, Ester Chen, Manasa Pisipati, Aiping Xiong, Sarah Rajtmajer

Published: 2025-08-03 20:00:10+00:00

AI Summary

This study investigates how AI performance influences human trust and decision-making in deepfake detection. Through an online experiment with 400 participants, the researchers examined how varying AI performance (false positive rate) and perceived risk impact human dependence when identifying synthetic images. Findings indicate that participants calibrate their dependence on AI based on perceived risk and the AI's prediction results, even when their overall trust in AI is low.

Abstract

Synthetic images, audio, and video can now be generated and edited by Artificial Intelligence (AI). In particular, the malicious use of synthetic data has raised concerns about potential harms to cybersecurity, personal privacy, and public trust. Although AI-based detection tools exist to help identify synthetic content, their limitations often lead to user mistrust and confusion between real and fake content. This study examines the role of AI performance in influencing human trust and decision making in synthetic data identification. Through an online human subject experiment involving 400 participants, we examined how varying AI performance impacts human trust and dependence on AI in deepfake detection. Our findings indicate how participants calibrate their dependence on AI based on their perceived risk and the prediction results provided by AI. These insights contribute to the development of transparent and explainable AI systems that better support everyday users in mitigating the harms of synthetic media.

Key findings

While participants' trust increased with lower AI false positive rates, their overall trust in AI remained low, which did not significantly impact their willingness to follow AI decisions. A significant three-way interaction between perceived risk, AI performance, and AI predictions showed that human dependence is dynamically calibrated. Under high perceived risk, participants were more likely to comply with high-FPR AI systems for synthetic predictions, suggesting a preference for over-detection to avoid potential threats.

Approach

The researchers conducted an online human subject experiment with 400 participants using a 2x2 split-plot design. They manipulated notional AI deepfake detection performance (high vs. low False Positive Rate) and participant-perceived risk (high vs. low) to study their impact on human trust and dependence when identifying synthetic images. Statistical analysis, including mixed ANOVA and moderated mediation analysis, was used to evaluate the relationships.

Datasets

Images sourced from a publicly available scientific dataset [37] (Nightingale and Farid, 2022).

Model(s)

UNKNOWN

Author countries

USA

← Previous