Beyond Seeing Is Believing: On Crowdsourced Detection of Audiovisual Deepfakes

Authors: Michael Soprano, Andrea Cioci, Stefano Mizzaro

Published: 2026-05-06 11:48:04+00:00

Comment: Accepted at ROMCIR 2026, the 6th Workshop on Reducing Online Misinformation through Credible Information Retrieval, held in conjunction with ECIR 2026

AI Summary

This paper investigates the effectiveness of crowdsourcing for audiovisual deepfake detection by evaluating how consistently crowd workers can identify authentic versus manipulated videos, pinpoint manipulation types, and localize them temporally. The study conducts two crowdsourcing experiments on Prolific, analyzing worker judgments across two distinct deepfake datasets.

Abstract

Deepfakes are increasingly realistic and easy to produce, raising concerns about the reliability of human judgments in misinformation settings. We study audiovisual deepfake detection by measuring how consistently crowd workers distinguish authentic from manipulated videos and, when they flag a video as manipulated, how accurately they identify the manipulation type (audio-only, video-only, or audio-video) and how consistently they report manipulation timestamps. We run two matched crowdsourcing studies on Prolific using AV-Deepfake1M and the Trusted Media Challenge (TMC) dataset. We sample 48 videos per dataset (96 total) and collect 960 judgments (10 per video). Results show that crowd workers rarely misclassify authentic videos as manipulated, but they miss many manipulations, and agreement remains limited across videos. Aggregating multiple judgments per video stabilizes the authenticity signal, but it cannot recover manipulations that most workers consistently miss. Manipulation type identification is substantially noisier than authenticity detection even when workers detect a manipulation, with joint audio-video cases being particularly hard to recognize. Overall, these findings suggest that crowdsourcing can provide a scalable screening signal for audiovisual authenticity, while reliable modality attribution remains an open challenge.

Key findings

Crowd workers exhibit low false positive rates for authentic videos but frequently miss manipulations, with detection performance and agreement varying significantly across datasets. While aggregating judgments improves authenticity detection, accurately identifying the specific manipulation type (audio-only, video-only, or audio-video) remains a significant challenge, particularly for joint audio-video deepfakes. However, when a manipulation is flagged, timestamp reports can still converge on plausible segments.

Approach

The researchers conduct two crowdsourcing studies on Prolific, where workers evaluate audiovisual deepfakes by judging authenticity, identifying the manipulation type (audio-only, video-only, or audio-video), and marking manipulation timestamps. They analyze accuracy, inter-worker agreement, and error profiles, comparing majority vote and Dempster-Shafer aggregation methods.

Datasets

AV-Deepfake1M, Trusted Media Challenge (TMC) dataset

Model(s)

UNKNOWN

Author countries

Italy

← Previous