ShortCheck: Checkworthiness Detection of Multilingual Short-Form Videos

View on arXiv ← Back to list

Authors: Henrik Vatndal, Vinay Setty

Published: 2025-09-24 18:37:45+00:00

AI Summary

ShortCheck is a modular, inference-only pipeline designed to automatically identify checkworthy short-form videos (like those on TikTok) in a multilingual setting to assist human fact-checkers. The system handles multimodal content by integrating components for speech transcription, OCR, visual analysis (including deepfake detection), and video-to-text summarization. Evaluated on two manually annotated TikTok datasets, the pipeline achieved robust performance with an F1-weighted score exceeding 70%.

Abstract

Short-form video platforms like TikTok present unique challenges for misinformation detection due to their multimodal, dynamic, and noisy content. We present ShortCheck, a modular, inference-only pipeline with a user-friendly interface that automatically identifies checkworthy short-form videos to help human fact-checkers. The system integrates speech transcription, OCR, object and deepfake detection, video-to-text summarization, and claim verification. ShortCheck is validated by evaluating it on two manually annotated datasets with TikTok videos in a multilingual setting. The pipeline achieves promising results with F1-weighted score over 70\\%.

Key findings

The ShortCheck pipeline achieved strong cross-lingual generalizability, with macro-averaged F1-weighted scores reaching 0.72 (Norwegian) and 0.74 (English). Ablation studies revealed that textual signals derived from audio transcripts and ideological buzzword detection contributed most significantly to reliable classification. Deepfake detection and object recognition modules provided limited standalone utility in the final aggregated system.

Approach

The system uses a modular, inference-only pipeline integrating feature extraction from various modalities, including speech transcription (Whisper), OCR (EasyOCR), and video-to-text summarization (LLaVA and LLaMA 3). Deepfake detection models are also incorporated as features. A rule-based logic engine aggregates the outputs and scores from these modules to classify the video as Checkworthy or Not Checkworthy.

Datasets

Norwegian influencer data (TikTok videos), TikTok Videos from Fact-Checking Websites (curated by Bu et al., 2024).

Model(s)

OpenAI Whisper, EasyOCR, LLaVA, LLaMA 3, MesoNet, EfficientNet, Wvolf/ViT.

Author countries

Norway

← Previous