Authors: David Fischinger, Marcel Hasenbalg, Silvia Poletti, Davide Antonutti, Mina Schütz, Lam Pham, Justin Ilyes, Péter Rjabcsenkó, Simon Freitter, Martin Boyer
Published: 2026-10-20 10:18:49+00:00
Comment: ECCV 2026, Workshop on AI for Multimedia Forensics & Disinformation Detection (AI4MFDD2026)
AI Summary
This paper introduces the Media Intelligence Platform (MIP), a unified web-based framework for multimodal media forensics. MIP integrates specialized modules for detecting various forms of manipulated content across images, videos, audio, and text, addressing the limitations of single-modality detection approaches. Its modular architecture allows for flexible integration of heterogeneous models and provides a user-friendly interface for forensic investigations.
Abstract
The rapid progress of generative artificial intelligence has led to a significant increase in manipulated multimedia content, including synthetic faces, forged videos, AI-generated images, fake audio, and locally edited images. Existing detection approaches are typically designed for a single manipulation type, limiting their applicability in real-world scenarios involving heterogeneous media sources. In this work, we present the Media Intelligence Platform (MIP), a unified web-based framework for multimodal media forensics integrating specialized modules for fake video detection, face morphing detection, AI-generated image detection, audio deepfake detection, localized image manipulation detection, and text analysis. Its modular architecture enables flexible integration of heterogeneous models and supports future extensions for emerging manipulation techniques. Experimental results on representative benchmark datasets demonstrate reliable detection performance across multiple media modalities while providing a practical and user-friendly interface for forensic investigations. The platform highlights the potential of unified multimodal systems for comprehensive deepfake detection in real-world environments, and is accompanied by a public demonstrator released with this workshop submission to support reproducibility and community evaluation.
Key findings
The integrated modules demonstrate reliable detection performance across multiple media modalities. The platform highlights the potential of unified multimodal systems for comprehensive deepfake detection in real-world environments. Strong cross-dataset generalization was observed in audio deepfake detection, and for video deepfake detection, the best-performing methods consistently relied on large pretrained vision models with lightweight fine-tuning strategies.
Approach
MIP is a modular, container-based framework that exposes forensic and AI-based analysis components through a unified REST interface. It uses a microservice architecture where each analysis component (Module) is independently developed and deployed as a Docker container, encapsulating underlying neural network models. The platform supports multimodal input, analysis tool selection, and job history tracking.
Datasets
For AI-generated image detection: MS-COCO, AIText2Image, ImgN100, VOC12, OImg-v7, FLUX.1, Firefly, MJ 5&6, DALL-E 3, GLIDE. For Image Forgery Detection and Localization: MS COCO, DF2023, MIML, CASIA, Columbia, DSO, NIST. For Face Morphing Detection: Diverse morphing dataset curated from multiple passport-like face datasets. For Face Swap Detection in Videos: Curated benchmark of 1000 videos from 10 datasets. For Audio Deepfake Detection: ASVspoof 2019 (LA), ASVspoof 2021 (LA & DF), ASVspoof5, In-The-Wild, FakeAVCeleb.
Model(s)
For AI-generated image detection: ResNet-50, ViT-B-16. For Image Forgery Detection and Localization: hybrid U-Net-based convolutional network with a ConvNeXtV2-Base encoder. For Face Morphing Detection: ArcFace with a ResNet-100 backbone. For Face Swap Detection in Videos: 19 state-of-the-art deepfake detection methods including EFFORT (ViT, CLIP), selimsef FbF (CNN, Ensemble), GenConvViT (CNN, ViT), CC ViT (CNN, ViT), Efficient ViT (CNN, ViT), UCF (CNN, Disentanglement, Autoencoder), Xception (CNN), ID-unaware (CNN, Artifact Detection), CORE (CNN, Representation Learning), RECCE (CNN, Autoencoder, Graph), F3Net (CNN, Frequency), FFD (CNN, Attention), SPSL (CNN, Frequency), SRM (CNN, Frequency, Attention), CapsuleNet (CNN, Capsule Network), EfficientNet-B4 (CNN), ResNet34 (CNN), MesoNet4Inception (CNN, Inception), LRNet (RNN, Landmark, Temporal). For Audio Deepfake Detection: WavLM backbone. For Text Analysis: XLM-RoBERTa (XLM-R).
Author countries
Austria