Morphology-optimized Multi-Scale Fusion: Combining Local Artifacts and Mesoscopic Semantics for Deepfake Detection and Localization

View on arXiv ← Back to list

Authors: Chao Shuai, Gaojian Wang, Kun Pan, Tong Wu, Fanli Jin, Haohan Tan, Mengxiang Li, Zhenguang Liu, Feng Lin, Kui Ren

Published: 2025-09-17 07:46:07+00:00

Comment: The 3rd Place, IJCAI 2025 Workshop on Deepfake Detection, Localization, and Interpretability

AI Summary

This paper proposes a novel morphology-optimized multi-scale fusion framework for deepfake detection and localization. It independently predicts manipulated regions using both local and mesoscopic perspectives, then employs morphological operations to fuse these outputs. This approach effectively combines local forgery artifacts with mesoscopic semantic information to achieve precise and robust localization, addressing limitations of existing methods that neglect complementary contexts or use naive fusion strategies.

Abstract

While the pursuit of higher accuracy in deepfake detection remains a central goal, there is an increasing demand for precise localization of manipulated regions. Despite the remarkable progress made in classification-based detection, accurately localizing forged areas remains a significant challenge. A common strategy is to incorporate forged region annotations during model training alongside manipulated images. However, such approaches often neglect the complementary nature of local detail and global semantic context, resulting in suboptimal localization performance. Moreover, an often-overlooked aspect is the fusion strategy between local and global predictions. Naively combining the outputs from both branches can amplify noise and errors, thereby undermining the effectiveness of the localization. To address these issues, we propose a novel approach that independently predicts manipulated regions using both local and global perspectives. We employ morphological operations to fuse the outputs, effectively suppressing noise while enhancing spatial coherence. Extensive experiments reveal the effectiveness of each module in improving the accuracy and robustness of forgery localization.

Key findings

The proposed framework achieved superior performance, with a Final Score of 0.8150, F1-score of 0.7759, and IoU of 0.6902, demonstrating significant improvements in both detection and localization accuracy. The ablation study confirmed that the Morphology-Driven Mask Fusion (MDMF) strategy, which refines masks through dilation and erosion, is crucial for enhancing spatial coherence and suppressing noise, effectively combining the strengths of local and global forgery cues.

Approach

The approach employs a dual-branch framework comprising a Local Facial Forgery Detection and Location (LFDL) network and a Mesoscopic Image Tampering Localization (MITL) network. LFDL focuses on fine-grained facial artifacts using a two-stream RGB and SRM architecture, while MITL captures broader semantic context and microscopic details using a Mesorch backbone with frequency-enhanced representations. A Morphology-Driven Mask Fusion (MDMF) strategy then intelligently merges the dilated LFDL masks and eroded MITL masks via a union operation, ensuring comprehensive and spatially coherent forgery localization.

Datasets

Deepfake Detection and Localization Image (DDL-I) dataset, and a custom-generated supplementary dataset created using various forgery techniques (FaceSwap, Simswap, MaskFaceGAN, Facedancer, BELM, SD-inpainting).

Model(s)

Xception backbone for the LFDL model, and a Segformer-B3 and ConvNeXt-Tiny hybrid backbone for the MITL model.

Author countries

China

← Previous