A Noise and Edge extraction-based dual-branch method for Shallowfake and Deepfake Localization

Authors: Deepak Dagar, Dinesh Kumar Vishwakarma

Published: 2024-09-02 02:18:34+00:00

AI Summary

This paper introduces a dual-branch model for Image Manipulation Localization (IML) designed to detect and localize both shallowfakes and deepfakes. The model integrates manually designed noise features with conventional RGB features using a hierarchical ConvNext Module and employs edge supervision loss for accurate boundary localization. It achieves an impressive 99% AUC score on various shallowfake and deepfake datasets, outperforming existing state-of-the-art models.

Abstract

The trustworthiness of multimedia is being increasingly evaluated by advanced Image Manipulation Localization (IML) techniques, resulting in the emergence of the IML field. An effective manipulation model necessitates the extraction of non-semantic differential features between manipulated and legitimate sections to utilize artifacts. This requires direct comparisons between the two regions.. Current models employ either feature approaches based on handcrafted features, convolutional neural networks (CNNs), or a hybrid approach that combines both. Handcrafted feature approaches presuppose tampering in advance, hence restricting their effectiveness in handling various tampering procedures, but CNNs capture semantic information, which is insufficient for addressing manipulation artifacts. In order to address these constraints, we have developed a dual-branch model that integrates manually designed feature noise with conventional CNN features. This model employs a dual-branch strategy, where one branch integrates noise characteristics and the other branch integrates RGB features using the hierarchical ConvNext Module. In addition, the model utilizes edge supervision loss to acquire boundary manipulation information, resulting in accurate localization at the edges. Furthermore, this architecture utilizes a feature augmentation module to optimize and refine the presentation of attributes. The shallowfakes dataset (CASIA, COVERAGE, COLUMBIA, NIST16) and deepfake dataset Faceforensics++ (FF++) underwent thorough testing to demonstrate their outstanding ability to extract features and their superior performance compared to other baseline models. The AUC score achieved an astounding 99%. The model is superior in comparison and easily outperforms the existing state-of-the-art (SoTA) models.

Key findings

The model achieved an astounding 99% AUC score on Faceforensics++ for deepfake localization, and high AUCs on shallowfake datasets (e.g., 99.79% on NIST16, 97.25% on CASIA). It consistently outperforms several state-of-the-art baselines. Ablation studies confirmed that edge supervision, the feature enhancement module, and both the noise and RGB branches are crucial for the model's superior performance and accurate manipulation localization, particularly at image boundaries.

Approach

The proposed approach utilizes a dual-branch model where one branch extracts noise characteristics using Bayar convolution and SRM filters, and the other processes RGB features. Both branches employ hierarchical ConvNext Modules and a Feature Enhancement Module for optimized attribute presentation. Additionally, the model incorporates edge supervision loss to specifically acquire and improve boundary manipulation information.

Datasets

CASIA (v1.0, v2.0), NIST16, COLUMBIA, COVERAGE, Faceforensics++ (FF++)

Model(s)

Dual-branch ConvNext-based architecture with Feature Enhancement Module and Edge Extraction Block; uses Bayar convolution, SRM filters, and Sobel operator for feature extraction; trained with a combined loss function of Binary Cross Entropy (BCE) Loss, Focal Loss (FL), and Edge Loss (EL).

Author countries

India

← Previous