Tex-ViT: A Generalizable, Robust, Texture-based dual-branch cross-attention deepfake detector

Authors: Deepak Dagar, Dinesh Kumar Vishwakarma

Published: 2024-08-29 20:26:27+00:00

AI Summary

This paper introduces Tex-ViT, a deepfake detection model that addresses generalization and robustness limitations of traditional methods by combining ResNet features with a texture module and a dual-branch cross-attention Vision Transformer. The model leverages empirical observations that fake images exhibit smoother textures and inconsistent long-range texture correlations. Tex-ViT significantly outperforms state-of-the-art models in cross-domain scenarios, achieving 98% accuracy, and demonstrates strong resilience against various post-processing procedures like blurring, compression, and noise.

Abstract

Deepfakes, which employ GAN to produce highly realistic facial modification, are widely regarded as the prevailing method. Traditional CNN have been able to identify bogus media, but they struggle to perform well on different datasets and are vulnerable to adversarial attacks due to their lack of robustness. Vision transformers have demonstrated potential in the realm of image classification problems, but they require enough training data. Motivated by these limitations, this publication introduces Tex-ViT (Texture-Vision Transformer), which enhances CNN features by combining ResNet with a vision transformer. The model combines traditional ResNet features with a texture module that operates in parallel on sections of ResNet before each down-sampling operation. The texture module then serves as an input to the dual branch of the cross-attention vision transformer. It specifically focuses on improving the global texture module, which extracts feature map correlation. Empirical analysis reveals that fake images exhibit smooth textures that do not remain consistent over long distances in manipulations. Experiments were performed on different categories of FF++, such as DF, f2f, FS, and NT, together with other types of GAN datasets in cross-domain scenarios. Furthermore, experiments also conducted on FF++, DFDCPreview, and Celeb-DF dataset underwent several post-processing situations, such as blurring, compression, and noise. The model surpassed the most advanced models in terms of generalization, achieving a 98% accuracy in cross-domain scenarios. This demonstrates its ability to learn the shared distinguishing textural characteristics in the manipulated samples. These experiments provide evidence that the proposed model is capable of being applied to various situations and is resistant to many post-processing procedures.

Key findings

Tex-ViT achieved superior generalization with 98% accuracy in cross-domain scenarios, effectively identifying shared textural characteristics in manipulated samples. The model demonstrated significant robustness against post-processing operations such as blurring, compression, and noise, outperforming existing state-of-the-art detectors. Empirical analysis confirmed that fake images consistently exhibit smoother surfaces and a lack of long-range texture correlation, which Tex-ViT successfully exploits for detection.

Approach

Tex-ViT integrates a ResNet-18 backbone with a parallel texture module, which calculates Gram matrices to extract multi-scale texture correlations from input images. These conventional CNN features and texture-specific features are then fed into a dual-branch cross-attention Vision Transformer. This transformer leverages self-attention for long-range global relationships and cross-attention between branches to fuse local and global information, enhancing scale invariance and generalization.

Datasets

FaceForensics++ (DF, f2f, FS, NT categories), DFDCPreview, Celeb-DF, ProGAN generated images, StyleGAN generated images, StarGAN generated images, STGAN generated images, CelebA-HQ, CelebA, FFHQ.

Model(s)

Tex-ViT (proposed), ResNet-18 (backbone), Vision Transformer (ViT) with dual-branch cross-attention, Gram matrices (for texture feature extraction).

Author countries

India

← Previous