ExpSpeech-Net: Multimodal Fusion of Expression and Speech for Deepfake Detection

Authors: Ruchika Sharma, Rudresh Dwivedi

Published: 2026-06-04 06:41:36+00:00

AI Summary

This study introduces ExpSpeech-Net (SqN-R-DFD), a lightweight and efficient deepfake detection model that fuses facial expressions and speech patterns. It employs advanced feature extraction methods like ISLBT for images and MPNCC for speech signals, along with a smart feature-selection strategy using SASMA. The framework combines SqueezeNet and RNN as its backbone to effectively capture subtle inconsistencies.

Abstract

Deepfake videos are increasingly challenging the credibility of online content. Many existing detection methodology relies on complex, resource-intensive models, which limit their practical use. The study introduces the ExpSpeech-Net deepfake detection (SqN-R-DFD) model, which utilizes SqueezeNet and RNN (Recurrent Neural Network) as its backbone, providing a lightweight and efficient deepfake detection framework that simultaneously analyzes facial expressions and speech patterns. The approach incorporates advanced feature extraction, such as ISLBT-based features for image and MPNCC for signals, along with a smart feature-selection strategy using SASMA (Sandpiper-Assisted Slime Mould Algorithm), ensuring optimal and balanced input to the detection models. By combining SqueezeNet and an RNN, subtle inconsistencies in deepfake videos are captured effectively. The framework achieves 94.5% accuracy, precision of 99.3%, and F-measure of 96.8%, outperforming conventional methods. This demonstrates that integrating multiple modalities with intelligent preprocessing and feature selection enables practical, real-time deepfake detection suitable for everyday applications.

Key findings

The ExpSpeech-Net (SqN-R-DFD) framework achieved high performance, with 94.5% accuracy, 99.3% precision, and 96.8% F-measure on the WLDR dataset, consistently outperforming conventional methods across various deepfake deformation cases. The intelligent preprocessing, feature extraction (ISLBT and MPNCC), and SASMA-based hierarchical feature selection significantly contribute to its superior detection capability and generalization across different datasets and conditions.

Approach

The proposed ExpSpeech-Net preprocesses face images using a face cascade method and speech signals with a Double Sigmoid Normalization (DSN) approach. It then extracts ISLBT-based, DSBME, and deep features (from VGG16, ResNet50) from images, and MPNCC, MFCC, and chroma features from speech. A Hierarchical Feature Selection (HFS) technique, guided by the SASMA algorithm, optimally selects and balances these multimodal features. Finally, SqueezeNet and a Recurrent Neural Network (RNN) are used in a hybrid detection phase to classify videos as real or fake.

Datasets

World Leader Dataset (WLDR), DeepfakeTIMIT dataset

Model(s)

SqueezeNet, Recurrent Neural Network (RNN), VGG16 (for deep features), ResNet50 (for deep features)

Author countries

India

← Previous