Deepfake News Detection: A Multimodal Framework Integrating LipNet, DeepSpeech and ResNET for Enhanced Audio-Visual Analysis

Authors: Ameena Khan, Muhammad Ahsan Aziz, Muhammad Junaid Asif, Naeem Akhter, Rana Fayyaz Ahmad

Published: 2026-07-22 11:39:53+00:00

AI Summary

This paper introduces a multimodal framework for deepfake news detection that integrates audio and visual cues. It leverages features extracted from lip movements, audio content, and video frames using specialized pre-trained models. The combined features are then classified by an ensemble of machine learning and deep learning models to discern the authenticity of video content.

Abstract

Deepfake news refers to AI-generated (or AI ma-nipulated) multimedia content intentionally generated to deceive audiences by manipulating the facial expressions, or speech while maintaining the realistic appearance. The rapid progress of generative AI has made the synthesis of highly realistic fake videos and cloned voices widely accessible, posing a serious threat to the authenticity of digital news media. This paper presents a multi-modal framework that discerns the authenticity of video content by jointly exploiting audio and visual cues, thereby addressing the challenge of detecting the deepfake videos. We proposed a framework that involves features extraction from lip movements, audio content and video frames. Lip movements and speech content are encoded using the LipNet and DeepSpeech2 models, while facial features are extracted by leveraging the use of BlazeFace and represented with ResNet18. The extracted feature vectors are concatenated into a holistic video representation and classified with an ensemble of machine learning and deep learning models, including Random Forest (RF), Multi-layer Perceptron (MLP) and Long Short-Term Memory (LSTM) networks. Exten-sive experiments performed on the FakeAVCeleb dataset shows that the proposed approach attains an accuracy of 94% using augmented audio features, outperforming a state-of-the-art multi-modal ensemble baseline. The results confirm the robustness and practical potential of the proposed framework for deepfake news detection.

Key findings

The proposed approach achieved an accuracy of 94% on the FakeAVCeleb dataset using augmented audio features, outperforming a state-of-the-art multimodal ensemble baseline. The study revealed that for current high-quality deepfakes, audio and lip-sync coherence provide more reliable discriminative cues than visual facial features, as synthetic faces are becoming visually indistinguishable from real ones. Data augmentation significantly improved detection performance.

Approach

The framework extracts features from lip movements (LipNet), audio content (DeepSpeech2), and facial features from video frames (BlazeFace and ResNet18). These extracted feature vectors are concatenated to form a holistic video representation. This representation is then fed into an ensemble of classifiers, including Random Forest, MLP, and LSTM, for final deepfake detection. Data augmentation with a sliding window technique is also applied to improve performance on imbalanced datasets.

Datasets

FakeAVCeleb, Deepfake Detection Challenge (DFDC)

Model(s)

LipNet, DeepSpeech2, BlazeFace, ResNet18, Random Forest, Multi-layer Perceptron (MLP), Long Short-Term Memory (LSTM), AdaBoost, Decision Trees, SVM, K-Nearest Neighbor, Naive Bayes

Author countries

Pakistan

← Previous