A Multimodal Framework for Deepfake Detection

Authors: Kashish Gandhi, Prutha Kulkarni, Taran Shah, Piyush Chaudhari, Meera Narvekar, Kranti Ghag

Published: 2024-10-04 14:59:10+00:00

Comment: 22 pages, 14 figures, Accepted in Journal of Electrical Systems

AI Summary

This research proposes a multimodal framework for deepfake detection, integrating both visual and auditory analyses to combat the growing threat of synthetic media. It employs advanced feature extraction for video (facial characteristics) and mel-spectrograms for audio, classifying overall content as deepfake if either component is detected as such. The comprehensive framework achieves an overall accuracy of 94%.

Abstract

The rapid advancement of deepfake technology poses a significant threat to digital media integrity. Deepfakes, synthetic media created using AI, can convincingly alter videos and audio to misrepresent reality. This creates risks of misinformation, fraud, and severe implications for personal privacy and security. Our research addresses the critical issue of deepfakes through an innovative multimodal approach, targeting both visual and auditory elements. This comprehensive strategy recognizes that human perception integrates multiple sensory inputs, particularly visual and auditory information, to form a complete understanding of media content. For visual analysis, a model that employs advanced feature extraction techniques was developed, extracting nine distinct facial characteristics and then applying various machine learning and deep learning models. For auditory analysis, our model leverages mel-spectrogram analysis for feature extraction and then applies various machine learning and deep learningmodels. To achieve a combined analysis, real and deepfake audio in the original dataset were swapped for testing purposes and ensured balanced samples. Using our proposed models for video and audio classification i.e. Artificial Neural Network and VGG19, the overall sample is classified as deepfake if either component is identified as such. Our multimodal framework combines visual and auditory analyses, yielding an accuracy of 94%.

Key findings

The multimodal framework achieved an overall detection accuracy of 94%. Individually, the Artificial Neural Network (ANN) model for video deepfake detection reached an accuracy of 93%, while the VGG19 model for audio deepfake detection demonstrated superior performance with 98% accuracy. This combined approach proved robust and more effective than various traditional machine learning algorithms and unimodal deep learning methods.

Approach

The approach utilizes separate pipelines for video and audio analysis. For video, it extracts nine distinct facial characteristics (e.g., nose/lip size, blinks, headpose) using Haar Cascade and FaceMesh, which are then classified by an Artificial Neural Network. For audio, mel-spectrograms are generated for feature extraction and classified using a VGG19 model. The final classification deems a sample a deepfake if either the video or audio component is identified as synthetic.

Datasets

DFDC Dataset, Fake-or-Real (FoR) dataset

Model(s)

Random Forest, XGBoost, CNN, VGG19

Author countries

India

← Previous