EMO-BOOST: Emotion-Augmented Audio-Visual Features for Improved Generalization in Deepfake Detection

Authors: Aritra Marik, Marcel Klemt, Anna Rohrbach

Published: 2026-05-19 10:11:10+00:00

Comment: Accepted at SAFE@CVPRW 2026

AI Summary

This paper introduces Emo-Boost, a multimodal deepfake detection framework that enhances generalization to unseen manipulations by fusing low-level deepfake detection with high-level emotion-based semantic cues. Emo-Boost integrates an off-the-shelf deepfake detector with EmoForensics, an emotion-based module that models intra- and inter-modal temporal consistency in audio-visual emotion representations. The framework significantly improves cross-manipulation generalization performance, particularly on the FakeAVCeleb dataset.

Abstract

With every advancement in generative AI models, forensics is under increasing pressure. The constant emergence of new generation techniques makes it impossible to collect data for each manipulation to train a deepfake detection model. Thus, generalizing to deepfakes unseen during training is one of the major challenges in current deepfake detection research. To tackle this challenge, we employ high-level semantic cues and argue that these cues can support low-level focused approaches in generalizing to unseen types of manipulations. In this work, we study emotions as a high-level semantic cue. We propose Emo-Boost, a multimodal deepfake detection framework that fuses an off-the-shelf RGB- and acoustic-focused deepfake detector with our emotion-based deepfake detector EmoForensics. EmoForensics utilises vision and audio emotion recognition modules and models intra- and inter-modal temporal consistency in emotion representations from an audio-visual stream. We found that EmoForensics and the low-level focused method capture complementary signals. Consequently, combining both signals in EmoBoost enhances the average cross-manipulation generalization AUC by 2.1% on FakeAVCeleb.

Key findings

Emo-Boost enhances the average cross-manipulation generalization AUC by 2.1% on FakeAVCeleb compared to the standalone low-level detector (SIMBA). While remaining competitive in in-domain evaluation, the framework shows significant improvements on specific unseen manipulation splits. EmoForensics captures complementary signals and exhibits greater stability in cross-manipulation scenarios compared to low-level detectors.

Approach

The proposed Emo-Boost framework fuses an off-the-shelf RGB- and acoustic-based deepfake detector with a novel emotion-based deepfake detector called EmoForensics. EmoForensics utilizes frozen visual and audio emotion recognition modules to extract frame-level emotion representations, then models intra-modal temporal consistency using transformer encoders and inter-modal consistency via a contrastive loss. The emotion-based features from EmoForensics are combined with the low-level features from the baseline detector using element-wise multiplication to improve generalization.

Datasets

FakeAVCeleb, DeepSpeak v2

Model(s)

SIMBA (as the off-the-shelf deepfake detector), POSTER (visual emotion encoder), emotion2vec (audio emotion encoder), Temporal Transformer (for intra-modal modeling)

Author countries

Germany

← Previous