Beyond Ambiguous Visual Cues: Studying Physiological Disruptions and Cross-Modal Inconsistencies in Deepfake Videos

View on arXiv ← Back to list

Authors: Chenxi Yang, Yassine Ouzar, Larbi Boubchir

Published: 2026-09-11 10:12:48+00:00

AI Summary

This paper investigates how deepfake manipulations disrupt natural physiological signals and facial behavior by constructing high-fidelity deepfake videos on established real rPPG datasets. They propose a bidirectional co-attention fusion detector that jointly models rPPG and facial behavior tokens to capture cross-level dependencies. The approach demonstrates superior performance over single-modality and shallow fusion methods in detecting deepfakes.

Abstract

Recent deepfake detection studies increasingly suggest remote photoplethysmography (rPPG) signals as an authenticity cue. However, existing benchmarks lack physiological ground truth, and current detectors underexplore the cross-level relationship between facial features and physiological dynamics, often relying on late fusion or rPPG features alone. In this paper, we construct high-fidelity deepfake manipulations on established real rPPG datasets (COHFACE and UBFC-rPPG) to investigate how forgeries disrupt natural physiological signals and facial behavior at the same time. Building on this analysis, we propose a bidirectional co-attention fusion detector that jointly models rPPG and facial behavior tokens. This mechanism explicitly captures the cross-level dependencies between pulse dynamics and facial motion to learn a robust, joint authenticity representation. Extensive experiments using a subject-disjoint 5-fold evaluation demonstrate the superiority of our approach. Achieving a 92.80\\% AUC on constructed datasets using face swapping and 96.78\\% AUC on motion transfer, our model outperforms both the rPPG-only single modality baseline and the best feature-level fusion methods. Furthermore, transfer-learning result of the fusion detector on Celeb-DF-v2 while keeping both feature extractors fixed achieves 91.20\\% accuracy and 86.08\\% AUC, which suggests applicability under target-domain adaptation.

Key findings
The proposed bidirectional co-attention fusion detector achieved 92.80% AUC on constructed face-swapping datasets and 96.78% AUC on motion transfer, outperforming rPPG-only and feature-level fusion methods. Transfer learning to Celeb-DF-v2 achieved 91.20% accuracy and 86.08% AUC, suggesting applicability beyond physiology-oriented datasets. Deepfake generation, especially motion transfer, significantly degrades recoverable physiological signals, and this degradation can be used for detection.
Approach
The authors construct deepfake manipulations (face swapping and motion transfer) on rPPG datasets with physiological ground truth. They then propose a bidirectional co-attention fusion detector that jointly models rPPG (physiology) tokens extracted by a PhysFormer model and facial behavior tokens (action units, pose, gaze) from OpenFace. This co-attention mechanism explicitly captures cross-level dependencies between pulse dynamics and facial motion for deepfake detection.
Datasets
COHFACE, UBFC-rPPG, Celeb-DF-v2
Model(s)
PhysFormer (for rPPG token extraction), OpenFace (for facial behavior token extraction), Bidirectional co-attention fusion detector (custom architecture)
Author countries
China, France
← Previous
Next →