When, Where, and Why: Hierarchical Causal Explainability for Video Face Understanding with MLLMs

Authors: Anubhav Kataria, Shreya Ghosh, Jingwen Ye, Abhinav Dhall

Published: 2026-10-13 06:20:08+00:00

AI Summary

This paper introduces VideoFace-X-LLM, a framework for explainable facial video understanding. It comprises ExplainFace, a weakly supervised video-text corpus for facial behavior, and H-TCC (Hierarchical Temporal Spatial Causal Concepts), which identifies 'when', 'where', and 'why' specific facial expressions occur through targeted occlusion. Fine-tuning on ExplainFace significantly improves performance across various facial analysis tasks, including deepfake detection, while providing faithful and measurable explanations.

Abstract

Understanding facial behaviour in video requires answering three fundamental questions: when a key facial expression occurs, where the model attends on the face at that moment, and why that evidence is necessary rather than merely correlated. Today's multimodal large language models (MLLMs) can label and describe video, but they rarely provide faithful explanations, especially for small facial movements that change over time. We present VideoFace-X-LLM, a framework for explainable facial video understanding comprising two key components. First, we introduce ExplainFace, a weakly supervised video-text corpus for facial behaviour understanding, automatically constructed from CelebV-HQ videos and validated through human evaluation and a two-model audit (at most 2.3% contradictions), demonstrating consistent gains across two backbone architectures. Second, H-TCC (Hierarchical Temporal Spatial Causal Concepts), which finds the key temporal moments (When), the facial regions (Where), and verifies its causal necessity (Why) through targeted occlusion. Fine-tuning on ExplainFace raises appearance Macro F1 from 19.48% to 32.87% (+13.39), with gains of +4.38 on actions, +6.77 on temporal emotion, and +11.84 F1 on deepfake detection. We also measure the explanations pointing lift, face/edge lift, stability, an Adebayo sanity check, and a necessity test rather than only showing examples. Our models, code, and corpus are available on request for research use.

Key findings

Fine-tuning on ExplainFace significantly improved Macro F1 scores across multiple facial understanding tasks, including appearance (+13.39), actions (+4.38), temporal emotion (+6.77), and deepfake detection (+11.84). The H-TCC framework provides faithful explanations, with high spatial pointing lift (1.40-1.42x) indicating accurate facial region identification, and causal scores (temporal: 1.28-1.76, spatial: 1.44-1.93) confirming the necessity of identified evidence for predictions.

Approach

The approach involves two main components: first, the creation of ExplainFace, a weakly supervised video-text corpus using Gemini to generate time-aligned descriptions of facial attributes, expressions, and audio-visual context from CelebV-HQ videos. Second, H-TCC (Hierarchical Temporal Spatial Causal Concepts) is introduced to provide explanations by identifying key temporal moments (When), facial regions (Where) using hierarchical temporal saliency and spatial attention analysis, and verifying their causal necessity (Why) through targeted occlusion of the identified evidence.

Datasets

CelebV-HQ, FaceForensics++, DISFA

Model(s)

InternVideo2.5, InternLM2.5-7B, Qwen3-VL, Qwen2.5-VL, LLaVA-NeXT, Phi-4, MiniCPM-V, Gemini (for corpus generation)

Author countries

Australia, India

← Previous