IMFD: End-to-end Multi-Face Forgery Detection through Instruction-based Large Vision-Language Models

View on arXiv ← Back to list

Authors: Dasom Choi, Sangjun Moon, Hyeongchan Im, Jaeeon Park, Jingun Kwon, Hidetaka Kamigaito, Taro Watanabe, Manabu Okumura

Published: 2026-09-17 04:40:20+00:00

Comment: 8 pages, 5 figures, 5 tables. Accepted to Findings of AACL-IJCNLP 2026

AI Summary

The paper introduces IMFD (Instruction-based Multi-face Forgery Detector), an end-to-end single-stage approach that leverages instruction-based Large Vision-Language Models (LVLMs) to jointly localize faces and predict per-face forgery labels. IMFD integrates predicted face bounding boxes as visual cues into textual instructions to enhance forgery detection. By converting existing multi-face forgery datasets into an instruction-based format, IMFD demonstrates improved performance over traditional methods, especially in scenarios with multiple faces.

Abstract

The rapid increase of deepfakes has raised significant concerns due to their spread on social media. Traditional multi-face forgery detectors crop and verify each face independently, ignoring background context and inter-face relationships, which often yields suboptimal performance. To overcome these limitations, we leverage instruction-based Large Vision-Language Models (LVLMs), which can interpret entire images and follow complex textual instructions. We propose a simple yet effective single-stage multi-face forgery detector, called IMFD (Instruction-based Multi-face Forgery Detector), which is trained end-to-end to jointly localize faces and predict per-face forgery labels. Rather than treating face box prediction only as a joint objective, IMFD explicitly integrates predicted face bounding boxes into the instruction as visual cues that enhance instruction grounding and forgery detection. To support the training and evaluation of IMFD, we convert existing multi-face forgery datasets into an instruction-based format. Experimental results and analyses show that IMFD improves multi-face forgery detection by integrating face bounding boxes into the instruction, and consistently outperforms various state-of-the-art methods.

Key findings

IMFD consistently outperforms state-of-the-art baselines in both two-stage and single-stage settings, achieving significantly lower maximum latency in the two-stage configuration. Integrating auxiliary details like the number of people, image resolution, and face coordinates into the instructions proves highly effective in improving detection performance. The quality of face localization significantly impacts the overall performance of the single-stage IMFD.

Approach

The proposed IMFD system utilizes instruction-based Large Vision-Language Models (LVLMs) for multi-face forgery detection. It operates in a single stage, jointly performing face localization and forgery classification. Crucially, predicted face bounding boxes are explicitly incorporated into the textual instructions as visual cues to improve instruction grounding and detection accuracy, addressing limitations of independent face cropping.

Datasets

OpenForensics

Model(s)

InternVL2-2B (backbone), LLaVA-1.5, LLaVA-1.6, InternVL2-4B, InternVL2-8B, MiniCPM-Llama3-V-v2.5, MiniCPM-V-v2.6, CLIP ViT-B/32

Author countries

South Korea, Japan

← Previous