Audio-Visual Person-of-Interest DeepFake Detection

Authors: Davide Cozzolino, Alessandro Pianese, Matthias Nießner, Luisa Verdoliva

Published: 2022-04-06 20:51:40+00:00

AI Summary

This work proposes POI-Forensics, a deepfake detector that leverages audio-visual features to characterize a person's identity, making it robust against various manipulation methods and real-world scenarios. By training exclusively on real talking-face videos using a contrastive learning paradigm, the detector learns discriminative embeddings for each identity, detecting manipulations when the audio-visual representation becomes inconsistent with the real identity. The method achieves high generalization ability and can detect single and multi-modality attacks, even in low-quality or corrupted videos.

Abstract

Face manipulation technology is advancing very rapidly, and new methods are being proposed day by day. The aim of this work is to propose a deepfake detector that can cope with the wide variety of manipulation methods and scenarios encountered in the real world. Our key insight is that each person has specific characteristics that a synthetic generator likely cannot reproduce. Accordingly, we extract audio-visual features which characterize the identity of a person, and use them to create a person-of-interest (POI) deepfake detector. We leverage a contrastive learning paradigm to learn the moving-face and audio segment embeddings that are most discriminative for each identity. As a result, when the video and/or audio of a person is manipulated, its representation in the embedding space becomes inconsistent with the real identity, allowing reliable detection. Training is carried out exclusively on real talking-face video; thus, the detector does not depend on any specific manipulation method and yields the highest generalization ability. In addition, our method can detect both single-modality (audio-only, video-only) and multi-modality (audio-video) attacks, and is robust to low-quality or corrupted videos. Experiments on a wide variety of datasets confirm that our method ensures a SOTA performance, especially on low quality videos. Code is publicly available on-line at https://github.com/grip-unina/poi-forensics.

Key findings

The POI-Forensics method achieves state-of-the-art performance, outperforming existing approaches by a significant margin, particularly in challenging scenarios involving low-quality, compressed, or adversarially attacked videos. The inclusion of audio similarity greatly enhances detection performance, and the fusion of audio and video information provides superior discrimination ability and robustness against various deepfake types.

Approach

The method, POI-Forensics, employs a contrastive learning paradigm to learn identity-discriminative moving-face and audio segment embeddings exclusively from real talking-face videos. Deepfakes are detected by comparing these extracted audio-visual features from a test video against a reference set of the person-of-interest (POI) using audio-only, video-only, and joint audio-video similarity indices. Manipulations are flagged when the representation in the embedding space is inconsistent with the real identity.

Datasets

VoxCeleb2 (for training), pDFDC, DF-TIMIT, FakeAVCelebV2, KoDF (for evaluation)

Model(s)

ResNet-50 with Group-Normalization (for both audio and video networks)

Author countries

Italy, Germany

← Previous