Visual Realism Assessment for Face-swap Videos

Authors: Xianyun Sun, Beibei Dong, Caiyong Wang, Bo Peng, Jing Dong

Published: 2023-02-02 07:34:27+00:00

Comment: Accepted by ICIG 2023

AI Summary

This paper addresses the underexplored area of visual realism assessment (VRA) for deepfake face-swap videos, which is crucial for evaluating their potential impact and comparing generation methods. The authors establish a benchmark using the DFGC-2022 dataset, annotated with Mean Opinion Scores (MOS), to evaluate automatic VRA models. They demonstrate the feasibility of effective VRA models by comparing traditional hand-crafted features with various deep-learning features, highlighting the particular usefulness of existing deepfake detection features for VRA.

Abstract

Deep-learning based face-swap videos, also known as deep fakes, are becoming more and more realistic and deceiving. The malicious usage of these face-swap videos has caused wide concerns. The research community has been focusing on the automatic detection of these fake videos, but the assessment of their visual realism, as perceived by human eyes, is still an unexplored dimension. Visual realism assessment, or VRA, is essential for assessing the potential impact that may be brought by a specific face-swap video, and it is also important as a quality assessment metric to compare different face-swap methods. In this paper, we make a small step towards this new VRA direction by building a benchmark for evaluating the effectiveness of different automatic VRA models, which range from using traditional hand-crafted features to different kinds of deep-learning features. The evaluations are based on a recent competition dataset named DFGC 2022, which contains 1400 diverse face-swap videos that are annotated with Mean Opinion Scores (MOS) on visual realism. Comprehensive experiment results using 11 models and 3 protocols are shown and discussed. We demonstrate the feasibility of devising effective VRA models for assessing face-swap videos and methods. The particular usefulness of existing deepfake detection features for VRA is also noted. The code can be found at https://github.com/XianyunSun/VRA.git.

Key findings

The study demonstrates the feasibility of devising effective VRA models for assessing face-swap videos and methods. Deep features generally outperform handcrafted features, with the DFGC-1st deepfake detection model achieving the best performance in realism assessment, suggesting a close relationship between deepfake detection and realism assessment. However, improving VRA models' generalization ability to new datasets remains an open challenge.

Approach

The proposed VRA method involves three main steps: data pre-processing to crop face areas, feature extraction using various existing models (hand-crafted, image/video quality assessment, image/face recognition, deepfake detection), and finally, feature fusion and selection before regressing the video realism score using a Support Vector Regressor (SVR). This approach builds a benchmark by evaluating 11 different models under various protocols.

Datasets

DFGC-2022

Model(s)

BRISQUE, GM-LOG, FRIQUEE, TLVQM, V-BLIINDS, VIDEVAL, Ensemble (custom VQA), ResNet50, VGG-Face, DFDC-ispl, DFGC-1st (deepfake detection models). These models are used as feature extractors, with Support Vector Regression (SVR) as the final regressor.

Author countries

P.R. China

← Previous