Semantics-Oriented Multitask Learning for DeepFake Detection: A Joint Embedding Approach

Authors: Mian Zou, Baosheng Yu, Yibing Zhan, Siwei Lyu, Kede Ma

Published: 2024-08-29 07:11:50+00:00

AI Summary

This paper introduces Semantics-Oriented Joint Embedding DeepFake Detector (SJEDD), a novel approach for DeepFake detection that utilizes semantics-oriented multitask learning. It proposes an automated dataset expansion technique to enrich existing face forgery datasets with semantic hierarchical labels and employs joint embedding of face images and text descriptions for prediction. The method also incorporates bi-level optimization to dynamically balance task loss weightings, leading to improved generalizability and interpretability.

Abstract

In recent years, the multimedia forensics and security community has seen remarkable progress in multitask learning for DeepFake (i.e., face forgery) detection. The prevailing approach has been to frame DeepFake detection as a binary classification problem augmented by manipulation-oriented auxiliary tasks. This scheme focuses on learning features specific to face manipulations with limited generalizability. In this paper, we delve deeper into semantics-oriented multitask learning for DeepFake detection, capturing the relationships among face semantics via joint embedding. We first propose an automated dataset expansion technique that broadens current face forgery datasets to support semantics-oriented DeepFake detection tasks at both the global face attribute and local face region levels. Furthermore, we resort to the joint embedding of face images and labels (depicted by text descriptions) for prediction. This approach eliminates the need for manually setting task-agnostic and task-specific parameters, which is typically required when predicting multiple labels directly from images. In addition, we employ bi-level optimization to dynamically balance the fidelity loss weightings of various tasks, making the training process fully automated. Extensive experiments on six DeepFake datasets show that our method improves the generalizability of DeepFake detection and renders some degree of model interpretation by providing human-understandable explanations.

Key findings

SJEDD consistently improves cross-dataset, cross-manipulation, and cross-attribute DeepFake detection performance, outperforming state-of-the-art detectors, including previous semantics-oriented and CLIP-based methods. The joint embedding and fidelity loss, coupled with bi-level optimization, significantly contribute to better generalizability and stable training. The approach also enhances model interpretability by providing human-understandable explanations for detections and accurately localizing manipulated face regions.

Approach

The authors propose SJEDD, which addresses DeepFake detection through semantics-oriented multitask learning. It first expands datasets with a semantic hierarchical graph through automated data augmentation and relabeling. Then, it uses a joint embedding approach with vision-language correspondence to encode face images and text descriptions of labels into a common feature space, enabling prediction and automated model capacity allocation via bi-level optimization with a fidelity loss.

Datasets

FF++, FFSC, CDF, FaceShifter (FSh), DeeperForensics-1.0 (DF-1.0), DeepFake Detection Challenge (DFDC), DiffusionFace, DiFF

Model(s)

CLIP (ViT-B/32 as image encoder, GPT-2 as text encoder)

Author countries

Hong Kong, Singapore, China, USA

← Previous