The eyes know it: FakeET -- An Eye-tracking Database to Understand Deepfake Perception

View on arXiv ← Back to list

Authors: Parul Gupta, Komal Chugh, Abhinav Dhall, Ramanathan Subramanian

Published: 2020-06-12 06:14:49+00:00

Comment: 8 pages

AI Summary

This paper introduces FakeET, a novel eye-tracking and EEG database designed to understand human visual perception of deepfake videos and evaluate the ease of detecting synthetic artifacts. The study, involving 40 users viewing 811 videos from the Google Deepfake dataset, confirms distinct eye movement characteristics and EEG responses for real versus fake videos. These human behavioral cues demonstrate utility for spatial forgery localization and can aid automated deepfake detection.

Abstract

We present \\textbf{FakeET}-- an eye-tracking database to understand human visual perception of \\emph{deepfake} videos. Given that the principal purpose of deepfakes is to deceive human observers, FakeET is designed to understand and evaluate the ease with which viewers can detect synthetic video artifacts. FakeET contains viewing patterns compiled from 40 users via the \\emph{Tobii} desktop eye-tracker for 811 videos from the \\textit{Google Deepfake} dataset, with a minimum of two viewings per video. Additionally, EEG responses acquired via the \\emph{Emotiv} sensor are also available. The compiled data confirms (a) distinct eye movement characteristics for \\emph{real} vs \\emph{fake} videos; (b) utility of the eye-track saliency maps for spatial forgery localization and detection, and (c) Error Related Negativity (ERN) triggers in the EEG responses, and the ability of the \\emph{raw} EEG signal to distinguish between \\emph{real} and \\emph{fake} videos.

Key findings
The study revealed distinct eye movement characteristics (exploratory for real, focused for fake) and neural responses (Error Related Negativity in EEG) that differentiate real from fake videos. Augmenting video frames with user gaze maps improved CNN-based deepfake detection performance by approximately 5%. Even raw EEG signals achieved better-than-chance deepfake detection performance using various classifiers.
Approach
The authors created FakeET, a database containing eye-tracking and EEG data from 40 users observing deepfake videos. They analyzed gaze patterns and neural responses to identify differences between real and fake video perception. Furthermore, they demonstrated that augmenting CNN inputs with eye-track gaze maps improves automated deepfake detection performance.
Datasets
FakeET, Google Deepfake Detection dataset
Model(s)
3D ResNet, Time-series CNN, Naive Bayes, Logistic Regression, k-Nearest Neighbors, Decision Tree, Linear SVM
Author countries
India, Australia
← Previous
Next →