Audio Deepfake Detection Using Temporal Coherence Analysis

Authors: Justin D. Norman, Sarah Barrington

Published: 2026-09-08 22:08:30+00:00

AI Summary

This paper introduces a temporal coherence analysis framework for detecting audio deepfakes across speech and music domains, utilizing statistical features derived from Contrastive Language-Audio Pretraining (CLAP) embeddings. The approach employs lightweight ensemble classifiers and uncovers two significant findings: a feature-label inversion phenomenon between training and in-the-wild deployment, and a speech-music direction reversal in entropy's discriminative power.

Abstract

The proliferation of AI-generated audio (so-called deepfake audio) poses significant threats to information integrity, from voice cloning fraud to synthetic music copyright disputes. We present a temporal coherence analysis framework built upon Contrastive Language-Audio Pretraining (CLAP) embeddings that spans speech, instrumental music, and music with vocals. By computing pairwise cosine similarities between audio segment embeddings and extracting statistical features from the resulting distributions, we train lightweight ensemble classifiers that reliably distinguish authentic from synthetic audio. Our work provides an interpretable, computationally efficient alternative to common deep learning methods while still achieving competitive performance across speech and music domains. Further, we reveal two notable empirical findings about audio deepfakes: (1) a feature-label inversion phenomenon in which 21 of 29 statistical features reverse their discriminative direction between training and in-the-wild deployment, and (2) a speech--music direction reversal in which entropy discriminates in opposite directions for speech and music deepfakes.

Key findings

The study reveals a feature-label inversion, where 21 of 29 statistical features reverse their discriminative direction between controlled training data and in-the-wild evaluation, explaining the failure of standard domain adaptation methods. Additionally, it identifies a speech-music direction reversal, where entropy discriminates in opposite directions for speech and music deepfakes, indicating different generation artifacts across domains.

Approach

The method segments audio into overlapping windows, extracts CLAP embeddings for each segment, and computes pairwise cosine similarities to create a distribution of similarities. From these distributions, 29 statistical features are extracted, which then feed into XGBoost ensemble classifiers for binary deepfake detection. For music, a domain-adaptive percentile-based classifier is used to handle domain shifts.

Datasets

LibriSpeech, ASVspoof 2019 (LA subset), DeepSpeak v2, M-AILABS (via MLAAD), AUDETER, MUSDB18, FakeMusicCaps, In-the-Wild, FakeAVCeleb, ASVspoof5, FMA, SONICS

Model(s)

CLAP embeddings, XGBoost

Author countries

United States

← Previous