Zero-Shot Temporal Localisation of Audio Deepfakes in Multi-Speaker Conversations

Authors: Soumyadeep Roy

Published: 2026-09-09 11:27:26+00:00

AI Summary

This paper introduces Temporal Deepfake Localisation in Multi-Speaker Conversations (TDLMC) and proposes a training-free, five-stage pipeline to identify synthetic speech segments within genuine conversations. By wrapping a frozen binary deepfake detector with a hysteresis finite-state-machine decoder, the system achieves segment-level output without retraining. The method demonstrates high temporal Intersection-over-Union (t-IoU) and low false-alarm rates on constructed multi-speaker conversations and genuine dialogue.

Abstract

Voice-cloning fraud increasingly relies on surgical injection: a genuine conversation in which only one or two sentences are replaced by synthetic speech. Utterance-level deepfake detectors emit a single real/fake label per clip and cannot report where the synthetic speech lies. We formalise this as Temporal Deepfake Localisation in Multi-Speaker Conversations (TDLMC), show that equal error rate and min-DCF are ill-posed once a file contains both classes, and propose temporal metrics for this regime. Our contribution is a training-free five-stage pipeline that wraps a frozen binary detector and adds segment-level output with no retraining, using a two-threshold hysteresis finitestate-machine decoder to turn noisy window scores into coherent intervals. On 180 constructed multi-speaker conversations from ASVspoof 5, the system attains temporal intersection-over-union 0.90, temporal detection rate 0.95, and MS-DCF 0.26 with a strong backbone, and its false-alarm rate on genuine speech is below 6%, falling under 2% on genuine real multi-speaker dialogue (AMI). Under an identical pipeline, a trained localiser improves temporal IoU by only about 0.04, bounding the cost of forgoing supervision. Evaluated across three frozen detectors under one decoder whose constants are selected on a held-out calibration split, and with a controlled analysis attributing the residual false-alarm rate to a backbone domain gap rather than to the decoder, this provides the first zero-shot baseline and a reusable benchmark for TDLMC.

Key findings

The zero-shot pipeline achieved a temporal Intersection-over-Union (t-IoU) of 0.90, temporal detection rate (TDR) of 0.95, and MS-DCF of 0.26 on constructed multi-speaker conversations using a strong backbone. The false-alarm rate on genuine speech was below 6% and dropped to under 2% on genuine real multi-speaker dialogue (AMI corpus). The training-free localiser performed comparably to a trained localiser, with only about a 0.04 t-IoU difference, bounding the cost of forgoing supervision.

Approach

The approach uses a training-free, five-stage pipeline that processes audio waveforms. It involves segmenting the audio into overlapping windows, scoring each window with a frozen binary deepfake detector, mapping these scores onto a uniform timeline, and then applying a median filter and Gaussian smoothing. Finally, a two-threshold hysteresis finite-state-machine (FSM) decoder converts these smoothed scores into coherent 'fake' intervals, suppressing boundary flicker.

Datasets

ASVspoof 5 (for constructing multi-speaker conversations), AMI Meeting Corpus (for validation on real conversational data)

Model(s)

DF Arena 1B (WavLM-Large + Conformer), wav2vec2-XLSR, AASIST

Author countries

India

← Previous