Synthetic Trust Attacks: Modeling How Generative AI Manipulates Human Decisions in Social Engineering Fraud

Authors: Muhammad Tahir Ashraf

Published: 2026-04-02 23:09:35+00:00

Comment: 15 pages, 3 figures, 2 tables

AI Summary

This paper proposes Synthetic Trust Attacks (STAs) as a formal threat category and introduces STAM, an eight-stage operational framework for AI-driven social engineering fraud. It argues that current defenses focusing on synthetic media detection are inadequate, advocating for a shift to decision-layer defenses that counter manufactured credibility rather than media authenticity. The paper presents a Trust-Cue Taxonomy, an Incident Coding Schema, and operationalizes the Calm, Check, Confirm protocol as a research-grade defense.

Abstract

Imagine receiving a video call from your CFO, surrounded by colleagues, asking you to urgently authorise a confidential transfer. You comply. Every person on that call was fake, and you just lost $25 million. This is not a hypothetical. It happened in Hong Kong in January 2024, and it is becoming the template for a new generation of fraud. AI has not invented a new crime. It has industrialised an ancient one: the manufacture of trust. This paper proposes Synthetic Trust Attacks (STAs) as a formal threat category and introduces STAM, the Synthetic Trust Attack Model, an eight-stage operational framework covering the full attack chain from adversary reconnaissance through post-compliance leverage. The core argument is this: existing defenses target synthetic media detection, but the real attack surface is the victim's decision. When human deepfake detection accuracy sits at approximately 55.5%, barely above chance, and LLM scam agents achieve 46% compliance versus 18% for human operators while evading safety filters entirely, the perception layer has already failed. Defense must move to the decision layer. We present a five-category Trust-Cue Taxonomy, a reproducible 17-field Incident Coding Schema with a pilot-coded example, and four falsifiable hypotheses linking attack structure to compliance outcomes. The paper further operationalizes the author's practitioner-developed Calm, Check, Confirm protocol as a research-grade decision-layer defense. Synthetic credibility, not synthetic media, is the true attack surface of the AI fraud era.

Key findings

Generative AI industrializes social engineering by manufacturing synthetic credibility at scale, making traditional synthetic media detection insufficient as human accuracy is near chance levels. The true attack surface is the victim's decision layer, not the synthetic media itself, necessitating architectural defenses against 'decision compression.' The Calm, Check, Confirm protocol offers a structured, stress-resilient defense by mandating cognitive pauses and independent verification.

Approach

The authors propose STAM, an eight-stage operational framework detailing the full attack chain of AI-enabled social engineering. They introduce a five-category Trust-Cue Taxonomy and a 17-field Incident Coding Schema to analyze these attacks. For defense, they operationalize the Calm, Check, Confirm protocol, designed to introduce cognitive pauses and mandatory out-of-band verification to combat decision compression.

Datasets

Hong Kong CFO deepfake case (January 2024) (pilot-coded example)

Model(s)

UNKNOWN

Author countries

Pakistan

← Previous