Which LLMs are Difficult to Detect? A Detailed Analysis of Potential Factors Contributing to Difficulties in LLM Text Detection

Authors: Shantanu Thorat, Tianbao Yang

Published: 2024-10-18 21:42:37+00:00

Comment: Accepted at NeurIPS 2024 - Safe Generative AI Workshop; Camera-ready version

AI Summary

This paper analyzes the factors contributing to the difficulty of detecting LLM-generated texts across various LLMs and writing domains. By training AI-generated (AIG) text classifiers with the LibAUC library on two datasets, the authors evaluate detection performance. Key findings indicate that scientific writing is challenging to detect, and OpenAI family LLMs are particularly difficult to distinguish from human texts, attributed to their higher text entropy and Out-Of-Vocabulary ratios.

Abstract

As LLMs increase in accessibility, LLM-generated texts have proliferated across several fields, such as scientific, academic, and creative writing. However, LLMs are not created equally; they may have different architectures and training datasets. Thus, some LLMs may be more challenging to detect than others. Using two datasets spanning four total writing domains, we train AI-generated (AIG) text classifiers using the LibAUC library - a deep learning library for training classifiers with imbalanced datasets. Our results in the Deepfake Text dataset show that AIG-text detection varies across domains, with scientific writing being relatively challenging. In the Rewritten Ivy Panda (RIP) dataset focusing on student essays, we find that the OpenAI family of LLMs was substantially difficult for our classifiers to distinguish from human texts. Additionally, we explore possible factors that could explain the difficulties in detecting OpenAI-generated texts.