A research team at Epoch AI has found that popular AI text detectors struggle to identify AI-generated content when language models mimic an author's writing style. The study, which tested three widely used detectors—Pangram, GPTZero, and Originality.ai—revealed that these tools have significantly lower detection rates when AI-generated texts are crafted to imitate human authors. The findings highlight a critical limitation in current AI detection systems, particularly in identifying content that closely resembles human writing.
When AI models are prompted to imitate a specific author's style using text samples, the detectors' performance drops substantially. The research team reported that an average of 13% of the generated passages slipped through undetected. Scientific writing proved to be the most challenging category, with detectors failing to flag between 24 and 29% of style-mimicking AI-generated content. This raises concerns about the reliability of these tools in academic and educational settings where scientific writing is common.
The study involved a corpus of 495 human passages from 99 authors, evenly split across blogging, fiction, and scientific writing. All texts were written before ChatGPT's release in November 2022, ensuring no contamination from language models. When dealing with plain AI-generated text, all three detectors performed almost flawlessly, with a false-negative rate of up to 0.7%. However, the results changed significantly when language models were asked to imitate an author's style, with detection rates dropping to about 13%.
Source: thedecoder
