Large language models, like humans, exhibit unique writing styles, according to a recent study. These AI models frequently enough favor specific words, phrases, and sentence structures, resulting in distinctive patterns.

Researchers at Carnegie Mellon University found they could accurately identify which LLM generated a text sample based on these stylistic quirks. The team was able to distinguish between different models with 97% accuracy.

“It was quite surprising that we achieved this level of accuracy,” says Mingjie Sun, a PhD student in the computer science department.

Sun and colleagues initially struggled to differentiate between two LLMs,achieving only 60% to 70% accuracy. However, their specialized classifier program excelled at the more complex task of distinguishing between five LLMs: ChatGPT, Claude, Gemini, Grok, and Deepseek.

The analysis revealed that each LLM has a distinct profile. For example, ChatGPT tended to produce detailed, explanatory texts, while Claude preferred concise answers.

These stylistic differences are deeply ingrained in each model and persist even when texts are scrambled, rephrased, translated, or summarized.

Implications for AI Training

The researchers suggest caution when using synthetic data (text generated by LLMs) to train new models. This practice could transfer the stylistic idiosyncrasies of the source model to the next generation of LLMs, potentially affecting their behavior.

Zico Kolter, professor and director of CMU’s machine learning department, notes that while synthetic data was once widely used for training, its popularity has declined.

“This work is much more about understanding the distinctive characteristics, the natures of different LLMs, the same way we think about different styles of writing by people,”

The research team did not focus on differentiating between AI-generated and human-generated text, as other groups have already explored this topic extensively. Instead, Sun, kolter, and their collaborators aimed to gain a deeper understanding of LLMs.

“Given how much content is being produced by LLMs on the internet these days, it is valuable to understand the distinguishing characteristics of these various models,” Kolter says.

The pre-print paper is yet to be peer-reviewed, and its findings are preliminary.