Language models such as ChatGPT and Gemini display fairly stable "character traits" when describing themselves. That is the conclusion of a group of psychologists from the Institute of Psychology at Johannes Gutenberg University Mainz (JGU). The results of the study were published in the journal Scientific Reports.
The experiment involved seven chatbots - ChatGPT-4o, ChatGPT-3.5, Llama-3, Gemma 2, Gemini 2.5, Phi-3 and Orca 2 - and 70 living people. Both were asked, in German, to fill out psychological questionnaires and answer open-ended questions about themselves: for example, to rate self-esteem and the so-called "Big Five" - five basic personality traits, including openness, agreeableness and extraversion. Participants were then asked about their interests, hopes for the future and typical behavior in stressful situations. Each model was surveyed ten times, yielding 140 texts for analysis.
It turned out that on repeated surveys the language models attribute a fairly stable personality profile to themselves. Compared with people, they presented themselves as more conscientious, more agreeable and more emotionally stable.
In the second stage, 10 people and six chatbots rated these self-descriptions - they had to determine the expression of traits such as openness, conscientiousness and self-esteem. Both people and machines generally handled the task, but with biases: the chatbots overrated some qualities in human authors, especially openness and self-esteem. People, by contrast, gave higher scores to texts written by neural networks - above all for emotional stability.
The main difference emerged in another task - distinguishing human text from generated text. People achieved 83% accuracy, while the chatbots managed only 67%. The best models, Gemma and Gemini, performed at roughly the level of an average human. Both groups more often mistook AI text for human text than the reverse - possibly to avoid accusing a person of using a neural network.
People mainly relied on the length of AI texts and their deliberately technical or analytical language. Computational linguistic analysis also showed differences: neural network texts contained more temporal expressions and function words - articles and prepositions - while people more often wrote about leisure, feelings and social ties.
The authors note that language models could be used for initial personality assessment, for example in psychological diagnostics or HR work. However, because the models are trained on human texts, they also absorb biases - distortions of judgment. Therefore the question of whether algorithms should be trusted to assess personality in sensitive areas requires separate study, including from an ethical standpoint.