Large language models (LLMs) are artificial intelligence systems like ChatGPT, trained to predict the next word in a text. Scientists have learned to extract from their internal activity information about where a city is located, when a historical figure lived, or what emotion a character is experiencing. Many took this as proof that the model builds an internal picture of the world.
A new work by Elan Barenholtz of Florida Atlantic University (USA) calls such conclusions into question. The preprint is published on arXiv - it is a version of the article before peer review.
Barenholtz tested whether the same information could be obtained from simpler mathematical structures - word vectors. A word vector is a numerical representation of a word based on which words it frequently co-occurs with. Unlike LLMs, such vectors do not take sentence context into account.
The researcher repeated the analysis of four previously published works in which data on geography, years of death of historical figures, pain and emotions were extracted from language models. He applied similar mathematical methods to static word vectors. A simple method predicted coordinates and years of death, distinguished sentences about pain from similar ones without pain, and identified 12 emotions in stories where they are not named directly. In tests on geography and history, word vectors explained a significant portion of the data variance, and in tests on pain and emotions, accuracy approached the performance of LLMs.
The advantage of language models mainly showed up where it was necessary to determine a country for a location or a century for a historical figure. Such broad categories are easily captured by word associations. When it was necessary to distinguish places within one country or people within one century, all methods had difficulty.
According to the author, word associations carry a lot of information about places and people. Texts about different regions may contain characteristic spellings, mentions of local institutions or universities. A system can use these linguistic patterns to infer geography without having anything resembling a map. Back in 1957, linguist J. R. Firth noted: "A word is known by its company".
Barenholtz also studied word vectors associated with specific Wikipedia articles. Such representations take into account the linguistic patterns associated with a person or place, not just the words in its name. This made it possible to almost completely close the remaining gap with LLMs.
It is important to emphasize: the study does not prove that LLMs lack internal representations of the world. It does not resolve questions about AI consciousness or the capacity to suffer. Barenholtz's argument is more modest: the very possibility of extracting information from a model's internal activity does not make it possible to establish whether the model stores a representation of a property or whether this information is explained by language patterns. "I am not claiming that these models definitely do not represent the world," the scientist says. "I am saying that such a test cannot tell us that. Therefore future research should compare results with a simple baseline before drawing big conclusions".
The work offers a practical step for future AI research: before claiming that a model knows or represents something, it is worth checking whether a simple word-based method does not yield the same. At the same time, the author notes that his comparison does not address causal or behavioral evidence that other ways of studying how AI works may provide.