The use of AI systems can appear to provide clear resolutions to many subjects. AI can summarize scientific papers, explain political conflicts or even build coherent arguments from scattered information across dozens of sources. AI can be said to function like a master math-machine that can observe and imitate patterns within language, leading to seemingly smooth “reasoning” with explanations that sound complete. However, is this just a result of complex calculations and matrices, or is there still something deeper to be revealed by our synthetic friend?
In his recent seminar on Sept. 10, Idan Blank set forth to test this assumption by “Understanding ‘Understanding’ in Language Models.” As an assistant professor at UCLA, Blank’s research suggests that language models may process some parts of language in surprisingly human-like ways while organizing other basic forms of meaning quite differently. At times, they may even reach a sensible answer through an alternative method that may have little to do with the reasoning we attribute to them.
Many question whether AI understands language at all. While some point to its fluent answers, others suggest a language model merely predicts the next word. Blank offered a different question: what parts of understanding emerge from next-word prediction, and how are they represented inside the model?
“All models are wrong... But language models, and specifically transformers, can master language better than any other model we've had from linguistics ... because they give us a computational representation of virtually any sentence, any linguistic input in natural language, and that is enough reason to at least take them seriously as candidate models, even if they turn out to not be great, and they allow us to test our theories with unprecedented resolution because we can feed them anything,” Blank explained to the audience.
Rather than judging language models only through prompts, his lab inspects transformers (the foundational neural network architecture that powers modern generative AI) and compares their behavior with patterns documented in human language processing.
“Our approach is to break meaning or comprehension into much, much smaller chunks that we can handle that are both theoretically motivated by psycholinguistic theory and relatively well characterized in humans... In my lab, we study how language models understand, so we open them and look under the hood.”
To start, Blank first asked whether meaning could influence the parts of a language model that process grammar. Consider the sentence, “The garden in the cat chased a butterfly.” Its grammatical pattern is recognizable, but its meaning is strange because gardens do not chase butterflies. Transformers contain components called attention heads, which move information between words. Some heads appear to specialize in connecting a verb with its grammatical subject. If they cared only about grammar, the verb “chased” would treat “garden” as its subject regardless of how little sense that relationship made.
Instead, the language model gave less attention to the relationship when it was implausible; for example, it sometimes shifted its subject toward “cat.” Humans have been noted to do something comparable: we do not sequentially process grammar and then meaning but rather meaning and context shape our interpretation as we read. This does not prove that a transformer understands language the same way, but it shows that a system trained through next-word prediction can develop a human-like interaction between grammar and meaning.
Thus, his second experiment probed deeper, looking at event roles. How do transformers represent one of the most basic elements of meaning (“who did what to whom”)? Compare “A doctor pulled a prisoner” with “A prisoner was pulled by a doctor.” Their grammatical forms differ, but both describe the same event. Humans are known to judge both as more similar to each other than to the sentence “A prisoner pulled a doctor,” which preserves the first sentence's structure but reverses its meaning.
In Blank’s findings, the models did the opposite. Across several transformers, their overall internal representations grouped sentences more strongly by grammatical structure than by the event described. Some attention heads could still identify the person acting and the person receiving the action, which indicated the relevant information existed somewhere in the network. However, it still did not become a prominent part of how the model represented the complete sentence. This showed that a system can contain the pieces of an idea without assembling them as humans do.
“For some sentences, you can recover the full syntactic tree of the sentence with reasonable accuracy, and that suggests that hidden representation of structure does not guarantee hidden representation of meaning,” Blank highlighted.
What happens when our context grows: looking at not just language, but also visual cognition? Blank explained his current project investigates whether a model combining images and language could infer why a person chose particular words.
Consider an experiment with images of three objects: a big hammer, a small hammer and a big cork. When humans are asked, for example, to choose the “big hammer,” he reveals humans understand the speaker’s intentions in distinguishing both the size and the type of object. When testing language models, they appeared to make this same pragmatic inference. For instance, they would expect the word “big” when the image contained a useful size contrast between two objects.
Despite this, there may still be a simpler explanation for the obtained results: images with more hammers also contained more hammer-shaped pixels. The model could appear sensitive to communicative intent while responding mainly to what occupied more space.
“At this point, I think we have like ten different experiments to try to figure out what's going on. I think tentatively... object salience really overshadows other things, but there is some small pragmatic [factor] underneath,” Blank explained.
These projects resist a simple verdict. The first project found a human-like interaction between meaning and grammar in language models, the second found that these models could gain information about an event without necessarily incorporating it into its final understanding and the third project showed how it could use a visual shortcut to imitate an inference about human intention. AI understanding is not a single capacity that we can mark as present or absent; it appears unevenly across mechanisms and conditions.
“Even if they succeed, we don't know why they succeed, right? It's very useful to see that they succeed at something, but we want to know how. So that's why we look under the hood,” Blank suggested.
A classroom model may explain a concept correctly, or it may merely recognize familiar wording. A research assistant may integrate evidence, or follow a superficial pattern associated with past discoveries. In a clinical setting, a recommendation that works for typical cases may fail when a patient no longer matches the usual statistical cues. Blank’s research shows that the output alone cannot define which process produced it.
While AI systems may appear to provide many resolutions, Blank gives a reason for caution through his seminar. Human-like capacities can emerge from calculations and matrices without arriving at human-like understanding. Blank’s conclusions highlight deeper questions regarding the features of our synthetic friend.
“They learn what event rules are because we find them in attention heads, but perhaps they don't represent them in a human-like way,“ Blank notes.
