Max Planck Institute for Psycholinguistics - Max Planck Neuroscience

Deep learning models to study sentence comprehension in the human brain

MPFI — Wed, 28 Jun 2023 14:18:28 +0000

Recent artificial neural networks that process natural language achieve unprecedented performance in tasks requiring sentence-level understanding. As such, they could be interesting models of the integration of linguistic information in the human brain. We review works that compare these artificial language models with human brain activity and we assess the extent to which this approach has improved our understanding of the neural processes involved in natural language comprehension. Two main results emerge. First, the neural representation of word meaning aligns with the context-dependent, dense word vectors used by the artificial neural networks. Second, the processing hierarchy that emerges within artificial neural networks broadly matches the brain, but is surprisingly inconsistent across studies. We discuss current challenges in establishing artificial neural networks as process models of natural language comprehension. We suggest exploiting the highly structured representational geometry of artificial neural networks when mapping representations to brain data.

Sophie Arana, Jacques Pesnot Lerousseau & Peter Hagoort (2023): Deep learning models to study sentence comprehension in the human brain, Language, Cognition and Neuroscience. Link

The post Deep learning models to study sentence comprehension in the human brain first appeared on Max Planck Neuroscience.

Commonalities and Asymmetries in the Neurobiological Infrastructure for Language Production and Comprehension

MPFI — Thu, 05 Jan 2023 14:48:01 +0000

The neurobiology of sentence production has been largely understudied compared to the neurobiology of sentence comprehension, due to difficulties with experimental control and motion-related artifacts in neuroimaging. We studied the neural response to constituents of increasing size and specifically focused on the similarities and differences in the production and comprehension of the same stimuli. Participants had to either produce or listen to stimuli in a gradient of constituent size based on a visual prompt. Larger constituent sizes engaged the left inferior frontal gyrus (LIFG) and middle temporal gyrus (LMTG) extending to inferior parietal areas in both production and comprehension, confirming that the neural resources for syntactic encoding and decoding are largely overlapping. An ROI analysis in LIFG and LMTG also showed that production elicited larger responses to constituent size than comprehension and that the LMTG was more engaged in comprehension than production, while the LIFG was more engaged in production than comprehension. Finally, increasing constituent size was characterized by later BOLD peaks in comprehension but earlier peaks in production. These results show that syntactic encoding and parsing engage overlapping areas, but there are asymmetries in the engagement of the language network due to the specific requirements of production and comprehension.

Giglio, L., Ostarek, M., Weber, K., & Hagoort, P. (2022). Commonalities and asymmetries in the neurobiological infrastructure for language production and comprehension. Cerebral Cortex, 32(7), 1405-1418. Link

The post Commonalities and Asymmetries in the Neurobiological Infrastructure for Language Production and Comprehension first appeared on Max Planck Neuroscience.

A hierarchy of linguistic predictions during natural language comprehension

MPFI — Thu, 05 Jan 2023 14:37:21 +0000

Understanding spoken language requires transforming ambiguous acoustic streams into a hierarchy of representations, from phonemes to meaning. It has been suggested that the brain uses prediction to guide the interpretation of incoming input. However, the role of prediction in language processing remains disputed, with disagreement about both the ubiquity and representational nature of predictions. Here, we address both issues by analyzing brain recordings of participants listening to audiobooks, and using a deep neural network (GPT-2) to precisely quantify contextual predictions. First, we establish that brain responses to words are modulated by ubiquitous predictions. Next, we disentangle model-based predictions into distinct dimensions, revealing dissociable neural signatures of predictions about syntactic category (parts of speech), phonemes, and semantics. Finally, we show that high-level (word) predictions inform low-level (phoneme) predictions, supporting hierarchical predictive processing. Together, these results underscore the ubiquity of prediction in language processing, showing that the brain spontaneously predicts upcoming language at multiple levels of abstraction.

Heilbron, M., Armeni, K., Schoffelen, J.-M., Hagoort, P., & De Lange, F. P. (2022). A hierarchy of linguistic predictions during natural language comprehension. Proceedings of the National Academy of Sciences of the United States of America, 119(32): e2201968119. Link.

The post A hierarchy of linguistic predictions during natural language comprehension first appeared on Max Planck Neuroscience.

Our Brain is a Prediction Machine that is Always Active

MPFI — Thu, 06 Oct 2022 19:41:23 +0000

Our brain works a bit like the autocomplete function on your phone – it is constantly trying to guess the next word when we are listening to a book, reading or conducting a conversation. Contrary to speech recognition computers, our brains are constantly making predictions at different levels, from meaning and grammar to specific speech sounds. This is what researchers at the Max Planck Institute for Psycholinguistics and Radboud University’s Donders Institute discovered in a new study. Their findings are published in PNAS.

This is in line with a recent theory on how our brain works: it is a prediction machine, which continuously compares sensory information that we pick up (such as images, sounds and language) with internal predictions. “This theoretical idea is extremely popular in neuroscience, but the existing evidence for it is often indirect and restricted to artificial situations,” says lead author Micha Heilbron. “I would really like to understand precisely how this works and test it in different situations.”

Brain research into this phenomenon is usually done in an artificial setting, Heilbron reveals. To evoke predictions, participants are asked to stare at a single pattern of moving dots for half an hour, or listen to simple patterns in sounds like ‘beep beep boop, beep beep boop, …. “Studies of this kind do in fact reveal that our brain can make predictions, but not that this always happens in the complexity of everyday life as well. We are trying to take it out of the lab setting. We are studying the same type of phenomenon, how the brain deals with unexpected information, but then in natural situations that are much less predictable.”

Hemingway and Holmes

The researchers analyzed the brain activity of people listening to stories by Hemingway or about Sherlock Holmes. At the same time, they analyzed the texts of the books using computer models, so-called deep neural networks. This way, they were able to calculate for each word how unpredictable it was.

For each word or sound, the brain makes detailed statistical expectations and turns out to be extremely sensitive to the degree of unpredictability: the brain response is stronger whenever a word is unexpected in the context. “By itself, this is not very surprising: after all, everyone knows that you can sometimes predict upcoming language. For example, your brain sometimes automatically ‘fills in the blank’ and mentally finishes someone else’s sentences, for instance if they start to speak very slowly, stutter or are unable to think of a word. But what we have shown here is that this happens continuously. Our brain is constantly guessing at words; the predictive machinery is always turned on.”

More than software

“In fact, our brain does something comparable to speech recognition software. Speech recognizers using artificial intelligence are also constantly making predictions and are allowing themselves to be guided by their expectations, just like the autocomplete function on your phone. Nevertheless, we observed a big difference: brains predict not only words, but make predictions on many different levels, from abstract meaning and grammar to specific sounds.”

There is good reason for the ongoing interest from tech companies who would like to use new insights of this kind to build better language and image recognition software, for example. But these sorts of applications are not the main aim for Heilbron. “I would really like to understand how our predictive machinery works at a fundamental level. I’m now working with the same research setup, but for visual and auditive perceptions, like music.”

Heilbron, M., Armeni, K., Schoffelen, J.-M., Hagoort, P., & de Lange, F. P. (2022). A hierarchy of linguistic predictions during natural language comprehension. Proceedings of the National Academy of Sciences, 119(32), e2201968119. Link

The post Our Brain is a Prediction Machine that is Always Active first appeared on Max Planck Neuroscience.

Language as a Marker of the Mind

MPFI — Thu, 06 Oct 2022 19:26:41 +0000

In a discussion article published in Cognition, Peter Hagoort, professor of Cognitive Neuroscience at Radboud University and director of the Max Planck Institute for Psycholinguistics, argues for the ‘language marker hypothesis’. According to Hagoort, language is the key to understanding the organisation of the human mind.

Modern neuroscientists often think of language – and cognition – as ‘embodied’. In this view, language is tightly linked to our body, a byproduct of perception and action. The more traditional view holds that language is a symbolic tool for thinking.

According to Peter Hagoort, the symbolic view of language should not be abandoned too soon. His ‘language marker hypothesis’ claims that language has given us a rich symbolic system for reasoning, which is able to interact with our bodily sensations and emotions. Hagoort shows how language makes humans stand out in the animal kingdom, in comparison to for instance apes, seals and bats.

Hagoort, P. (2022). The language marker hypothesis. Cognition. Link

The post Language as a Marker of the Mind first appeared on Max Planck Neuroscience.

Commonalities and asymmetries in the neurobiological infrastructure for language production and comprehension

Helena.Decker — Mon, 29 Nov 2021 20:06:03 +0000

Giglio, L., Ostarek, M., Weber, K., & Hagoort, P. (2021). Commonalities and asymmetries in the neurobiological infrastructure for language production and comprehension. Cerebral Cortex. Advance online publication.
Article Link

The post Commonalities and asymmetries in the neurobiological infrastructure for language production and comprehension first appeared on Max Planck Neuroscience.

CHANGING THE CONNECTION BETWEEN THE HEMISPHERES AFFECTS SPEECH PERCEPTION

MPFI — Thu, 11 Feb 2021 20:36:47 +0000

When we listen to speech sounds, our brain needs to combine information from both hemispheres. How does the brain integrate acoustic information from remote areas? In a neuroimaging study, a team of researchers led by the Max Planck Institute of Psycholinguistics, the Donders Institute and the University of Zurich applied electrical stimulation to participants’ brains during a listening task. The stimulation affected the connection between the two hemispheres, which in turn changed participants’ listening behaviour.

When we listen to speech sounds, the information that enters our left and right ear is not exactly the same. This may be because acoustic information reaches one ear before the other, or because the sound is perceived as louder by one of the ears. Information about speech sounds also reaches different parts of our brain, and the two hemispheres are specialised in processing different types of acoustic information. But how does the brain integrate auditory information from different areas?

To investigate this question, lead researcher Basil Preisig from the University of Zurich collaborated with an international team of scientists. In an earlier study, the team discovered that the brain integrates information about speech sounds by ‘balancing’ the rhythm of gamma waves across the hemispheres—a process called ‘oscillatory synchronisation’. Preisig and his colleagues also found that they could influence the integration of speech sounds by changing the balancing process between the hemispheres. However, it was still unclear where in the brain this process occurred.

Did you hear ‘ga’ or ‘da’?

The researchers decided to apply electric brain stimulation (high density transcranial alternating current stimulation or HD-TACS) to 28 healthy volunteers while their brains were being scanned (with fMRI) at the Donders Centre for Cognitive Neuroimaging in Nijmegen. They created a syllable that was somewhere in between ‘ga’ and ‘da’, and played this ambiguous syllable to the right ear of the participants. At the same time, the disambiguating information was played to the left ear. Participants were asked to indicate whether they heard ‘ga’ or ‘da’ by pressing a button. Would changing the connection between the two hemispheres also change the way the participants integrated information played to the left and right ear?

The scientists disrupted the ‘balance’ of gamma waves between the two hemispheres, which in turn affected what the participants reported to hear (‘ga’ or ‘da’).

Phantom perception

“This is the first demonstration in the auditory domain that interhemispheric connectivity is important for the integration of speech sound information”, says Preisig. “This work paves the way for investigating other sensory modalities and more complex auditory stimulation”. “These results give us valuable insights into how the brain’s hemispheres are coordinated, and how we may use experimental techniques to manipulate this” adds senior author Alexis-Hervais Adelman.

The findings, to be published in PNAS, may also have clinical implications. “We know that disturbances of interhemispheric connectivity occur in auditory ‘phantom’ perceptions, such as tinnitus and auditory verbal hallucinations”, Preisig explains. “Therefore, stimulating the two hemispheres with (HD-)TACS may offer therapeutic benefits. I will follow up on this research by applying TACS in patients with hearing loss and tinnitus, to improve our understanding of neural attention control and to enhance speech comprehension for this group.”

Basil C. Preisiga, Lars Riecked, Matthias J. Sjerpsa, Anne Kösema, Benjamin R. Kopa, Bob Bramsona, Peter Hagoorta, and Alexis Hervais-Adelman (2021). Selective modulation of interhemispheric connectivity by transcranial alternating current stimulation influences binaural integration. PNAS 118, 7, e2015488118.
Article Link

The post CHANGING THE CONNECTION BETWEEN THE HEMISPHERES AFFECTS SPEECH PERCEPTION first appeared on Max Planck Neuroscience.

Word contexts enhance the neural representation of individual letters in early visual cortex

Helena.Decker — Fri, 11 Dec 2020 15:46:51 +0000

Visual context facilitates perception, but how this is neurally implemented remains unclear. One example of contextual facilitation is found in reading, where letters are more easily identified when embedded in a word. Bottom-up models explain this word advantage as a post-perceptual decision bias, while top-down models propose that word contexts enhance perception itself. Here, we arbitrate between these accounts by presenting words and nonwords and probing the representational fidelity of individual letters using functional magnetic resonance imaging. In line with top-down models, we find that word contexts enhance letter representations in early visual cortex. Moreover, we observe increased coupling between letter information in visual cortex and brain activity in key areas of the reading network, suggesting these areas may be the source of the enhancement. Our results provide evidence for top-down representational enhancement in word recognition, demonstrating that word contexts can modulate perceptual processing already at the earliest visual regions.

Micha Heilbron, David Richter, Matthias Ekman, Peter Hagoort, and Floris P. de Lange. (2020). Word contexts enhance the neural representation of individual letters in early visual cortex. Nature Communications 11, 321.
Article Link

The post Word contexts enhance the neural representation of individual letters in early visual cortex first appeared on Max Planck Neuroscience.

Adaptation In Single Neurons Provides Memory For Language Processing

Helena.Decker — Wed, 12 Aug 2020 14:47:56 +0000

To understand language, we have to remember the words that were uttered and combine them into an interpretation. How does the brain retain information long enough to accomplish this, despite the fact that neuronal firing events are very short-lived? Hartmut Fitz from the Max Planck Institute for Psycholinguistics and his colleagues propose a neurobiological explanation bridging this discrepancy. Neurons change their spike rate based on experience and this adaptation provides memory for sentence processing.

Did the man bite the dog, or was it the other way around? When processing an utterance, words need to be assembled into the correct interpretation within working memory. One aspect of comprehension is to establish ‘who did what to whom’. This process of unification takes much longer than basic events in neurobiology, like neuronal spikes or synaptic signaling. Hartmut Fitz, lead investigator at the Neurocomputational Models of Language group at the Max Planck Institute for Psycholinguistics, and his colleagues propose an account where adaptive features of single neurons supply memory that is sufficiently long-lived to bridge this temporal gap and support language processing.

Model comparisons

Together with researchers Marvin Uhlmann, Dick van den Broek, Peter Hagoort, Karl Magnus Petersson (all Max Planck Institute for Psycholinguistics) and Renato Duarte (Jülich Research Centre, Germany), Fitz studied working memory in spiking networks through an innovative combination of experimental language research with methods from computational neuroscience.

In a sentence comprehension task, circuits of biological neurons and synapses were exposed to sequential language input which they had to map onto semantic relations that characterize the meaning of an utterance. For example, ‘the cat chases a dog’ means something different than ‘the cat is chased by a dog’ even though both sentences contain similar words. The various cues to meaning need to be integrated within working memory to derive the correct message. The researchers varied the neurobiological features in computationally simulated networks and compared the performance of different versions of the model. This allowed them to pinpoint which of these features implemented the memory capacity required for sentence comprehension.

Towards a computational neurobiology of language

They found that working memory for language processing can be provided by the down-regulation of neuronal excitability in response to external input. “This suggests that working memory could reside within single neurons, which contrasts with other theories where memory is either due to short-term synaptic changes or arises from network connectivity and excitatory feedback”, says Fitz.

Their model shows that this neuronal memory is context-dependent, and sensitive to serial order which makes it ideally suitable for language. Additionally, the model was able to establish binding relations between words and semantic roles with high accuracy.

“It is crucial to try and build language models that are directly grounded in basic neurobiological principles,” declares Fitz. “This work shows that we can meaningfully study language at the neurobiological level of explanation, using a causal modelling approach that may eventually allow us to develop a computational neurobiology of language.”

Fitz, H., Uhlmann, M., Van den Broek, D., Duarte, R., Hagoort, P., & Petersson, K. M. (2020). Neuronal spike-rate adaptation supports working memory in language processing. Proceedings of the National Academy of Sciences of the United States of America. 117 (34) 20881-20889.
Article Link

The post Adaptation In Single Neurons Provides Memory For Language Processing first appeared on Max Planck Neuroscience.

Constrained Structure of Ancient Chinese Poetry Facilitates Speech Content Grouping

Helena.Decker — Mon, 06 Apr 2020 13:23:44 +0000

Ancient Chinese poetry is constituted by structured language that deviates from ordinary language usage [1, 2]; its poetic genres impose unique combinatory constraints on linguistic elements [3]. How does the constrained poetic structure facilitate speech segmentation when common linguistic [4, 5, 6, 7, 8] and statistical cues [5, 9] are unreliable to listeners in poems? We generated artificial Jueju, which arguably has the most constrained structure in ancient Chinese poetry, and presented each poem twice as an isochronous sequence of syllables to native Mandarin speakers while conducting magnetoencephalography (MEG) recording. We found that listeners deployed their prior knowledge of Jueju to build the line structure and to establish the conceptual flow of Jueju. Unprecedentedly, we found a phase precession phenomenon indicating predictive processes of speech segmentation—the neural phase advanced faster after listeners acquired knowledge of incoming speech. The statistical co-occurrence of monosyllabic words in Jueju negatively correlated with speech segmentation, which provides an alternative perspective on how statistical cues facilitate speech segmentation. Our findings suggest that constrained poetic structures serve as a temporal map for listeners to group speech contents and to predict incoming speech signals. Listeners can parse speech streams by using not only grammatical and statistical cues but also their prior knowledge of the form of language.

Teng, X., Ma, M., Yang, J., Blohm, S., Cai, Q., & Tian, X. (2020). Constrained structure of ancient Chinese poetry facilitates speech content grouping. Current Biology 30, 7, P1299-1305.E.
Article Link

The post Constrained Structure of Ancient Chinese Poetry Facilitates Speech Content Grouping first appeared on Max Planck Neuroscience.