Multi-stage integration during co-speech gesture comprehension: a systematic investigation combining EEG and fMRI

• Applicant: Dr. Yifei He
• Subject Area: General, Cognitive and Mathematical Psychology ; Human Cognitive and Systems Neuroscience
• Term: from 2019 to 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 430897432

Gesture is a substantial component of interpersonal communication. The processing of speech-related gestures requires the integration of different sensory modalities, i.e., the meaningful combination of auditory and visual input. However, gesture-speech integration (GSI) is not only multisensory in nature, but at a higher semantic level. This unique feature has rendered GSI an emerging topic in cognitive neuroscience.Based on previous fMRI, EEG, and simultaneous EEG-fMRI studies on GSI, we proposed an MS-GSI model (the multi-stage model of gesture-speech integration), to account for the neurobiological basis of GSI. The MS-GSI model hypothesizes that GSI consists of cascaded sub-stages from lower-order audio-visual perception and integration, to higher-level semantic matching and revision processes, and that these stages can be characterized by distinct and dissociable neural markers from EEG and fMRI. The model forms a theoretical basis for further consolidation and empirical testing concerning GSI. However, there remain several challenges: firstly, there is limited effort so far to directly unify the neural markers from EEG and fMRI to the sub-stages of GSI. Secondly, the research paradigms, which are mostly adopted from studying of lower-level multisensory integration, may require further refinements for the investigation of semantic-level GSI. Thirdly, while semantic complexity of gesture plays a pivotal role during GSI, it is largely ignored in the existing literature. Lastly, despite the prevalence of top-down influence from attention/prediction to unisensory perception and language comprehension, it is not clear how they modulate different levels of GSI in a top-down manner.To resolve these issues, we propose the current research proposal with three work packages, using EEG and simultaneous EEG-fMRI to 1) investigate the interaction between semantic congruency and semantic complexity during GSI, 2) examine the neural basis of gestural metaphoricity, a semantic complexity that derives predominantly from gesture, with a new design that circumvents confounds from existing multisensory paradigms, and 3) testing the top-down influence of semantic prediction on lower-level GSI processes. Each work package targets at specific sub-stages of GSI, and will reveal ground breaking new insights into the interplay between sensory perception and higher-level perceptual, semantic, and constructive brain processes relevant for social communicative functioning. In sum, the current research project will help us to better understand the neurobiological basis of different levels of processing with the MS-GSI model. Such understanding may not only optimize the neurobiological models concerning GSI, but also form a solid basis for translational applications to clinic populations, which were shown to be impaired GSI, a process which is highly socially relevant.

• DFG Programme: Research Grants
• Co-Investigator: Professor Benjamin Straube, Ph.D.