Empirical and computational study of iconic gesture-speech integration and its development in preschool children (EcoGest)

• Applicants: Professorin Dr. Friederike Kern; Professor Dr.-Ing. Stefan Kopp, since 8/2018; Professorin Dr. Katharina Rohlfing
• Subject Area: General and Comparative Linguistics, Experimental Linguistics, Typology, Non-European Languages ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2017 to 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 322326507

Gesture and speech form an integrated communicative system. This integration has been investigated for the purpose of communication as well as learning. Recent research attempting to explain and model this integration focus on how gesture influences speech, and how gesture -- as a special form of action -- forces the speaker to incorporate sensorimotor information into the representation. In these cognitive approaches, the fact that gesture is employed in various communicative genres and thus interacts with speech for different discursive purposes is disregarded. Yet types of communicative genres have been shown to bear different cognitive and communicative requirements on the speaker and hearer: For example, a narration requires different structural knowledge and verbal skills from a speaker than an explanation. While for competent speakers, the appearance of gestures in different communicative genres might occur fluent, for children learning to cope with the variety of discursive contexts, the role of gesture for thinking and communicating might be better observable: One possibility is that for particular genres, gestures are required resources for depicting strategies facilitating mutual understanding, and children discover them with increasing experience in interaction and with growing cognitive and linguistic competencies. Another -- not exclusive -- possibility is that gesture plays an important role on the way to master increasingly changing communicative requirements, which becomes visible when children learn to produce different communicative genres. Thus, in addition to sensorimotor information that is recruited for the act of speaking, gesture might be an integral resource of speaking in relation to different communicative purposes. This possibility raises new questions for models of gesture-speech integration: Is gesture used in dependence of the experience in communicative genre and if so how is the pragmatic aspect incorporated into the production of speech? In our proposed project, we apply qualitative and quantitative analyses as well as computational evaluation methods in investigating longitudinally how the integration of gesture and speech develops in 4 and 5 years old preschool children and how context (meaning the communicative genre and physical surroundings) relates to the different gestural techniques that children use in interaction. The results of our project will provide a crucial contribution towards a comprehensive developmental theory of gesture and speech integration involving cognitive and pragmatic aspects of speaking.

• DFG Programme: Research Grants
• International Connection: USA
• Cooperation Partners: Dr. Erica Cartmill; Marjorie Harness Goodwin
• Ehemalige Antragstellerin: Dr.-Ing. Kirsten Bergmann, until 7/2018