How non-verbal cues determine the interpretation of pointing gestures

• Applicant: Privatdozent Dr. Oliver Herbort
• Subject Area: General, Cognitive and Mathematical Psychology
• Term: since 2018
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 410915886

Pointing gestures are a cornerstone of human communication and have garnered significant attention over recent decades. Surprisingly, how humans infer locations in the environment from pointing gestures is yet only partially understood. The initial phase of this research project addressed this issue and allowed for three overarching conclusions. First, pointing gestures are subjectively ambiguous, because humans may use incomesurable rules to interpret them. Second, further ambiguity is added because pointing gestures seem to be linked quite flexibly to entities in the environment. Third, the project confirmed that the form of a pointing gesture carries meaning beyond the spatial information conveyed by the position and orientation of the pointing finger. Building on these findings, we aim to further our research on the comprehension of pointing gestures. Our proposal explores how addressees use non-verbal cues besides the position and orientation of the pointing finger to resolve the ambiguity of a pointing gesture, both in terms of which interpretation rule to use (and thus which location is perceived as pointed-at) and whether the pointer refers to a larger, global object or one of its local sub-components (e.g., whether pointing at the Azores implies the island group or the Atlantic) .A series of experiments tests whether the pointer’s gaze direction, arm configuration, and the dynamics of the pointing gesture carry meaning for the addressee, reduce pointing ambiguity, and consequently co-determine interpretations. Additionally, we examine the extent to which any effects are purely evoked by the gestural form or depend on a more comprehensive understanding of situational context and the pointer’s characteristics. Experiments will be conducted by presenting naturalistic virtual reality scenes. The scenes will include computer-generated pointers and several larger objects with distinct local subcomponents. Thus, in any scene, different interpretations of pointing gestures are conceivable, depending on which interpretation rule the addressee selects and whether they favor a global or local interpretation. In all experiments, the spatial information conveyed by the pointing finger will be held constant while other non-verbal cues, such as the pointer’s gaze direction, are systematically manipulated. We test the hypotheses that such cues are picked up and used to resolve pointing ambiguity by asking participants to freely describe the scene as well as asking to rate their agreements to different potential interpretations. We anticipate that the proposed project will enhance our understanding of how humans process pointing gestures. Furthermore, the knowledge gained could facilitate pointing-based communication in everyday situations, such as teaching, and improve the communication between humans and robots, virtual agents, or human-controlled avatars.

• DFG Programme: Research Grants