The Hierarchical Structure of Abstract Scene Representations

• Applicant: Professorin Dr. Melissa Le-Hoa Vo
• Subject Area: General, Cognitive and Mathematical Psychology ; Human Cognitive and Systems Neuroscience
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 459426179

A key feature of the efficient human mind is the ability to abstract away from specific previous experiences and build abstract representations that allow for their successful application to our constantly changing environment. That is, just like we are able to understand sentences we have never heard before because we know the meaning of the words and the rules of how they need to be arranged to form meaning, we can easily understand new environments by accessing the identities of objects and knowing the rules that govern their spatial arrangements (scene grammar; Võ, 2021). Along the lines of ARENA’s overarching objectives, the aim of this project is therefore to understand the emergence and composition of abstract representations of objects in real-world scenes which allow for efficient behavior. We intend to do pursue this aim by systematically probing the existence of a hierarchical structure in scene grammar using different input modalities (e.g. images vs. words) and different tasks (e.g. memorization vs. search). A key assumption concerns the existence of subclusters within scenes – “phrases” – that again consist of a larger object ¬– an “anchor object” – which in turn predicts the identity and location of other, smaller objects. In order to interact with our environment, it is important to establish internal representations that parallel the structure of the external world, e.g. by setting up predictions that allow for more efficient object identification or localization. Thanks to the multi-disciplinary composition of ARENA, we will be able to test our hypothesis by joining forces with our colleagues from computer science. For instance, we will compare behavioral and neural responses to the presentations of objects and scenes with outputs of early versus late layers of deep neural networks. Our comprehensive approach will include three core lines of research to ensure a thorough, principled investigation of I) the structure of abstract scene representations elicited by either images of objects or word labels, II) the content and structure of existing search templates estimated from brain responses using EEG and fMRI, and III) the generation and modification of new abstract object representations over the course of interactions in artificial 3D VR environments. Successfully tackling the proposed aims could have transformational effects on how we study cognitive processes in complex, naturalistic settings within computer science and psychology, and in a next step could inform the development of new, more efficient computational models of scene understanding.

• DFG Programme: Research Units
• Subproject of: FOR 5368: KI-FOR Abstract Representations in Neural Architectures (ARENA)
• International Connection: USA
• Cooperation Partner: Professor Paul Sajda, Ph.D.