Architectures for Non-Image-Generating L2S Vision Systems

• Applicant: Professor Dr. Volker Blanz
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 459284860

The main goal of the Learning-to-Sense (L2S) research unit is to jointly optimize the design parameters of an application-specific sensor along with a neural network to analyze the resulting data.In this subproject, we explore and implement L2S architectures and machine learning (ML) algorithms for a number of novel sensor types that we develop together with partners in the research group. These sensors are for visible light and for mm-wave and THz synthetic aperture radar (SAR).The common feature of the proposed sensors is that they are (a) not designed for generating images, unlike cameras and many existing sensors, and (b) they use differential, contrast-based measurements. For visual light, the new sensors are partially motivated by findings in the human visual system, and they have a potential of being superior to standard cameras in high dynamic range environments. We propose to built sensors that have center-surround receptive fields and opponent-color coding (similar to retinal ganglion cells) and that calculate spatial and temporal derivatives on the analog signal on chip. In the research unit, we also develop and explore foveated sensors, and the focus of this subproject is on differential low-bandwidth data from the periphery of the visual field. For mm-waves and THz radiation, the proposed differential sensors remove substantial parts of the background signal which is due to direct reflections and multipath scattering, giving them the potential to simplify the inverse problems that are involved in computer vision.While many standard cameras and sensors can be optimized by improving on measures of image quality, the non-image-generating sensors can only be optimized and trained in an end-to-end application scenario with specific ML tasks. We explore such settings in the contexts of face and pedestrian detection, optical flow, peripheral vision and localization of structures in mm-wave and THz SAR. The results of this project contribute to the general development of the L2S paradigm, and provide several new and experimental approaches to computer vision.

• DFG Programme: Research Units
• Subproject of: FOR 5336: KI-FOR Learning to Sense