L2S-Training with Continuous Sensor System Parameters and Irregular Data

• Applicant: Professor Dr. Michael Möller
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 459284860

The main idea of the Learning to Sense (L2S) research unit is to jointly optimize design parameters of a sensor system along with a neural network to analyze the resulting data on specific tasks. Degrees of freedom in the sensor system design include the spatial and spectral layout of the sensor as well as the pixel shape, the optics of the sensor system, choices of active illumination, and on-chip processing capabilities. We expect a joint, true end-to-end optimization of sensor systems and neural networks to have significant benefits in terms of the resulting performance, its efficiency with respect to memory and power consumption, and its hunger for recorded and annotated training data. This subproject will study several fundamental machine learning aspects for realizing the aforementioned L2S concept. First, as soon as the sensor layout itself is subject to an optimization, the network architectures analyzing the produced data need to be adaptive to changes in the geometric distribution of the data. Thus, I will study how graph and function-valued neural networks can be used to handle the representation changes during the optimization. Besides the input, the representation of the output of the neural network needs to be addressed. Second, whenever the data formation process of the sensor system is highly complex and does not yield classical image data, e.g. in the case of coded optics and illumination, I expect the inclusion of physical knowledge into the network architecture to be crucial. Providing the network with such knowledge should avoid a data-hungry learning of known physical relations.Third, once the architecture is chosen, the joint optimization for network and sensor system parameters is non-trivial. I will address this optimization for the case of continuously adaptable sensor system parameters with differentiable renderers and lay a particular focus on how real data can be included into the training process. Finally, this subproject will consider training schemes that aim at an improved generalization by separating the data sets the network and sensor system parameters are trained on via a bi-level optimization problem. The above fundamental machine learning questions will be exemplified in classification, localization, and reconstruction problems on RGB, Terahertz (THz), and microscopic imaging sensor systems.

• DFG Programme: Research Units
• Subproject of: FOR 5336: KI-FOR Learning to Sense