Forward and Differentiable Simulation of L2S Sensor Data

• Applicant: Professor Dr.-Ing. Andreas Kolb
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 459284860

The overall goal of the Learning to Sense (L2S) research unit, i.e., the joint optimization of the design parameters of a sensor system and the associated neural network to analyze the resulting data in an end-to-end machine learning fashion, requires a large amount of training data for different sensor and scene configurations. Since collecting training and test data with real sensors is costly or partially not possible at all, simulation of the sensor data formation process is a key success factor to establish the link between sensor system parameters and the given application task.This subproject focuses on the efficient simulation of the sensor data formation process, which includes the simulation of physical, real-world effects occurring in the scene at different wavelengths (visual between 0.4-0.7µm and terahertz (THz) between 0.4-0.6mm) for a potentially large number of up to 10³ frequencies. This also includes the simulation of coherent radiation and material interaction using complex refractive indices and synthetic, i.e., unfocused imaging methods, and aspects of sensor system design, e.g., pixel and spectral filter placement.Technically, the focus of this project is on forward as well as differentiable simulation of sensor data, enabling application and hardware development based on machine learning for arbitrary sensor and scene parameters. In this context, three main aspects are investigated. First, the design and development of a simulation framework capable of simulating both focused imaging in the visual domain and unfocused coherent THz radiation, including the conversion of incident radiation to sensor data, is considered. The second research focus relates to the extension of existing forward simulation approaches to achieve high-performance path-tracing simulation techniques that produce physically plausible sensor outputs and allows end-to-end mapping of scene and sensor parameters to the resulting photoelectric properties. Third, efficient differentiable methods for the simulation process will be developed to support machine learning to efficiently identify optimal sensor parameters.To achieve these goals, the project will work closely with hardware projects P4 on sensor layout and on-chip calculations, P6 on simulation of wave-optical effects, and P7 regarding simulation of coherent THz- radiation and material interaction. In addition, there is intensive collaboration with machine learning projects P1 regarding the handling of irregular sensor arrangements, P2 on image classification and semantic segmentation, and P3 regarding localization and object reconstruction directly in the THz frequency domain.

• DFG Programme: Research Units
• Subproject of: FOR 5336: KI-FOR Learning to Sense