FOR 2987:  Learning and Simulation in Visual Computing

• Subject Area: Computer Science, Systems and Electrical Engineering
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 413611294

In recent years, we have seen tremendous progress in machine learning algorithms for processing visual data, including images, videos, geometric models, and volumetric fields. Specifically the developments in deep learning have created incredible attention in many research communities. In particular in computer vision, these developments have led to significant improvementsin tasks such as image classification or semantic segmentation. However, despite the recent success of deep learning methods, there are still fundamental challenges faced by researchers and practitioners who want to apply these methods for solvingreal-world problems: Automated Training Data Generation and Self-Supervision Knowledge Transfer Between Domains Analysis and Understanding of Learning ProcessesLeveraging simulated data provides a unique opportunity in this area since it allows for generating perfect ground truth for training data. For instance, instead of annotating millions of images, one can possibly render synthetic images with ground truth labels from an existing virtual environment. This is particularly important for higher dimensional data such as temporally-consistentannotations in videos (e.g., motion, optical flow, etc.) or volumetric labels in 3D scenes or physics simulations (e.g., smoke, fluids, cloths, etc.). At the same time, the simulated data offers full control of the training data generation process: data imbalances can be created or avoided, data can be augmented to learn certain invariances, or arbitrary physics effects, such as lighting, can bealtered while monitoring the training process. This level of control also opens up new avenues, for instance, in the context of analyzing training processes based on specific dataset statistics; this enables the use of analytic techniques to better understand the learning processes, their sensitivity to the training data as well as their capabilities to encode specific data features.The major challenge then becomes the domain transfer between simulated and real data in order to apply the learned results and insights from the synthetic setting to the real world. Realizing and leveraging this domain transfer is the main goal of this proposal.

• DFG Programme: Research Units

Projects

• Differentiable Physics Simulators for Realistic 3D Reconstruction (Applicant Cremers, Daniel )
• Domain Transfer with Generative Models and Neural Rendering (Applicant Nießner, Matthias )
• Learning to Sample for Visual Computing (Applicant Westermann, Rüdiger )
• Synthetic data and unsupervised annotations for data-driven video analysis. (Applicant Cremers, Daniel )
• Transfer and Representation Learning for Physical Simulations (Applicant Thuerey, Nils )

• Spokesperson: Professor Dr.-Ing. Matthias Nießner