Scene Dynamics Prediction using Physically based Simulation (P5)

• Applicant: Professor Dr. Reinhard Klein
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2017 to 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 313421352

The first phase of the project focused on the real-time reconstruction of static scenes and the creation of activity maps. For this purpose, geometric data as well as data on material properties were used. Both data sources provide valuable information for applications such as service robot navigation and object manipulation. The main goal of the second project phase is the anticipation of scene dynamics. The planned investigations cover 3D reconstruction, physical-based simulation including inference of material parameters in dynamic environments in real time. In order to focus our research, we first limit ourselves to anticipating the dynamics of textiles as required by robots to properly handle textiles. To this end, we will first extend the approaches to real-time reconstruction of static scenes developed in the first project phase to include scene dynamics and the deformation of objects. In addition to the reconstruction of the rough geometry, we also plan to capture the underlying structures of the textiles, such as weaving or knitting patterns of fabrics, in order to derive physical material parameters that are relevant for the dynamic behaviour. The focus is on a differentiable physically based fabric simulation. The resulting simulation is to be combined with new approaches for efficient 3D scene segmentation in order to derive a parametrized model that fits to the observed data and allow for the prediction of the corresponding dynamics of objects. The latter enables e.g. a robot to compare the anticipated movement of a textile when gripping it with real observations and to correct it if necessary.

• DFG Programme: Research Units
• Subproject of: FOR 2535: Anticipating Human Behavior