Automatic labelling of anatomies in large-scale medical image datasets through self-supervised and multimodal learning

• Applicant: Professor Dr. Mattias Heinrich
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Medical Physics, Biomedical Technology
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 500498869

The analysis of medical volume data has made great progress in recent years based on novel deep learning methods. However, large-scale datasets that reflect a large cross-section of the population and reliably recognise both normal anatomy and abnormalities are missing for successful and wide-spread deployment in healthcare applications. The scientific goal of the project is to develop robust, automatic and efficient algorithms for the segmentation of internal organs, bones and body surfaces based on 3D MRI scans from the large-scale NAKO population study with 30,000 volumes. Here, methods of learning-based multimodal registration and learning from non-annotated 3D datasets from the preceding DFG project will be further developed and exploited in a software demonstrator. In the context of this knowledge transfer, the project partners will systematically extend the complementary prior work of Fraunhofer MEVIS and the University of Lübeck to integrate a combination of self-supervised pre-training, multimodal transfer learning, image registration and segmentation in novel deep learning methods. These will then be applied to automatically segment several thousand 3D MRI volumes from the NAKO population study and evaluate them on a validation dataset compared to the manual gold standard. Together with the application partner Philips, uncertainties and anomalies will be estimated from the segmentation models and a 3D geometric atlas will be created that will use point cloud networks to localise internal anatomies based on body surfaces. Source code and trained models, of the neural networks developed at UzL, as well as anatomical labels of the NAKO data will be made freely available to the research community. Together, a demonstrator (TRL 6-7) will be created during the project, which will realise a higher automation of the MRI acquisition process in clinical routine using surface-based anatomy recognition based on depth images, offering a significant economic advantage and as well as an acceleration of the acquisition process and enabling further exploitation possibilities through the automatic analysis of the scans.

• DFG Programme: Research Grants (Transfer Project)
• Application Partner: Philips Medical Systems Development and Manufacturing Centre (DMC) GmbH
• Cooperation Partner: Dr. Stefan Heldmann