Harnessing multimodal learning signals to advance ophthalmological imaging

• Applicant: Dr. Martin Menten
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Ophthalmology ; Clinical Neurology; Neurosurgery and Neuroradiology
• Term: since 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 532139938

Deep learning has revolutionized medical imaging, matching or even surpassing expert performance on many tasks like image classification or segmentation. Additionally, it has become an integral component of tools for image acquisition, reconstruction and registration. While indisputably effective, deep learning still has several limitations. It is reliant on large datasets with ground truth annotations, whose acquisition is time-consuming and requires expert knowledge. Furthermore, deep neural networks are prone to exploit spurious correlations in the data while offering only limited interpretability. One potential solution to overcome these limitations is the integration of prior clinical knowledge with deep learning. Examples of such multimodal learning signals include geometrical and topological priors, time series of data, laws of physics or medical texts. These learning signals can be combined with deep learning by explicitly using them as training data. Alternatively, prior knowledge can be implicitly encoded as inductive bias during the design of the network's architecture or training strategy. In the proposed work program, we will investigate strategies to harness multimodal learning signals for accurate, efficient, and interpretable deep learning. These algorithmic innovations will directly inform the development of tools to tackle acute clinical needs in the fields of ophthalmology and neurology. The first of four work packages will explore the choice of non-Euclidean data representations as inductive bias while developing tools for the staging of diabetic retinopathy. The second work package will investigate methods to integrate geometric and topological priors with deep learning and systematically benchmark their effectiveness. This will facilitate the automated prediction of the progression of age-related macular degeneration. The third work package will enhance the acquisition and reconstruction of optical coherence tomography angiography, a non-invasive imaging modality that allows visualization of the retinal vasculature. Physics-based neural networks will facilitate higher image acquisition speeds while mitigating artifacts. Finally, the fourth work package will develop clinical decision support tools that can process a wide range of multimodal clinical information and aid with the treatment of patients with multiple sclerosis.

• DFG Programme: Independent Junior Research Groups