Multi-Phase Probabilistic Optimizers for Deep Learning

• Applicant: Professor Dr. Philipp Hennig
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Mathematics
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 463889763

This proposal to SPP 2298/1 proposes to investigate a novel paradigm for the training of deep neural networks. The peculiarities of deep models, in particular strong stochasticity (SNR<1), preclude the use of classic optimization algorithms. And contemporary alternatives, of which there are already many, are wasteful with resources. Rather than add yet another optimization rule to the long and growing list of such methods, this project aims to make substantial conceptual progress by investigating two key ideas: First, leveraging the entire *probability distribution* of gradients across the dataset (empirically estimated at runtime) to identify algorithmic parameters that the user would otherwise have to manually tune. And second, splitting the optimization process into at least three distinct *phases* with differing mathematical objectives. The goal is to develop (concretely, as a software library) an optimizer that requires no manual tuning, and achieves good generalization performance without repeated re-starts.

• DFG Programme: Priority Programmes
• Subproject of: SPP 2298: Theoretical Foundations of Deep Learning