Multiscale Dynamics of Neural Nets via Stochastic Graphops

• Applicants: Dr. Maximilian Engel; Professor Christian Kühn, Ph.D.
• Subject Area: Mathematics
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 463883089

Deep learning in artificial neural networks has demonstrated to be extremely powerful and flexible regarding highly complicated analytical and predictive tasks in a large variety of applications. A complete theoretical, in particular mathematical, understanding and explanation of deep neural networks and associated algorithmic procedures is yet to be found. In the proposed research project, we aim at combining several recently developed mathematical approaches from dynamical systems theory to provide a new rigorous framework for machine learning in the context of deep neural networks.We will analyze deep neural networks by identifying two different dynamical time scales which are then combined in a stochastic multiscale dynamical system. On the one hand, we will study the "fast" dynamics of information propagation through a large adaptive network by deriving and analyzing novel limiting integro-partial differential equations in the infinite network limit, using and extending the theory of graph operators, also called graphops. On the other hand, we will investigate the "slow" stochastic weight dynamics for adapting the coefficients of the neural network, representing the learning procedure, with a focus on stochastic gradient descent, its metastability properties and the random dynamical systems interpretation.The two dynamical scales will then be combined in a full model of the neural network dynamics, where we can study the interplay between stochastic learning, dynamical robustness, and analytic expressivity, explaining and predicting particular patterns and their representational significance. The mathematical tools will comprise stochastic dynamics, ergodic theory, adaptive networks, graph limits and multiscale dynamics.

• DFG Programme: Priority Programmes
• Subproject of: SPP 2298: Theoretical Foundations of Deep Learning