Random Matrix Approaches to Approximate Bayesian Inference in Machine Learning

• Applicant: Professor Dr. Manfred Opper
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Electronic Semiconductors, Components and Circuits, Integrated Systems, Sensor Technology, Theoretical Electrical Engineering ; Statistical Physics, Nonlinear Dynamics, Complex Systems, Soft and Fluid Matter, Biological Physics
• Term: from 2018 to 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 407712271

The Bayesian paradigm provides important methods for learning from data. It combines a probabilistic model for the generation of data together with prior knowledge over likely parameters within a probability distribution over the parameters of the model. However, practical applications of this idea to models with a large number of parameters are often plagued by computational problems related to the intractability of high--dimensional probability distributions. Approximate inference methods of machine learning provide algorithms for approximating such distributions by simpler ones - typically by multivariate Gaussian distributions. These inference methods yield often excellent results in applications. But, the update of covariance matrices (which give important information on uncertainties and dependencies between variables) of these Gaussian distributions within the iterative inference algorithms requires matrix operations per iteration which grows cubic in the number of parameters of the model. This makes the applications of such methods problematic when the number of variables is large. Hence, further approximations are necessary. And these may deteriorate the quality of the predictions. A second relevant problem is the fact that there is no guarantee of convergence for some popular inference algorithms. It is unclear if the failure to converge is an artefact of the algorithm or is related to the complexity of the Bayesian model. Motivated by recent research in the fields of information theory and statistical physics, this project will address these problems from a new angle. Assuming that data matrices can be considered as random (in a mathematically well-defined way), results of random matrix theory suggest novel ways to efficiently approximate the required matrix operations. These approximations are expected to perform well in the asymptotic limit when matrices are large. Random matrix methods will also provide new ways for analyzing the performance of iterative inference algorithms for large problems under certain statistical assumptions on the data. We will use these random matrix techniques to speed up existing algorithms as well as designing novel algorithms with optimized convergence properties. We will investigate the quality and robustness of such methods. Finally, we will validate our approach on various Bayesian models in machine learning and compare the performance with that of competing methods on simulated as well as real data.

• DFG Programme: Research Grants
• International Connection: Denmark, Switzerland
• Cooperation Partners: Professor Giuseppe Caire, Ph.D.; Dr. Nicolas Macris; Professor Dr. Ole Winther