Pacific Institute for the Mathematical Sciences - PIMS

Non-negative Gauss-Newton Methods for Empirical Risk Minimization

We consider the problem of minimizing the average of a large number of smooth but possibly non-convex functions. For machine learning applications, the loss functions are mostly non-negative and thus can be written as the composition of the square and their real-valued square roots. With such a simple reformulation, we can apply the Gauss-Newton method, or the Levenberg-Marquardt method with an extra quadratic regularization. We show that the resulting algorithms are highly adaptive and can automatically warm up and decay the effective step size while tracking the loss landscape. We provide convergence analysis of the methods in convex, non-convex and stochastic settings. Both the convergence rates and empirical evaluations compare favorably to the classical (stochastic) gradient method. This is joint work with Antonio Orvieto.

Speaker biography: Lin Xiao is a Research Scientist Manager on the Fundamental AI Research (FAIR) team at Meta. He received his Ph.D. from Stanford University in 2004, spent two years as a postdoctoral fellow at California Institute of Technology, and then worked at Microsoft Research until 2020 before joining Meta. His current research focuses on optimization theory and algorithms for deep learning and reinforcement learning.

Location: Gates Commons (CSE 691) Allen Center & Online. Livestream link

Time: 3.30pm Pacific