DMS Applied and Computational Mathematics Seminar

Speaker: Yi Liu (Auburn University)

Title: Convergence Analysis of the ADAM Algorithm for Linear Inverse Problems

Abstract:  The ADAM algorithm is one of the most popular stochastic optimization methods in machine learning. Its remarkable performance in training models with massive datasets suggests its potential efficiency in solving large-scale inverse problems. In this work, we apply the ADAM algorithm to solve linear inverse problems and establish the sub-exponential convergence rate for the algorithm when the noise is absent. Based on the convergence analysis, we present an a priori stopping criterion for the ADAM iteration when applied to solve inverse problems at the presence of noise. The convergence analysis is achieved via the construction of suitable Lyapunov functions for the algorithm when it is viewed as a dynamical system with respect to the iteration numbers. At each iteration, we establish the error estimates for the iterated solutions by analyzing the constructed Lyapunov functions via stochastic analysis. Various numerical examples are conducted to support the theoretical findings and to compare with the performance of the stochastic gradient descent (SGD) method.