Quasi-Newton methods for solving nonsmooth optimization problems in learning and control

This thesis is concerned with the design and analysis of some quasi-Newton methods for solving optimization problems in machine learning and control of nonlinear dynamical systems. The proposed algorithms are designed to exploit approximate second-order information to improve convergence rates, stability, and generalization of the learned models and control policies. The thesis is organized into two main parts. In the first part, we present a generalized Gauss-Newton algorithm that uses an adaptive step-size selection strategy and preserves the affine-invariant property of Newton’s method. This algorithm significantly reduces the computational cost of Gauss-Newton methods, particularly in mini-batch supervised learning. We then extend this with a proximal method for nonsmooth convex composite optimization, resulting in two new algorithms. In the second part, we treat learning and control problems in the training of neural networks. First, we present a rigorous theoretical study of the generalized Gauss-Newton algorithm for the optimization of feedforward neural networks. This study establishes a non-asymptotic guarantee for the convergence of feedforward neural networks with a general explicit regularizer. Then, an inexact sequential quadratic programming framework is proposed for optimal control in recurrent neural networks, using a two-stage approach for system identification and optimal control policy selection. Several practical applications of all the proposed algorithms are demonstrated through numerical experiments.