This paper proposes a novel algorithm for training recurrent neural network models of nonlinear dynamical systems from an input/output training dataset. Arbitrary convex and twice-differentiable loss functions and regularization terms are handled by sequential least squares and either a line-search (LS) or a trust-region method of Levenberg–Marquardt (LM) type for ensuring convergence. In addition, to handle non-smooth regularization terms such as ℓ1, ℓ0, and group-Lasso regularizers, as well as to impose possibly non-convex constraints such as integer and mixed-integer constraints, we combine sequential least squares with the alternating direction method of multipliers (ADMM). We call the resulting algorithm NAILS (nonconvex ADMM iterations and least squares) in the case line search (LS) is used, or NAILM if a trust-region method (LM) is employed instead. The training method, which is also applicable to feedforward neural networks as a special case, is tested in three nonlinear system identification problems. © 2023 Elsevier Ltd

Training recurrent neural networks by sequential least squares and the alternating direction method of multipliers

Bemporad, A.
2023-01-01

Abstract

This paper proposes a novel algorithm for training recurrent neural network models of nonlinear dynamical systems from an input/output training dataset. Arbitrary convex and twice-differentiable loss functions and regularization terms are handled by sequential least squares and either a line-search (LS) or a trust-region method of Levenberg–Marquardt (LM) type for ensuring convergence. In addition, to handle non-smooth regularization terms such as ℓ1, ℓ0, and group-Lasso regularizers, as well as to impose possibly non-convex constraints such as integer and mixed-integer constraints, we combine sequential least squares with the alternating direction method of multipliers (ADMM). We call the resulting algorithm NAILS (nonconvex ADMM iterations and least squares) in the case line search (LS) is used, or NAILM if a trust-region method (LM) is employed instead. The training method, which is also applicable to feedforward neural networks as a special case, is tested in three nonlinear system identification problems. © 2023 Elsevier Ltd
2023
Recurrent neural networks, Nonlinear system identification, Nonlinear least-squares, Generalized Gauss–Newton methods, Levenberg–Marquardt algorithm, Alternating direction method of multipliers, Non-smooth loss functions
File in questo prodotto:
File Dimensione Formato  
1-s2.0-S0005109823003448-main.pdf

non disponibili

Tipologia: Versione Editoriale (PDF)
Licenza: Copyright dell'editore
Dimensione 1.06 MB
Formato Adobe PDF
1.06 MB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/20.500.11771/27880
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 1
social impact