Temporal-Difference Methods for Learning State Estimators and Smoothers

This thesis addresses the problem of learning recursive state estimators and smoothers from sequences of measured system outputs, for linear and nonlinear discrete-time stochastic systems with additive noise. The process and measurement models are assumed known, while the corresponding ground-truth state trajectories are not available. In the linear setting, dynamic programming is used to decompose the discounted optimal estimation problem into a sequence of subproblems indexed by time, leading to a value function satisfying a Bellman equation tailored to state estimation. Under standard assumptions, this value function is shown to be stationary and quadratic, and is then learned via temporal difference learning. The convergence of the resulting learning scheme to the optimal observer is confirmed numerically. The thesis then turns to the undiscounted nonlinear setting, where the central contribution is the introduction of the λ- function, which is defined as the gradient of the optimal value function and provides a unified characterization of prediction, filtering, and smoothing. The value function and the associated λ-function are shown to be locally well defined. Whereas the value function satisfies a Bellman-type recursion, the λ-function satisfies a novel forward recursion that relates the first-order optimality conditions at consecutive time steps, with past measurement information entering implicitly through these conditions. This forward recursion provides the basis for a temporal difference approach to learning the λ- function directly, without explicitly approximating the value function or bootstrapping on its estimates. xvi An ideal learning problem is then formulated from sequences of measured system outputs, and gradient descent applied to its reduced formulation is shown to converge to a critical point under suitable smoothness, regularity, and Kurdyka– Łojasiewicz assumptions. The convergence analysis is developed under local assumptions. Since the ideal formulation is computationally demanding, the λ-learning algorithm is introduced as a practical heuristic, although the convergence result does not directly apply to this algorithm. Numerical results demonstrate a favorable trade-off between accuracy, empirical consistency diagnostics, and computational cost compared with selected Bayesian and optimization-based estimation and smoothing methods. xvii