AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-21
Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms
Continuous-time reinforcement learning seeks an optimal policy under unknown dynamics governed by stochastic differential equations, using observations collected at discrete times. The authors develop a policy gradient framework with rigorous convergence analysis, covering a performance difference lemma and policy gradient formulas for randomized and parametrized policies, exponential convergence of an entropy-regularized policy flow, and a discrete-update bound that explicitly separates critic error, policy-regression error, and learning step size. Integrating the framework with PhiBE, which builds a continuous-time surrogate generator from discrete transition data, yields model-free on-policy and off-policy actor-critic algorithms whose performance and stability are demonstrated on an entropy-regularized linear-quadratic regulator and a nonlinear wheeled robot.