AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-22
A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees
The paper considers continuous-time reinforcement learning where the state follows an unknown stochastic differential equation but only a fixed off-policy batch of discrete-time transitions is available. Building on the PhiBE-Q formulation, it develops a model-free PhiBE policy iteration (PhiBE-PI) framework that learns a single state-action critic directly from sampled increments, improving the policy while reusing the same batch throughout and without explicitly estimating drift or diffusion. Under linear function approximation, the authors establish a local end-to-end suboptimality guarantee quantifying how discretization, approximation, and finite-sample errors propagate, and identify a bias-variance trade-off whereby finer time discretization does not necessarily improve policy performance.