AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-21
PhiBE-Q-Learning: Bridging Off-Policy Reinforcement Learning and Continuous-Time Control
The paper develops an off-policy method for continuous-time reinforcement learning, where the system dynamics follow an unknown SDE and only discrete-time trajectory data are available. To address the loss of action-dependent signal in the classical discrete-time state-action value function in the continuous-time limit, the authors introduce a new definition of the state-action value function Q(s,a), derive its governing equation, and propose iterative algorithms based on the PhiBE approximation in both model-based and model-free settings. Under linear function approximation they establish error bounds and explicit convergence rates at the population level, and evaluate the finite-data algorithm numerically.