AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-23
Analysis of On-policy Policy Gradient Methods under the Distribution Mismatch
Many state-of-the-art policy gradient methods estimate update directions using a sampling distribution that differs from the discounted state distribution the policy gradient theorem requires. The study analyzes how this distribution mismatch affects these methods and shows global convergence of objective values under biased updates for tabular direct and softmax parameterizations. It also gives an explicit criterion for whether the mismatch speeds up or slows asymptotic convergence, and extends the bounds to more general policy parameterizations.