AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-21
Confessions of a Masked MDP
In some reinforcement learning applications, determining which state-action pairs are invalid—the action mask—is the main computational bottleneck, even though the action set is usually presumed known. The authors study combining the search for an optimal policy with learning the mask itself under a linear MDP in which each attempted action reveals the exact value of a linear score determining its validity. A deterministic optimistic optimization algorithm proves minimax optimal and computationally efficient but sub-optimal on non-degenerate instances, while an alternative that plans its exploration, interpretable as an analogue of information-directed sampling for deterministic feedback, is also efficient, with an application to transfer learning where the invalid action set changes over time.