AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-21
Physics-Inspired Optimizers beyond the Edge of Stability: Period-Two Orbits and Delayed Instability
Because learning rates in deep learning are large, training behaves as a discrete dynamical system in which sharpness typically rises to the optimizer's stability limit, the edge of stability (EoS). The study analyzes kinetic maps, where the step is the gradient of a kinetic energy of the momentum, showing that their symmetric period-two orbits beyond the edge are stable only when the map's elasticity lies between 0 and 1, that instability is delayed as in Neishtadt's theory of slowly swept bifurcations, and that VRAdam's damper fails above twice the critical learning rate while its Legendre dual passes at every learning rate. Experiments on an exact quadratic problem, a full-batch MLP and ResNet-32 support the theory, though on ResNet-18 much of the apparent benefit is reproduced by matching the effective learning rate.