AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-26
Convergence of Large-Stepsize SGD for the Cross-entropy Loss via Stochastic Lyapunov Stability
Deep learning often uses learning rates far larger than classical optimization theory justifies, yet the stochastic setting has been largely unexplored. The authors give sharp convergence guarantees for SGD on multiclass cross-entropy loss for linear classifiers and two-layer neural networks, showing that the dynamics can alternate between an edge-of-stability regime dominated by curvature-driven oscillations and a stable regime where the expected loss decreases at a controlled rate. They prove that SGD returns to stability within a fixed number of iterations, allowing best-iterate convergence even with large learning rates, and experiments validate these findings.