AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-19
Mesoscopic learning dynamics of extensive-width neural networks
This study examines how stochastic gradient descent learns internal representations in two-layer neural networks whose width and input dimension grow proportionally. A Gaussian approximation of the distribution of feature overlaps reduces early learning to a few collective variables tracking the linear and quadratic components of the target. The authors report that learning these components filters gradient fluctuations and accelerates specialization of individual features.