AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-21
A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
The study isolates a mechanism by which the mismatch between smooth surrogate losses such as softmax cross-entropy and discrete hard labels produces power-law learning curves, using a one-layer softmax student trained by online SGD on Gaussian inputs with argmax teacher labels. At late times only shrinking layers of examples around teacher decision boundaries remain active, yielding an α^{-1/3} law in training time for both the test loss and the generalization error. The error law is an online-noise effect, and learning-rate annealing improves its exponent from 1/3 toward 1/2 while slowing the loss, which persists even without gradient noise.