AI and the World's Hard Problems · Theoretical computer science · Anthropic · 2026-09-28
Superposition, memorization, and double descent
This preliminary study trains toy models that exhibit superposition on limited datasets to investigate overfitting and memorization. The authors find that overfitting corresponds to storing data points, rather than features, in superposition. They present the toy model as a surprisingly rich case study for overfitting.