AI and the World's Hard Problems · Theoretical computer science · Anthropic · 2026-09-28
Scaling laws of learning from repeated data
The study examines how repeated training data affects language models. Repeating just 0.1% of the training data 100 times reduced an 800M-parameter model to the performance of a 400M-parameter model, even though the other 90% of tokens remained unique. The authors also found that repetition damages copying and induction heads, which are linked to generalization, and suggest this may explain the shift from generalization to memorization.