AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-26
Generalization properties of the token perceptron: the role of embedding and vocabulary
Token prediction in large language models is a classification problem whose geometry is set by the embedding, so the authors study a minimal linear model called the token perceptron. Using the replica method, they compute its generalization error and find that for continuous embeddings the teacher is recovered exactly above a threshold of d²/(d−1). For vocabulary embeddings, the optimal learner first resolves the directions in which the vocabulary varies most, even when the teacher is isotropic.