AI and the World's Hard Problems · Theoretical computer science · Anthropic · 2026-09-28
Privileged bases in the transformer residual stream
The study examines why individual coordinates in the Transformer residual stream show privileged basis behavior, even though theory predicts they should be arbitrary. The authors provisionally attribute the effect to the per-dimension normalizers in the Adam optimizer. Layer normalization and finite-precision floating-point arithmetic were ruled out as sources.