AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-22
Geometric knowledge spaces, Attention pairing and Architectures
The paper proposes a geometric approach to deriving attention mechanisms and exploring potential new transformer architectures. It formalizes the notions of knowledge worlds and attention pairings, then demonstrates that this framework naturally leads to the architectures of standard attention, causal attention, linear attention, tensor attention, and recurrent neural networks. Finally, it introduces several new geometric spaces that may inspire novel neural network architectures.