AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-26
Sharpness-aware Orthogonalized Momentum Algorithms
Sharpness-aware minimization improves generalization in deep learning, but its extra gradient computation is costly for large language models, and its combination with the Muon optimizer has been little explored. The authors propose SOMA, which uses stale stochastic gradient noise for adversarial perturbations to pair an efficient perturbation scheme with Muon. They show theoretically that SOMA's sample complexity does not depend on the perturbation direction, and they explore its use against quality loss from post-training quantization and catastrophic forgetting during LLM fine-tuning.