AI and the World's Hard Problems · Physics · arXiv · 2026-10-22
How Far Can Foundation Models Be Distilled for Efficient Jet Tagging?
The study examines how far foundation models can be distilled into small networks for jet tagging at the Large Hadron Collider, where experiments impose strict limits on inference cost. Six teachers differing in architecture, symmetry, and pretraining, including two foundation models, were distilled into five small students, and every teacher improved every student, with the gain depending mainly on teacher strength. A distilled L-GATr-slim with 20 thousand parameters outperforms every published jet tagger trained from scratch at a fraction of the teachers' inference cost, and with two thousand parameters it matches ParT, a model with a thousand times as many parameters.