AI and the World's Hard Problems · Mathematics · arXiv · 2026-10-19
Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization
Lean checks that a generated declaration is well typed but not that it matches the user's intent, so the authors measure how far compilation overstates faithfulness in statement autoformalization. Their criterion requires Lean compilation and agreement of two LLM judges from different model families, and it agrees with the human majority on 91.5% of cases in an independent random audit. On 227 university-level statements across eight systems, the gap between compiling and accepted outputs ranges from 1.3 to 29.5 percentage points, with the tool-augmented agent compiling 87.2% of statements but only 57.7% being accepted.