AI and the World's Hard Problems · Mathematics · arXiv · 2026-10-23
Beyond LLM Judge: Mechanical Verification for Agentic Autoformalization
This paper presents a method for verifying Lean formalizations that goes beyond LLM-judged faithfulness by combining LLM-based semantic review with mechanical numerical and vacuity checks. Applied with an extract-verify-proof autoformalization pipeline, it was used to build FLMF, a partial formalization of the NIST Digital Library of Mathematical Functions. On three labeled datasets, the method had the lowest accepted-error rate among the methods evaluated, and expert review found translation and source defects in existing datasets.