AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-20
Verifying Human-Written Reasoning References
Human-written reasoning chains are often used as references for judging AI reasoning, but the logical support for their steps is rarely checked. The authors use an LLM to translate step conclusions into first-order logic and the Z3 solver to check entailment, applying this to 386 derivations from P-FOLIO. Of these, 202 (52.3%) met all criteria, and 191 of the 230 steps not entailed by their stated support became entailed when checked against the full premises.