AI and the World's Hard Problems · Medicine & drug discovery · Nature (computational biology) · 2026-10-16
A multi-stage validation framework for trustworthy large-scale clinical information extraction using large language models
Validating large language models for clinical information extraction usually requires costly manual annotation. The authors proposed a multi-stage framework combining rule-based filtering, semantic grounding, judge-LLM evaluation, expert review, and predictive validity analysis. Applied to substance use disorder diagnoses across 919,783 clinical notes, it removed 14.59% of LLM-positive extractions as unsupported or implausible. The primary LLM achieved an F1 score of 0.80 against judge evaluations, and its extracted diagnoses predicted later specialty care engagement better than structured-data baselines.