AI and the World's Hard Problems · Earth & climate science · arXiv · 2026-10-23
Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings
Studies disagree on whether AI weather models can forecast extreme events, and the authors trace part of this to differing evaluation protocols. In a common experimental design, model rankings depended strongly on the metric: the physics-based model did best on observed extremes, while AI models did better under weighted scoring. A contingency analysis showed AI models underpredict extremes and miss them, while the physics model forecasts them more often and produces false alarms, with the score gaps arising mainly from miscalibration rather than discrimination.