AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-20
Post-Selection Distributional Model Evaluation
Comparing candidate models over performance-reliability trade-offs requires reliable estimation of test-time KPI distributions, but using the same data both to pre-select models and estimate their distributions introduces post-selection bias. The authors introduce PS-DME, a framework built on e-values that provides statistically valid distributional model assessment after arbitrary data-dependent pre-selection, controlling the post-selection false coverage rate and provably improving sample efficiency over a sample-splitting baseline under explicit conditions. Experiments on synthetic data, text-to-SQL decoding with large language models, and telecom network performance evaluation show reliable comparison of candidate configurations across a range of reliability levels.