AI and the World's Hard Problems · Life sciences · arXiv · 2026-10-23
Gate-Based Feature Importance in Simulation-Based Inference for EEG: A Cautionary Tale
Simulation-based inference fits mechanistic models to data but offers little insight into which information drives the inferred parameters. The study tested whether learned feature-gate weights reflect feature importance, using neural posterior estimation of four physiological parameters of the Jansen-Rit model and comparing against a retrain-based ablation benchmark. Across nine training seeds, gate-based and ablation-based importance were negatively correlated (Spearman's rho about -0.61 for sigmoid and -0.56 for softmax gating), and total signal power, the most important feature by ablation, was among the most down-weighted by the gates. This calls for caution in reading learned gate weights as causal feature importance.