AI and the World's Hard Problems · Life sciences · arXiv · 2026-10-22
Interpretable Embeddings of Speech Enhance and Explain Brain Encoding Performance of Audio Models
Speech foundation models are increasingly hailed as computational models of human speech perception, but their black-box representations make it unclear what drives their alignment with brain responses. The authors built linear encoding models from six interpretable feature families and from embeddings of Whisper, HuBERT, and WavLM, and partitioned the variance shared with electrocorticography (ECoG) responses. The models' brain alignment is mostly explained by their encoding of simple interpretable speech features, with a systematic layer-wise trade-off between low- and high-level features, while brain-relevant semantics not explainable by lower-level features increase with model size and context length.