AI and the World's Hard Problems · Life sciences · arXiv · 2026-10-19
Brain-Aligned Multimodal LLMs for Social and Emotional Understanding of Cartoon Movies
Multimodal large language models can annotate social and emotional content at scale, but their agreement with human judgments and their value for predicting brain responses needed testing. The authors compared diffusion-based and autoregressive models with continuous human ratings of Theory of Mind and affective valence across three short cartoon movies. The model ratings captured aspects of human temporal variation, with stronger alignment for valence than for Theory of Mind, while in one movie more brain regions were significantly encoded by model ratings than by human ratings, and overlap between the two was limited.