AI Reports · Stanford Digital Economy Lab · 2026-10-29 · Research paper
Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems? - Stanford Digital Economy Lab
Real-time interaction requires agents to perceive changing environments and act under time constraints, yet existing evaluations focus on conversational quality and streaming perception rather than effective action. The introduced Realtime-Bench uses dynamic tasks where delayed actions directly affect success; under wall-clock time, the best success rate is 44% and the best Pass^3 is 36%. Comparing against a paused-clock reference and a token-based simulated clock, the authors find that temporal costs limit success and that higher reasoning effort has mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution.