AI Reports · Amazon · 2026-10-19 · Blog / commentary
Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM
Multi-turn agentic workloads reprocess a growing conversation context on every turn, which increases latency. This post describes bi-directional KV cache transfer, which lets prefill and decode nodes reuse previously computed context across turns. On Amazon EC2 P5en instances with EFA, it cuts time to first token by up to 1.6x for Qwen3-32B and up to 3.3x for GLM-5-FP8.