AI Reports · Amazon · 2026-10-19 · Blog / commentary
Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
vLLM's prefill-decode disaggregation requires transferring the KV cache between engine instances, which affects time to first token. This post introduces a push-based NIXL connector that moves transfer coordination off the critical path. In benchmarks on Amazon EC2 P5en instances with AWS EFA, it delivers up to 3x better TTFT and 30% better TPOT.