Push-based KV transfer in vLLM cuts TTFT up to 3x for disaggregated inference
AWS describes a push-based NIXL KV connector for vLLM's prefill-decode disaggregated serving, contributed upstream as an accelerator-agnostic feature. Instead of the decode instance pulling KV blocks after prefill,...