LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction
LEAP is a length prediction system for LLM serving that reuses prefill hidden states to estimate output length via a linear readout, avoiding costly retraining. It adapts to domain shifts at test time using recursive lea...