Study Derives Fundamental Limits of Prefill-Based Decode-Length Prediction
A new arXiv paper argues that existing benchmarks for predicting how many tokens an LLM will decode from prefill-phase hidden states conflate predictor quality with task difficulty, causing reported accuracy to collapse...