arXiv:2611.05306v1 Announce Type: new
Abstract: Later tokens are costlier to generate than earlier tokens. On frontier open weight models, the disparity in cost may rise to as much as 7x. Through frequent compaction, we show across a broad range of long-horizon and long-input tasks that frontier language models sacrifice little to no task accuracy when maintaining short contexts. We further demonstrate that LLM performance does not collapse on long-horizon software engineering tasks even when taking at most one action per context, and based on that we outline the concept of semantic state space models (SSSMs). Our findings question the need for long-context language models.
arxiv
You don't need long context
The paper shows that frontier language models can maintain task accuracy on long-horizon and long-input tasks by frequently compacting context, despite later tokens being up to 7x costlier. Even with at most one action per context, performance on software engineering tasks does not collapse, leading to the proposal of semantic state space models (SSSMs). This challenges the necessity of long-context models.
Provides concrete evidence and a new model concept that could shift AI architecture and cost efficiency discussions.
Comments & Questions (0)
No comments yet
Be the first to comment!