Back to AI News
arxiv

Intent-Aware Caching Raises LLM Prefix Cache Hit Rates

The paper proposes Intent-Aware Caching (IAC), an online eviction policy for key-value prefix caches in LLM serving systems. It classifies incoming requests by user intent, learns lightweight reuse statistics from recent serving logs, and uses them to guide an oracle-inspired eviction strategy, motivated by the observation that different intents show different continuation probabilities and inter-turn delays. On the WildChat and ShareChat workloads, IAC reaches a 34.6% cache hit rate versus 18.8% for a prior learning-based approach and 12.2% for LRU, while cutting average serving latency by up to 46.1%.

AI Brief

It offers a concrete, measurable technique for improving KV-cache reuse and latency in multi-turn LLM serving, useful to inference infrastructure teams.

arXiv:2611.09715v1 Announce Type: new Abstract: Modern LLM serving systems use prefix caching to accelerate multi-turn conversational workloads. They cache the key-value states of previous conversation prefixes and reuse them when subsequent requests share the same prefix, to avoid redundant prefill computation and reduce serving latency. However, achieving a high cache hit rate remains challenging because cache reuse depends on whether a conversation continues and how soon the next turn arrives. Our analysis of real workloads shows that user intent is a useful signal for this behavior, as different intents exhibit different continuation probabilities and inter-turn delays. Based on this key insight, we propose Intent-Aware Caching (IAC), an online eviction policy that classifies requests by intent, learns lightweight reuse statistics from recent serving logs and leverages these statistics to guide an oracle-inspired eviction strategy. Extensive evaluation on real workloads (WildChat and ShareChat) shows that IAC reaches a 34.6% cache hit rate. This substantially outperforms both the previous learning-based approach (18.8%) and traditional methods such as least recently used (LRU, 12.2%), while reducing average serving latency by up to 46.1%.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!