Intent-Aware Caching Raises LLM Prefix Cache Hit Rates
The paper proposes Intent-Aware Caching (IAC), an online eviction policy for key-value prefix caches in LLM serving systems. It classifies incoming requests by user intent, learns lightweight reuse statistics from recent serving logs, and uses them to guide an oracle-inspired eviction strategy, motivated by the observation that different intents show different continuation probabilities and inter-turn delays. On the WildChat and ShareChat workloads, IAC reaches a 34.6% cache hit rate versus 18.8% for a prior learning-based approach and 12.2% for LRU, while cutting average serving latency by up to 46.1%.
It offers a concrete, measurable technique for improving KV-cache reuse and latency in multi-turn LLM serving, useful to inference infrastructure teams.
Comments & Questions (0)
No comments yet
Be the first to comment!