Live intelligence

Latest AI updates

1 updates
Research

Intent-Aware Caching Raises LLM Prefix Cache Hit Rates

The paper proposes Intent-Aware Caching (IAC), an online eviction policy for key-value prefix caches in LLM serving systems. It classifies incoming requests by user intent, learns lightweight reuse statistics from recent serving logs, and uses them to guide an oracle-inspired eviction strategy, motivated by the observation that different intents show different continuation probabilities and inter-turn delays. On the WildChat and ShareChat workloads, IAC reaches a 34.6% cache hit rate versus 18.8% for a prior learning-based approach and 12.2% for LRU, while cutting average serving latency by up to 46.1%.

LLM inference infrastructureKV cache managementCaching policies