Back to AI News
arxiv

KV Cache Reuse Across Model Updates Preserves Task Utility but Diverges From Native Behavior

This arXiv paper studies whether key-value caches computed by one model version can be reused after weights are updated, examining four model updates with a detailed case study of DeepSeek-V4-Flash to Flash-0731. The authors find caches can retain useful task information without faithfully reproducing the updated model's behavior, with retrieval accuracy ranging from near-native performance to large losses. Where source caches already exist, reuse yields median time-to-first-token speedups of 2.10x to 5.08x on three longer-prefix QA benchmarks. They introduce Functional Cache Divergence, comparing next-token distributions under reused and native caches, and show compatibility is better for natural-text continuation than for single-token retrieval as prefix length grows.

AI Brief

It gives practitioners a concrete, measured view of when cross-version KV cache reuse saves compute and when it silently degrades output fidelity.

arXiv:2611.14051v1 Announce Type: cross Abstract: Reusing key--value (KV) caches across language model updates could avoid repeated computation of shared context, but changes in model weights raise a fundamental question about what these caches preserve. We investigate direct cache reuse across four model updates, with a detailed study of DeepSeek-V4-Flash to Flash-0731. Our central finding is that caches can retain useful task information without faithfully reproducing the updated model's behavior. Retrieval accuracy varies substantially across updates, ranging from near-native performance to large losses. With source caches already available, reuse achieves median time-to-first-token speedups of $2.10$--$5.08\times$ on three longer-prefix QA benchmarks. To characterize the behavioral differences behind these outcomes, we introduce Functional Cache Divergence, which compares next-token distributions under reused and native caches. Direct KV cache reuse better preserves native model behavior in natural-text continuation, whereas compatibility degrades with increasing prefix length in single-token retrieval tasks. Answer-margin analyses help explain how correct retrieval decisions survive substantial internal state changes. Yet even agreement on the first generated token does not ensure agreement in answer content or stopping behavior. These findings establish task utility and fidelity to native inference as distinct criteria for evaluating cross-version KV cache reuse.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!