KV Cache

KV cache stores earlier attention key/value representations in autoregressive Transformer decoding, reducing repeated computation. It is computation state, distinct from stored user preferences.

Cache use generally grows with sequence length and concurrent work. PagedAttention examines dynamic allocation, fragmentation and sharing in LLM serving.^[1] Actual memory also depends on layers, attention structure, precision and engine implementation. Attention structure and cache allocation should be checked before translating a model size into capacity.

Measure memory, first-token delay, completed work and failures under specified lengths and concurrency. A cache optimization alone does not establish factual accuracy or source freshness.

This is hypothetical arithmetic, not a hardware specification or recommendation. A KV budget of 3 GiB after weights and other runtime memory, with an assumed maximum of 384 MiB per request, gives 3×1,024/384 = eight requests. At 768 MiB for longer requests, the simple bound is four. Allocation, sharing, reserves and unequal input lengths prevent treating these figures as guaranteed serving capacity.

Record weights and activations separately from cache, and identify how input and output lengths affect peak use. The PagedAttention study ↗ addresses dynamic-cache waste and sharing.^[1] An efficiency result does not prescribe one batch size for every device.

Reusing earlier token calculations within one request and sharing a matching prefix across requests have different scopes. Check engine conditions when models, adapters, tokens or input configuration change. Similar strings alone do not establish compatible computation state.

For a policy revision from v1 to v2, separately check correct computation for the new input and authorization to supply that policy. Source versions and user state are not replaced by one cache key. A cache hit proves neither the current policy answer nor access permission.

For exhaustion, inspect peak cache, concurrency and allocation. For low hit rates, inspect prefix changes, reuse conditions and lifetime. For delayed first output, inspect queueing, prefill and retrieval. Reducing cache does not repair retrieval, and increasing cache does not automatically shorten a queue.

Use vLLM's operational metrics ↗ to inspect the engine's measurement boundaries.^[2] Compare latency, throughput and failures at the same lengths, concurrency and quality. When changing cache precision, reevaluate task accuracy as well as memory use.