ORBiS TREE · ENGLISH DOCUMENT
Concepts →KV Cache
A cache of attention keys and values used to avoid recomputing earlier context during autoregressive generation.
한국어
English
This English page is a curated translation layer linked to the Korean source article. Community changes are currently made on the Korean source, where the full revision history and anonymous edit trail are preserved.
KV cache is an inference optimization used by Transformer decoders. During token-by-token generation, the model stores attention key and value tensors from prior tokens so they do not have to be recomputed at every decoding step.
How it works
Caching reduces repeated computation and is critical for fast generation, but the cache consumes memory that grows with sequence length, batch size, layer count, and representation size. Systems therefore trade memory against throughput and context length.
Why it matters
KV-cache behavior is a major factor in production LLM serving. Techniques such as paged memory management, quantized caches, and cache reuse can improve efficiency.
Related concepts
SOURCES
Sources
KNOWLEDGE LINKS
Continue from here
BACKLINKS