Why this matters
- Without a cache, every new token repeats attention over the whole prompt. A 2,000-token handbook chunk becomes unusably slow on token 50.
- The cache, not the weight file, is what blows GPU memory on long RAG prompts and many concurrent users.
- Interviewers ask for the idea, the tensor shape, and one memory-management strategy.
- Paged attention and eviction are how you say you have operated a server, not only read a diagram.
What is being cached
In a transformer layer, attention compares a query vector for the current token with key vectors of tokens it may look at, then mixes the matching value vectors. During training you can compute queries, keys, and values for a whole sequence in parallel. During autoregressive inference you generate one token at a time. The keys and values of tokens you already accepted do not change.
So you store them. For the next token you compute Q, K, and V for that token only, append K and V to the cache, and attend over the full cached sequence. That is the entire trick. It is exact, not an approximation, as long as you keep the real K and V. It is also why decode is memory-bandwidth heavy: each new token reads a cache that covers every previous token, at every layer.
Shape and cost
For one layer, a common layout is:
K: [batch, num_heads, seq_len, head_dim]
V: [batch, num_heads, seq_len, head_dim]
Some runtimes store [batch, seq_len, num_heads, head_dim]. The numbers are the same. What matters:
How to talk about the size
- There is a K and a V for every layer, not one pair for the whole model.
seq_lengrows by one each decoded token, and it starts already large after prefill of a long prompt.- Memory scales with
batch * layers * seq_len * heads * head_dim * bytes. Long prompts, many concurrent sequences, and a large model multiply. - The cache exists at inference. Training uses a different activation pattern and does not "keep a KV cache" in this serving sense.
- It is why multi-turn chat can be fast on turn two if you kept the cache, and why you must drop that cache when the conversation ends or moves to another machine.
Prefill (reading the prompt) builds the initial cache and is compute-heavy. Decode (emitting tokens) is often limited by how fast you can read that cache from GPU memory. A serving engineer who only talks about FLOPs will mis-diagnose decode.
The rest of this guide is part of Pro
Free covers every fresher guide in full. Pro unlocks Senior, Staff and Architect guides across every track.
Still to come · 3 sections
- The step
- How you manage it
- What you gain, and what you must watch
Test yourself
Answer these before moving on — recall is what makes it stick.