THE KV CACHE

Build the formula up rather than memorising it. For one token, at one layer, in one attention head, you store:

• a key vector of head_dim numbers,
• a value vector of head_dim numbers.

That is 2 × head_dim numbers. The leading 2 is K and V — nothing more mysterious than that.

Now multiply out the dimensions you have:

x kv_heads      each KV head has its own K and V (this is n_kv_heads, NOT n_heads)
x layers        every layer attends independently and keeps its own cache
x seq_len       one entry per position, and it only ever grows
x batch         no sharing between sequences
x dtype_bytes   2 for fp16/bf16, 1 for fp8/int8

Giving:

kv_bytes = 2 * n_layers * n_kv_heads * head_dim * seq_len * batch * dtype_bytes

The term people get wrong is n_kv_heads. On a GQA model this is much smaller than n_heads, and using the wrong one inflates your estimate by the GQA ratio — 4× on Llama-3-8B, 8× on Llama-3-70B. Check the config, not the head count.

The most useful form is bytes per token per sequence, because it strips out the two things that vary at runtime:

bytes_per_token = 2 * n_layers * n_kv_heads * head_dim * dtype_bytes

For Llama-3-8B at fp16: 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token. For Llama-3-70B at fp16: 2 × 80 × 8 × 128 × 2 = 327,680 bytes = 320 KiB per token.

Commit those two numbers. From them everything else is one multiplication: an 8k-context 70B sequence is 320 KiB × 8192 = 2.5 GiB, and thirty-two of them is 80 GiB — an entire H100, holding nothing but cache.

Note what is not in the formula: n_heads, d_model, and vocab_size are all absent. The KV cache is governed by the KV side of attention alone, which is precisely why architectural attacks on it (MQA, GQA, MLA in Module 6) are so effective — they change one term in a product.

Weights and KV cache behave completely differently as you scale, and confusing them is the most common planning error in LLM serving.

Weights are fixed and shared. A 70B model at fp16 is 140 GB whether you serve one request or a thousand. Load once, use for everyone.

KV cache is per-sequence and grows. Every admitted request adds its own, and every token generated grows it.

Put real numbers on it. Serving Llama-3-70B on 4× H100 80GB (320 GB total):

total memory                        320 GB
- weights (fp16)                    140 GB
- activations, workspace, frag       ~20 GB
                                    -------
available for KV cache              ~160 GB

At 320 KiB per token, that budget buys you 160e9 / 327680 ≈ 488,000 tokens of cache — total, across all sequences. How that divides up depends entirely on context length:

Read the last row. On a $120,000 machine, a 70B model at 128k context can hold three concurrent requests. Not three hundred. Three.

Now connect it to Module 1. Decode arithmetic intensity equals batch size, and an H100's ridge point is 295 FLOP/byte. At 8k context you can reach batch 59 — roughly 20% of the way to the ridge, so you are still 5× off saturating the machine's arithmetic. At 128k context you reach batch 3, which is 1% of the ridge. The KV cache is what stands between you and the throughput the hardware is capable of.

This is why memory optimization (Module 6) is not a side quest. Halving the KV cache does not just save memory; it doubles the batch size, which doubles arithmetic intensity, which approximately doubles throughput. Memory work is throughput work.

Run the formula with the KV-head count as the only variable, on Llama-3-70B (n_layers=80, head_dim=128, fp16, 8k context, batch 32):

The MHA row is the one to sit with. A hypothetical multi-head Llama-3-70B would need 640 GiB of KV cache to serve 32 sequences at 8k — eight H100s of pure cache, on top of the two you need for weights. Ten GPUs to serve 32 users. That configuration is not expensive, it is unshippable.

GQA-8 turns it into 80 GiB, which fits alongside the weights on a 4-GPU node with room to spare. Grouped-query attention is not a minor efficiency tweak. It is the architectural decision that made long-context serving economically possible, and it is why essentially every model released since 2023 uses it.

The trade is quality, and the honest framing is that it is small but real. Ainslie et al. found GQA lands close to MHA quality while training much faster than converting to MQA, and MQA — sharing a single KV head across all queries — degrades noticeably more. The sweet spot the field converged on is 4 to 8 KV heads, and it was found empirically rather than derived.

There is a second, subtler benefit that matters in Module 9. Tensor parallelism splits attention heads across GPUs, and you cannot split a KV head across two devices without replicating it. With 8 KV heads you can run TP-8 cleanly, one KV head per GPU. Beyond TP-8 you must duplicate KV heads across devices, and the cache stops shrinking as you add GPUs. The KV head count silently sets a ceiling on how far you can usefully parallelize.

The arithmetic above assumes you use every byte you allocate. Pre-vLLM systems did not come close.

The problem is that you do not know how long a sequence will be until it finishes. A request arrives, you must give it cache memory, and its final length is unknown. The obvious safe answer is to allocate for the worst case: reserve max_seq_len up front, contiguously.

If max_seq_len is 8,192 and the request generates 200 tokens, you reserved 2.5 GiB and used 61 MiB. 97.6% waste. And because you took a contiguous block, the unused portion cannot be lent to anyone else.

The vLLM paper breaks the waste into three kinds, and the distinction is worth keeping:

• Internal fragmentation — space reserved inside a sequence's own allocation that it never fills. The dominant term, and the one above.
• External fragmentation — free memory that exists but is split into chunks too small to satisfy a contiguous request. The classic malloc problem.
• Reservation waste — memory held for future tokens of a sequence that is currently running. Legitimately needed eventually, but idle right now.

Their measurement is the memorable one: in existing systems, only 20.4% to 38.2% of allocated KV cache memory held actual token state. Between 60% and 80% was waste. Meaning that a system reporting "GPU memory full, cannot admit requests" was, most of the time, two-thirds empty.

There is a subtler cost too. Because you must reserve for the worst case, max_seq_len becomes a global tax. Raising your advertised context window from 8k to 32k shrinks your maximum batch size by 4× even if no request ever uses more than 2k, because every request still reserves 32k. Operators discovered this the hard way.

PagedAttention (Module 6) fixes this by borrowing the idea operating systems settled on in the 1960s: allocate in small fixed-size blocks, keep a per-sequence block table, and let physical memory be non-contiguous. Waste drops below 4%. It is a genuinely lovely result — a hard systems problem solved by noticing it was a solved problem in a different field.

A decode step reads two things: all the weights, and all the KV cache of every sequence in the batch. Which dominates determines what you should be optimizing.

bytes_per_step = weight_bytes  +  batch * seq_len * kv_bytes_per_token
                 (constant)       (grows with load and context)

Llama-3-8B at fp16: weights ≈ 15.0 GB, KV ≈ 128 KiB/token. Setting the terms equal:

batch * seq_len = 15.0e9 / 131072 = 114,441 tokens

So the crossover is at about 114,000 cached tokens in flight. Below that you are weight-dominated; above it, cache-dominated.

This table should change how you read optimization advice, because the two regimes want opposite things:

Weight-dominated (small batch, short context — interactive single-user, local inference). Quantize the weights. int4 weight-only quantization nearly quadruples decode speed here because weight bytes are your step time. KV optimizations do almost nothing. This is why llama.cpp is obsessed with weight quantization: it is the correct obsession for its regime.

Cache-dominated (large batch, long context — production serving). Weight quantization helps much less. You want KV quantization, GQA/MLA, PagedAttention, prefix caching, eviction. This is why vLLM and SGLang are built the way they are.

Most arguments about "the best inference optimization" are really two people in different regimes talking past each other. Ask which side of the crossover they are on and the disagreement usually dissolves.

Consider a chat application. Turn 3 of a conversation sends:

[system prompt: 500 tokens]
[user turn 1] [assistant turn 1]
[user turn 2] [assistant turn 2]
[user turn 3]                        <- only this is new

A stateless server prefills all of it — say 3,000 tokens — to produce one new token. But the KV entries for the first 2,900 tokens are identical to what it computed last turn. Causality guarantees it: appending tokens cannot change the K or V of anything earlier.

If you keep that cache, turn 3's prefill is 100 tokens instead of 3,000. A 30× reduction in TTFT, for free, with no approximation.

The same idea applies across different requests that share a prefix. A thousand users hitting the same 800-token system prompt share those 800 tokens' worth of KV state exactly. Compute it once, point every sequence at it. This is prefix caching, and with a radix tree to find the longest matching prefix (SGLang's RadixAttention) it generalises to arbitrary sharing patterns — few-shot examples, document QA over a shared document, agent loops that replay a growing trajectory.

Agentic workloads make this decisive rather than merely nice. An agent that takes 20 steps re-sends its entire growing history every step. Without prefix caching, total prefill work is quadratic in the number of steps. With it, each step prefills only the newest action and observation.

Three things make this harder than it sounds, and all three are real bugs people ship:

1. RoPE position indices. Cached keys were rotated by their absolute position. Reuse a prefix at a different offset without handling this and you get silently wrong attention.
2. Eviction policy. Cache memory is the same memory sequences need. Keeping prefixes for possible future hits costs you batch size now. LRU is the usual answer; the right answer depends on your traffic.
3. Isolation. Sharing cache across users is sharing derived state across users. The state is keyed by exact token prefix, so a hit requires an exact match — but it is worth knowing this is a cross-tenant surface, and that hit/miss timing can in principle leak whether someone else has sent a given prefix.