METRICS AND THE ROOFLINE

TTFT — time to first token. Queue time plus prefill. Scales with prompt length (roughly linearly, with a quadratic attention term at long context) and with how much other work is ahead of you. This is what determines whether an interface feels responsive.

TPOT / ITL — time per output token. The steady-state gap between tokens once streaming has started. Set by decode, so it is set by bandwidth. Roughly independent of context length, and degrades gently with batch size. Human reading is around 5–8 tokens/second, so anything under ~150 ms/token feels adequate and anything under ~30 ms feels instant.

End-to-end latency. TTFT + TPOT × output_tokens. What actually matters to someone waiting for a complete answer, and dominated by the second term for any substantial output.

Throughput. Total output tokens per second across all concurrent requests. This is what sets your cost per million tokens, and it is the only one of the four that your finance team cares about.

The reason a single number is meaningless is that these move in opposite directions:

batch size  1 ->  256

throughput   ▲▲▲▲▲▲▲▲▲  up ~100x or more
TPOT         ▼          slightly worse
TTFT         ▼▼▼▼       much worse (queueing + prefill contention)

A system advertising "20,000 tokens/second" at batch 256 may be delivering 10 tokens/second to each user. Both numbers are honest; only one of them is relevant to any given question.

Two further points that separate real measurement from marketing:

Percentiles, not means. Latency distributions in serving are heavily right-skewed — a mean TPOT of 30 ms is compatible with a p99 of 400 ms if a few requests get preempted or stuck behind long prefills. Report p50, p95, p99. A mean alone hides exactly the behaviour users complain about.

Goodput, not throughput. If you have an SLO — say, TTFT under 500 ms and TPOT under 50 ms — then tokens delivered outside that SLO are worth nothing. Goodput counts only requests that met their SLO. A system can have excellent throughput and terrible goodput by running at a batch size that violates latency targets for everyone. Optimizing throughput without an SLO constraint is optimizing the wrong thing, and it is a common way for benchmarks to mislead.

Arithmetic intensity is the ratio you can compute from the operation alone, with no knowledge of the hardware:

I = FLOPs performed / bytes moved between DRAM and the chip

"Moved from DRAM" is the important qualifier. Data already resident in cache or registers is free by this accounting; the roofline is about traffic to main memory.

Work the fundamental case. A matmul Y = W @ X with W of shape [n, k] and X of shape [k, B]:

FLOPs  = 2 * n * k * B          (one multiply and one add per element pair)
bytes  = 2 * n * k              (read W once, fp16; X and Y are small by comparison)

I = 2nkB / 2nk = B

The arithmetic intensity of a weight-stationary matmul equals the batch size. That is one of the cleanest and most useful results in the field, and worth stopping on. It says the only way to raise the intensity of a linear layer is to put more sequences through it. Not a better kernel, not a faster GPU — more concurrent work per weight load.

Applying it to the transformer:

Two consequences that people find surprising:

Attention during decode is memory-bound no matter what you do to it. Each cached key element is read and used in a single multiply-accumulate. The intensity is O(1), independent of batch size — because every sequence has its own cache, so batching does not amortize KV reads the way it amortizes weight reads. This is why FlashDecoding exists as a separate kernel from FlashAttention (Module 7), and why KV cache size directly and unavoidably costs you decode time.

The elementwise operations are pure overhead. RMSNorm, residual adds and RoPE do almost no arithmetic per byte. Left unfused, each one is a full round trip to HBM for nothing. In a decode step where you are already bandwidth-starved, they can account for 10–20% of the time. This is the entire justification for kernel fusion.

Two hard limits bound any computation on a machine:

compute ceiling:   P_peak                    FLOP/s
memory ceiling:    I * BW                    FLOP/s  (intensity x bandwidth)

attainable = min(P_peak, I * BW)

Plot attainable performance against intensity on log-log axes and you get a roof: a diagonal line of slope 1 (the bandwidth limit) meeting a horizontal line (the compute limit). The corner where they meet is the ridge point:

I_ridge = P_peak / BW

Below it you are on the slanted roof, memory-bound, and performance rises linearly with intensity. Above it you are on the flat roof, compute-bound, and more intensity buys nothing.

Here are the machines you are likely to care about. Compute figures are dense BF16 (not the sparsity-doubled numbers vendors like to headline):

(Treat these as good working numbers rather than gospel — vendors quote compute under varying assumptions, and you should confirm against the current datasheet for anything load-bearing.)

Read the table for structure, not for individual entries.

The H200 has a lower ridge point than the H100. Same silicon compute, 43% more bandwidth. A lower ridge is better for decode: you need less intensity — a smaller batch — to saturate the machine. The H200 is a straightforwardly better decode part than the H100 while being an identical prefill part. This is what it looks like when a vendor targets the memory wall directly.

The L40S has a ridge point of 419. GDDR6 instead of HBM, and a lot of compute. You would need batch 419 to saturate it, which the 48 GB of memory will not let you reach for any large model. It is a fine prefill and fine-tuning card and a poor decode card, and the ridge point tells you that in one number.

Every ridge point is in the 150–420 range. Modern accelerators have added FLOPs far faster than bandwidth for a decade, so the ridge keeps drifting right and decode keeps getting relatively worse. This trend is why this course exists.

Since decode is memory-bound, its floor follows directly from bandwidth. The recipe:

1.  bytes_per_step = weight_bytes + batch * seq_len * kv_bytes_per_token
2.  TPOT_min       = bytes_per_step / bandwidth
3.  throughput_max = batch / TPOT_min

Worked: Llama-3-8B, fp16, batch 1, 2k context, on an H100.

weights:      8.03e9 x 2       = 16.06 GB   (15.0 GB excluding the embedding lookup)
KV:           131072 x 2048    =  0.27 GB
                                  --------
                                   15.27 GB

TPOT_min = 15.27 / 3350 GB/s   = 4.56 ms
max rate = 1 / 0.00456         = 219 tokens/second

That is the ceiling. No implementation, however good, beats 219 tok/s for this model on this GPU at batch 1. If a vendor claims 400, they have changed something — quantization, speculative decoding, or the definition of a token.

Now the same model at batch 64:

weights:  15.0 GB      (unchanged -- shared)
KV:       131072 x 2048 x 64 = 17.2 GB
                               -------
                               32.2 GB

TPOT_min   = 32.2 / 3350        = 9.6 ms
throughput = 64 / 0.0096        = 6,667 tokens/second

Per-user latency worsened by 2.1× while aggregate throughput improved by 30×. And notice why TPOT got worse: not because of the weights, which are shared, but because the KV cache now outweighs them. This is the two-regime crossover from Module 2 showing up directly in a latency number.

Now account for the gap to reality. Measured TPOT is typically 1.5–3× the roofline floor. The causes, roughly in order:

1. You do not get peak bandwidth. Real kernels achieve 70–85% of the spec figure. That alone is a 1.2–1.4× gap, and it is unavoidable.
2. Kernel launch overhead. Hundreds of tiny kernels per decode step at ~5 µs each is ~1 ms of CPU-side cost against a 4.5 ms budget. CUDA Graphs largely fix this; without them it is brutal.
3. Unfused elementwise ops. Every unfused RMSNorm or residual add is an extra HBM round trip.
4. Attention kernel inefficiency. Decode attention has query length 1, which is a genuinely awkward shape to parallelize.
5. Sampling and detokenization. Sorting 128k logits per sequence is not free at large batch.
6. Python and scheduler overhead. Real in naive implementations, largely engineered away in vLLM/TRT-LLM.
7. Communication, if you are running tensor-parallel (Module 9).

The discipline worth building: compute the roofline number first, measure second, and then account for the gap. If you are within 1.5× your implementation is good. At 5× something specific is wrong and the list above tells you where to look.

Module 2 said the KV cache caps batch size. This module lets you say what that costs in throughput terms.

Llama-3-8B, fp16, on one H100 80GB, 4k context:

weights 16.06 GB, workspace ~4 GB  ->  ~60 GB for KV
KV per sequence: 131072 x 4096 = 0.54 GB
max batch = 60 / 0.54 = 111

Three things to take from this table.

Throughput saturates well before the ridge point. Going from batch 64 to 111 — nearly double the sequences — buys only 14% more throughput. The reason is that KV traffic now dominates, and unlike weight traffic, KV traffic scales with batch. Batching amortizes weights; it does not amortize the cache. Once you are cache-dominated, additional batch adds proportional bytes and buys proportionally little.

You never reach the ridge. At maximum batch you are at 38% of the intensity you would need to saturate the H100's arithmetic. The machine is memory-bound at every operating point available to you. This is the normal condition of LLM serving, not an edge case.

Latency degrades 5× across the range. TPOT goes from 4.6 ms to 22.4 ms. That is the price of the 23× throughput gain, and whether it is acceptable is an SLO question, not a technical one.

Now the important corollary. Since KV traffic is what stops the table from continuing, halving the KV cache moves the whole table. With fp8 KV you get to batch ~200 and past 6,000 tok/s. This is the concrete mechanism by which Module 6's memory work turns into throughput, and it is why "we reduced memory" and "we increased throughput" are frequently the same sentence.

Vendor marketing is optimized for a benchmark you are not running. A short field guide.

The FLOP number is probably doubled by sparsity. NVIDIA quotes structured 2:4 sparsity figures prominently — an H100 is listed at 1,979 TFLOP/s BF16 "with sparsity" and 989 dense. Almost no production LLM uses 2:4 sparsity. Always use the dense number. If a spec sheet does not say which it is, assume sparse and halve it.

FP8 and FP4 numbers are real but conditional. FP8 on Hopper genuinely doubles throughput over BF16, and it is usable — fp8 inference is in production. FP4 on Blackwell likewise. But these apply only to operations you have actually quantized, and they do nothing for a memory-bound decode step except insofar as they shrink the bytes.

Bandwidth is the number that matters for decode, and it is honest. HBM bandwidth figures are close to achievable — you will see 70–85% of spec in a good kernel. Unlike FLOPs, there is no sparsity asterisk.

Memory capacity determines what you can run at all. 80 GB versus 141 GB is the difference between a 70B model fitting on two GPUs or one, which changes the communication picture entirely (Module 9).

Interconnect matters the moment you need more than one GPU. NVLink between H100s is around 900 GB/s per GPU; PCIe Gen5 x16 is about 64 GB/s bidirectional. That is a 14× difference, and tensor parallelism does an all-reduce twice per layer. TP over PCIe is usually a mistake.

A quick decision procedure for a serving workload:

1. Will the weights fit?           -> memory capacity
2. How fast can I decode?          -> bandwidth / weight_bytes
3. How many sequences fit?         -> (capacity - weights) / kv_per_seq
4. Will I ever be compute-bound?   -> compare max batch to the ridge point
5. If multi-GPU, what links?       -> NVLink vs PCIe

Note that step 2 uses only bandwidth and step 4 usually answers "no". For decode-dominated serving, peak FLOPs is close to irrelevant — which is a strange thing to say about a GPU purchase, and precisely the point of this module.