LLM INFERENCE SCORE000000 CLEARED00/15

FIFTEEN CHAPTERS

LLM
INFERENCE

Pac-Man is memory-bound. One pellet per move, however fast he runs.

So is your GPU. Generating a single token means reading every weight in the model out of memory — 15 GB for an 8B model — to do about 16 GFLOP of arithmetic with them. On an H100 that is roughly 0.4% of the machine's arithmetic capability; the other 99.6% of the time it is sitting still, waiting on the memory bus.

Batching, quantization, GQA, PagedAttention, FlashAttention, speculative decoding — every technique in these chapters is an answer to one question: how do we do more useful work per byte moved from memory? Hold that in view and the field stops looking like a pile of tricks.

START HOW TO WORK THROUGH IT

THE MAZE · PICK A LEVEL

Module 0 — TRANSFORMER FUNDAMENTALS0Module 1 — THE AUTOREGRESSIVE LOOP1Module 2 — THE KV CACHE2Module 3 — DECODING AND SAMPLING3Module 4 — METRICS AND THE ROOFLINE4Module 5 — BATCHING5Module 6 — MEMORY OPTIMIZATION6Module 7 — ATTENTION KERNELS7Module 8 — SPECULATIVE DECODING8Module 9 — DISTRIBUTED INFERENCE9Module 10 — SYSTEMS AND THE FRONTIER10Module 11 — MODEL ARCHITECTURES11Module 12 — HOW VLLM WORKS12Module 13 — GPUS AND TPUS13Module 14 — FINE-TUNING14Module 15 — CROSS-MODEL KV CACHE TRANSFER15
HOVER A NODE TO SEE THE LEVEL

Or play it. Press start — or just tap left or right while the maze is on screen — and Pac-Man finds a level on his own; take over with the arrow keys, WASD or a swipe to pick one yourself. Whichever numbered node he eats is the level that opens. The ghosts doze until you take control — after that they will send you back to the start, and the power pellets in the bottom corners make them edible. Clicking a node still just opens it.

  • PINKYThe big idea — They target four tiles ahead of you.
  • PELLETSConcepts — Eaten one at a time.
  • INKYMath by hand — Their target is computed, not chased.
  • CLYDECode lab — Chases, then retreats to their corner.
  • BLINKYPitfalls — They come straight at you.
  • POWER PELLETCheckpoint — Eat it and the ghosts turn blue.

The colours are the navigation. Every level uses the same six, so once you know that pink means "the big idea" and red means "here is what people get wrong", you can scan any page without reading it.

LEVEL SELECT

  1. 00 TRANSFORMER FUNDAMENTALS You cannot reason about inference without a mechanical picture of the forward pass. 8–12 hours
    8 concepts
  2. 01 THE AUTOREGRESSIVE LOOP Prefill and decode are not two phases of one job. They are two different jobs. 4–6 hours
    6 concepts
  3. 02 THE KV CACHE The single most important object in inference. Everything downstream is memory management. 6–8 hours
    6 concepts
  4. 03 DECODING AND SAMPLING The model gives you a distribution. Turning it into a token is a separate design problem. 5–7 hours
    7 concepts
  5. 04 METRICS AND THE ROOFLINE The analytical core. Everything after this module is applied roofline reasoning. 8–10 hours
    6 concepts
  6. 05 BATCHING The largest throughput lever in serving, and the scheduling problem it creates. 5–7 hours
    6 concepts
  7. 06 MEMORY OPTIMIZATION Paging, prefix reuse, architectural KV reduction, and quantization — four attacks on the same denominator. 8–10 hours
    6 concepts
  8. 07 ATTENTION KERNELS Attention performance is decided by the memory hierarchy, not the FLOP count. 7–9 hours
    6 concepts
  9. 08 SPECULATIVE DECODING Buy back the compute that memory-bound decoding leaves on the floor — and provably change nothing. 6–8 hours
    5 concepts
  10. 09 DISTRIBUTED INFERENCE When the model does not fit, the interconnect becomes the memory bus — and it is much slower. 8–10 hours
    7 concepts
  11. 10 SYSTEMS AND THE FRONTIER How the engines differ, how to benchmark without lying to yourself, and what is still unsolved. 6–8 hours
    6 concepts
  12. 11 MODEL ARCHITECTURES Every architecture is a different answer to one question: which bytes have to move to produce the next token? 7–9 hours
    8 concepts
  13. 12 HOW VLLM WORKS The serving-engines reference gave you the philosophy. This level is the mechanism: the block table, the scheduler loop, and the arithmetic behind preemption. 6–8 hours
    6 concepts
  14. 13 GPUS AND TPUS Two chips built on opposite philosophies, bolted to the same kind of memory. What a systolic array buys, what a warp scheduler buys, and why decode barely notices either. 6–8 hours
    6 concepts
  15. 14 FINE-TUNING Pretraining teaches a model to complete anything. Fine-tuning teaches it to do what you actually asked — and the newest alignment methods do that by running the exact decode loop the rest of this book spent optimizing. 7–9 hours
    7 concepts
  16. 15 CROSS-MODEL KV CACHE TRANSFER Every earlier level treated the KV cache as private to the model that built it. This is the weeks-old research arguing that was only ever a convenience — and the arithmetic for what it costs to relax it. 6–8 hours
    7 concepts

REFERENCE · ALWAYS OPEN IN A SECOND TAB

  1. GPU HARDWARE TABLE Bandwidth, dense FLOPs and ridge points for the accelerators you will actually meet. You need this from Level 04 onward. reference
  2. 📅 TIMELINE The papers and systems that changed how inference is done, 2017 to now, each tied back to the bandwidth thesis. reference
  3. ⚙ SERVING ENGINES vLLM, SGLang, TensorRT-LLM, llama.cpp — what each one optimizes for and what it gives up. reference
  4. ? OPEN PROBLEMS What is genuinely unsolved. Read this last, or read it first to know where you are heading. reference
  5. A–Z GLOSSARY Every term defined across all sixteen levels, in one place. reference