A COURSE IN ELEVEN LEVELS
LLM
INFERENCE
Pac-Man is memory-bound. One pellet per move, however fast he runs.
So is your GPU. Generating a single token means reading every weight in the model out of memory — 15 GB for an 8B model — to do about 16 GFLOP of arithmetic with them. On an H100 that is roughly 0.4% of the machine's arithmetic capability; the other 99.6% of the time it is sitting still, waiting on the memory bus.
Batching, quantization, GQA, PagedAttention, FlashAttention, speculative decoding — every technique in this course is an answer to one question: how do we do more useful work per byte moved from memory? Hold that in view and the field stops looking like a pile of tricks.
THE MAZE · PICK A LEVEL
Or play it. Press start and Pac-Man finds a level on his own; take over with the arrow keys, WASD or a swipe to pick one yourself. Whichever numbered node he eats is the level that opens. The ghosts doze until you take control — after that they will send you back to the start, and the power pellets in the bottom corners make them edible. Clicking a node still just opens it.
- PINKYThe big idea — They target four tiles ahead of you.
- PELLETSConcepts — Eaten one at a time.
- INKYMath by hand — Their target is computed, not chased.
- CLYDECode lab — Chases, then retreats to their corner.
- BLINKYPitfalls — They come straight at you.
- POWER PELLETCheckpoint — Eat it and the ghosts turn blue.
The colours are the navigation. Every level uses the same six, so once you know that pink means "the big idea" and red means "here is what people get wrong", you can scan any page without reading it.
LEVEL SELECT
- 00 TRANSFORMER FUNDAMENTALS You cannot reason about inference without a mechanical picture of the forward pass.
- 01 THE AUTOREGRESSIVE LOOP Prefill and decode are not two phases of one job. They are two different jobs.
- 02 THE KV CACHE The single most important object in inference. Everything downstream is memory management.
- 03 DECODING AND SAMPLING The model gives you a distribution. Turning it into a token is a separate design problem.
- 04 METRICS AND THE ROOFLINE The analytical core. Everything after this module is applied roofline reasoning.
- 05 BATCHING The largest throughput lever in serving, and the scheduling problem it creates.
- 06 MEMORY OPTIMIZATION Paging, prefix reuse, architectural KV reduction, and quantization — four attacks on the same denominator.
- 07 ATTENTION KERNELS Attention performance is decided by the memory hierarchy, not the FLOP count.
- 08 SPECULATIVE DECODING Buy back the compute that memory-bound decoding leaves on the floor — and provably change nothing.
- 09 DISTRIBUTED INFERENCE When the model does not fit, the interconnect becomes the memory bus — and it is much slower.
- 10 SYSTEMS AND THE FRONTIER How the engines differ, how to benchmark without lying to yourself, and what is still unsolved.
REFERENCE · ALWAYS OPEN IN A SECOND TAB
- GPU HARDWARE TABLE Bandwidth, dense FLOPs and ridge points for the accelerators you will actually meet. You need this from Level 04 onward.
- 📅 TIMELINE The papers and systems that changed how inference is done, 2017 to now, each tied back to the bandwidth thesis.
- ⚙ SERVING ENGINES vLLM, SGLang, TensorRT-LLM, llama.cpp — what each one optimizes for and what it gives up.
- ? OPEN PROBLEMS What is genuinely unsolved. Read this last, or read it first to know where you are heading.
- A–Z GLOSSARY Every term defined across all eleven levels, in one place.