SYSTEMS AND THE FRONTIER

Start with the convergence, because it is the more informative fact. Every serious engine now has: continuous batching, paged KV cache, prefix caching, chunked prefill, CUDA graphs, fused kernels, FlashAttention-family kernels, weight quantization, increasingly KV quantization, and speculative decoding.

Those are table stakes, not differentiators. Four independently designed systems converging on the same list is good evidence the list is correct — and it means a feature-comparison table tells you almost nothing.

What differs is the organizing bet.

vLLM — memory management as the foundation. The thesis is that if you fix KV fragmentation, everything else follows: recovered memory becomes batch size, which becomes arithmetic intensity, which becomes throughput. PagedAttention drove the whole architecture, and continuous batching depends on it. The V1 engine rewrite moved scheduling out of the Python hot path, which was the main remaining overhead. What it gives up: peak single-request latency, and some raw kernel performance relative to a compiled engine, because it optimizes for generality across a very large model zoo.

SGLang — the workload is structured, not a stream of independent requests. Real traffic shares system prompts, few-shot examples, documents, and conversation history. RadixAttention makes prefix reuse the default rather than a special case, and the frontend language lets you express branching, parallel calls and constrained generation as a program. Structured output is first-class, not a wrapper. What it gives up: simplicity, and some generality — the benefits concentrate in workloads that genuinely share prefixes.

TensorRT-LLM — decide everything at build time. The bet is that the last 20–30% lives in kernel selection, fusion and layout choices that are only correct for a specific model on specific hardware at a specific batch size, so make them ahead of time with full knowledge of the deployment. Typically the fastest option on NVIDIA hardware with the tightest tail latency. What it gives up: a great deal of flexibility. The engine is tied to the GPU it was built for; changing max batch size or context length means recompiling; new architectures land later.

llama.cpp — the weight-dominated regime, taken seriously. One user, short context, not enough memory. That is Module 2's weight-dominated regime, where weight bytes are essentially the whole decode cost — so aggressive weight quantization is not one optimization among many, it is the optimization. Hence the k-quant family, CPU inference, partial GPU offload, and a zero-dependency build. What it gives up: throughput at scale. Serving hundreds of concurrent users is not the design centre.

The pattern worth extracting: each engine is right about a different operating point. Most arguments about which is "best" are two people in different regimes talking past each other — the same failure mode as the weight-versus-KV-quantization argument in Module 6. Ask what batch size and context length someone is running before taking their recommendation.

Error 1: closed-loop load generation. The most common and most damaging. A closed-loop harness keeps N requests in flight, sending a new one only when an old one finishes. That is not how users behave, and it has a specific pathology: it cannot show you overload. If the system slows down, the harness sends fewer requests, so the queue never grows and latency degrades gracefully — hiding exactly the behaviour you needed to see.

Real traffic is open-loop: requests arrive on their own schedule, typically Poisson, whether or not you are ready. Under open-loop load a system past its capacity shows queue growth and latency blowup, which is the honest picture. Benchmark open-loop, sweep the arrival rate, and find where it breaks.

Error 2: means instead of percentiles. Serving latency distributions are heavily right-skewed. A mean TPOT of 30 ms is entirely compatible with a p99 of 400 ms — preempted requests, long prefills, unlucky scheduling. Users experience the tail. Report p50, p95, p99, and report them for TTFT and TPOT separately, since they have different causes.

Error 3: no SLO, so no goodput. Throughput without a latency constraint is maximized by an absurd batch size that violates every latency target. Define the SLO first — "TTFT under 500 ms and p95 TPOT under 50 ms" — then measure goodput: requests per second that met it. That number is what you are actually selling. A system at 20,000 tok/s aggregate with 80 ms TPOT against a 50 ms budget has excellent throughput and zero goodput.

Error 4: unrepresentative length distributions. Results depend enormously on input and output lengths, and fixed-length synthetic workloads (every request exactly 512 in, 128 out) produce numbers that do not transfer. Real distributions are heavy-tailed, and the tail is what stresses the scheduler. Use a real trace if you have one; failing that, a lognormal with a long tail.

Error 5: no warmup. First requests pay CUDA context creation, kernel autotuning, CUDA graph capture, memory pool growth, and cold prefix caches. Discard the first 10–20% of measurements.

Error 6: comparing at different operating points. Engine A at batch 256 versus engine B at batch 32 is not a comparison of engines. Fix the SLO, let each system choose its own best configuration under that constraint, and compare goodput. That is the only comparison that answers a real question.

A checklist worth keeping:

[ ] open-loop arrivals, Poisson, sweep the rate
[ ] realistic length distribution, heavy-tailed
[ ] warmup discarded
[ ] p50 / p95 / p99 for TTFT and TPOT separately
[ ] an explicit SLO, and goodput measured against it
[ ] each system tuned for the SLO, not copied settings
[ ] the same model, precision, and hardware -- state all three
[ ] enough duration to reach steady state

And when reading someone else's benchmark, the fastest tell: if it does not state batch size, context length distribution, and percentiles, it is not a measurement, it is a claim.

Module 6 made this point; it deserves expanding, because it is the weakest link in most production optimization work.

Why perplexity fails. It is the average negative log-likelihood over a corpus, dominated by common tokens in ordinary contexts — precisely what a quantized model reproduces almost perfectly. Damage concentrates in the tail. A 4-bit model can match fp16 perplexity to two decimals and fail visibly at 20-step arithmetic. Worse, perplexity is usually measured at 2k–4k tokens, exactly the regime where KV quantization looks harmless.

Why MMLU-style benchmarks are weak too. Multiple choice with four options is forgiving: the model needs the right answer to be marginally more likely than three alternatives, not to be confidently right. Degradation that would be obvious in open generation is invisible. Useful as a regression check, weak as a quality signal.

What to measure, in rough order of information per unit effort:

1. Long-context retrieval. RULER is the good one — multi-needle, aggregation, variable tracking, at configurable lengths. Needle-in-a-haystack is easier and less informative but better than nothing. This is where KV quantization and eviction damage shows first.
2. Multi-step reasoning. GSM8K, MATH, or your own chain-of-thought tasks. Errors compound, so small per-token degradation becomes visible.
3. Your actual task. A few hundred real prompts, graded against the fp16 baseline by a human or a strong judge model. Unglamorous, and the most informative item here.
4. Distributional divergence. KL between the optimized model's logits and the baseline's on a fixed prompt set. Cheap, sensitive, and catches problems before task metrics move.
5. Standard suites. As a regression gate only.

Control the comparison. Same prompts, same sampling parameters, same seed. Evaluating a quantized model at different settings than the baseline measures nothing, and it happens constantly.

Test at your operating point. If you serve 32k contexts at batch 64, evaluating at 2k context and batch 1 tells you very little. Batch composition itself can shift outputs (Module 3), so evaluate under batching.

Watch for the specific failure shapes. Quantization damage is not uniform noise. Common patterns: degradation that appears only past some context length; failures concentrated on rare tokens, numbers, and non-English text; loss of instruction-following precision while fluency is preserved; and reasoning that stays plausible while becoming wrong. Fluent-but-wrong is the dangerous one, because it survives casual inspection.

The honest position on most published quantization results is that they show the method does not catastrophically fail — not that it is free. Whether it is free for your workload is a question only your own evaluation answers.

The largest recent shift in inference is not a technique, it is a change in what the workload looks like.

A conventional request might be 2,000 prompt tokens and 200 output tokens. A reasoning request is 2,000 prompt tokens and 20,000 output tokens, most of which are thinking the user never sees.

Every consequence follows from Module 1's asymmetry:

Almost all wall-clock becomes decode. Prefill was already a small fraction; now it is negligible. For Llama-3-8B on an H100, 2k prefill is ~35 ms and 20,000 decode tokens are ~90 seconds. Prefill is 0.04% of the request. All optimization effort belongs on the memory-bound side.

TTFT stops mattering. The user waits 90 seconds regardless; whether the first thinking token appears at 35 ms or 300 ms is irrelevant. This weakens the case for chunked prefill and strengthens the case for large prefill batches — the opposite of the conventional-chat conclusion.

Batch slots are held far longer. A sequence occupying a slot for 20,000 steps instead of 200 means 100× less slot turnover. Continuous batching's advantage over static batching shrinks, because there is less variance to exploit. Admission control matters more, because a bad admission decision costs you for minutes.

KV cache grows much larger per request. 20,000 tokens of cache per sequence rather than 2,200. At 128 KiB/token that is 2.5 GiB per sequence for an 8B model — so concurrency falls sharply, and every technique in Module 6 becomes more valuable.

Speculation becomes more attractive. Thinking tokens are often highly predictable — self-consistent reasoning has a lot of structure — and since the user never sees them, per-token latency is irrelevant; only total time matters. That combination favours aggressive speculation even at moderate batch, and n-gram lookup can do well on repetitive reasoning patterns.

New questions with no settled answers. Should thinking tokens use a cheaper decode path, or a smaller model, or more aggressive quantization, given they are never displayed? Can you detect a reasoning trace that is going nowhere and prune it mid-flight? Should reasoning requests be scheduled on a separate pool given their radically different profile? Is there a principled way to trade thinking length against answer quality at serve time?

The honest summary: the field's serving infrastructure was designed for a workload that no longer describes a growing share of traffic, and the adaptation is very much in progress.

Where the machinery in this course runs out.

1. Long context is still fundamentally unaffordable. The arithmetic does not improve with cleverness: Llama-3-70B at 128k context needs 40 GiB of cache per sequence, which is three concurrent users on four H100s. Every attack trades something real — eviction is lossy in ways you cannot bound in advance, compression concentrates its damage on exactly the long-context tasks you bought the context for, architectural fixes need training from scratch, and offload runs into a 50× bandwidth cliff. There is no approach that is simultaneously lossless, architecture-agnostic and cheap.

2. We cannot evaluate optimized models honestly. Arguably the most important unglamorous problem in the field. There is no standard, trusted suite that answers "is this quantized model still the model I tested?", which means a great deal of production quantization is deployed on faith and spot-checks.

3. Speculation and batching fight each other. Both spend the same idle arithmetic (Module 8), so speculation stops paying around the batch size where you approach the ridge point. Adaptive schemes — choosing draft length per request per step from current occupancy and observed acceptance — are the obvious answer, and doing it well, with a scheduler that models the interaction, is unsolved.

4. Agentic workloads are a cache-policy problem nobody has solved. An agent re-sends its whole growing trajectory every step, so prefix caching turns quadratic prefill work into linear. That makes the cache the working set of the system, and the hard questions become: what do you evict when holding hundreds of partially-shared trajectories? Should you keep a trajectory warm while its agent waits on a slow external tool call, paying memory for latency you do not control? How do you schedule fairly when one agent's cache is worth 50× another's? Under-explored relative to its importance.

5. Disaggregation at scale is awkward in practice. Clearly right in principle (Module 9), but placement across heterogeneous pools, sizing the two pools as traffic mix shifts, and hiding the KV transfer are all open engineering problems.

6. The memory wall keeps widening. Compute has grown faster than bandwidth for a decade — the ridge point went from 153 on an A100 to 295 on an H100. Every generation makes decode relatively worse. FP4 and FP8 help by shrinking bytes and larger HBM helps capacity, but the structural fix would be a memory-centric architecture with far more bandwidth per FLOP, or compute placed nearer memory. Several companies are betting on variants of this. Whether any displaces GPUs at scale is genuinely uncertain, and it is the question that would most change everything else in this course.

7. The economics have no shared vocabulary. Inference is now the dominant cost of operating a model, and the field has no common language for cost-quality trade-offs. What is the right price for a token that took 20,000 thinking tokens to produce? How do you expose the latency/throughput dial to customers who do not know they want it? Not technical questions, but they increasingly determine which technical work gets done.

A note on currency: this course was written from the author's knowledge and not verified against live sources at build time. Points 3, 4 and 6 move fastest. Before relying on any specific claim, search for recent work — and if something here has been solved since, that is a good sign.

You have the framework. The way to convert it into understanding is to use it on something real.

Reproduce a published benchmark and account for the gap. This is the highest-value exercise in the course, and it is the one in the original outline for good reason. Pick a published number — a vLLM blog post, an engine comparison, a model card's throughput claim. Reproduce it. You will not match it. Then find out why.

The discrepancies are where the understanding is, and the list of causes is a summary of this whole course: different hardware SKU, different achieved bandwidth, different length distribution, closed-loop versus open-loop load, mean versus percentile, different batch size, warmup included, different precision, speculative decoding silently enabled, chunked prefill settings, or a prefix-cache hit rate you did not have. Working through that list on a real discrepancy will teach you more than re-reading any of these modules.

Then, in rough order of value:

Instrument something you run. Measure TTFT and TPOT percentiles on a real workload. Compute the roofline floor. Calculate your gap. If it is over 2×, find out which item from Module 4's list is responsible. This is a genuinely useful skill and few people have it.

Read the code. vLLM's block manager and scheduler are readable and are the clearest expression of Modules 2, 5 and 6 in practice. Read v1/core/ and follow one request through admission, scheduling and eviction.

Pick one module and go deeper than this course did. Each of these has a substantial literature behind it: KV compression, speculative decoding variants, MoE serving, long-context attention, structured decoding. Depth in one is worth more than breadth across all.

Build the small thing. A KV cache calculator you actually use. A roofline planner. A batching simulator. The labs in this course are starting points — the versions you extend for your own hardware and models are the ones you will keep.

Stay current deliberately. This field moves fast enough that specific numbers age within months. What does not age is the framework: bytes moved per unit of useful work, and which side of the ridge point you are on. When you meet a new technique, ask what term of the cost model it attacks and at what operating point. If the answer is not clear, the technique probably is not either.