GLOSSARY

Probability a drafted token survives verification. Equals sum min(p, q), i.e. 1 minus the total variation distance between draft and target.

The fraction of peak HBM bandwidth a real kernel reaches. Typically 70-85%.

Deciding how many sequences to admit given that their eventual lengths, and therefore memory needs, are unknown.

Collective that sums a tensor across all GPUs and returns the result to all. Ring implementation moves 2(N-1)/N of the data per GPU.

FLOPs performed per byte moved from memory. Compare it against the hardware ratio to find your bottleneck.

A stack mixing mostly SSM or linear-attention layers with a few full-attention layers, trading some memory saving for exact recall.

The first few tokens of a sequence, which receive large attention weight regardless of content. Keeping them enables indefinite streaming.

Activation-aware weight quantization: scale up the ~1% of channels identified as salient from activation magnitude, then quantize.

The property that a kernel produces bit-identical results regardless of batch composition. Required for true determinism; costs performance.

Per-sequence map from logical token positions to physical KV cache blocks. What allows non-contiguous allocation.

The extra token available when all gamma drafts are accepted, since the verification pass already computed p at the final position.

2 x n_layers x n_kv_heads x head_dim x dtype_bytes. The most useful derived number in inference planning.

A cap on tokens routed to any single MoE expert, bounding load imbalance at the cost of dropping or rerouting overflow.

Splitting a long prefill into fixed-size pieces processed one per iteration, mixed with ongoing decode.

Fixed concurrency; a new request is sent only when one finishes. Self-throttling, so it hides the capacity cliff.

Megatron's arrangement: split the first matmul by columns and the second by rows, so the nonlinearity between them needs no communication.

Masking logits each step so only tokens consistent with a grammar or schema can be sampled.

Iteration-level scheduling: evict finished sequences and admit waiting ones at every decode step. From Orca.

Sequences sharing a prefix point at the same physical blocks; a block is duplicated only when one of them writes to it.

The number of cached tokens in flight at which KV traffic overtakes weight traffic. About 114k tokens for Llama-3-8B at fp16.

A captured, replayable sequence of kernel launches. Removes per-launch CPU overhead, which is significant when a decode step is hundreds of tiny kernels.

The per-token forward passes that follow prefill, one position at a time. Memory-bound; sets TPOT.

The repetitive-loop failure mode of likelihood-maximizing decoders. Named and diagnosed by Holtzman et al.

Vendors quote 2:4 structured sparsity figures that are double the dense number. Production LLMs use dense.

Running prefill and decode on separate machine pools, transferring the KV cache between them.

A small, cheap model that proposes candidate tokens for the target model to verify.

Self-speculation by autoregressing over the target's hidden features rather than tokens. The strongest general method.

Distributing MoE experts across GPUs. Requires all-to-all routing and suffers from load imbalance.

Free memory that exists but is broken into pieces too small to satisfy a contiguous allocation.

Tiled exact attention that never materializes the N x N score matrix in HBM. Cuts HBM accesses to O(N^2 d^2 / M).

Decode-phase attention that parallelizes along the KV dimension (split-K) because query-length-1 offers no query parallelism.

Subtracts a term proportional to how many times a token has appeared. Additive in logit space, better behaved than division.

Draft length: how many tokens are proposed before each verification pass.

Throughput counting only requests that met their latency SLO. The metric that matches what you are actually selling.

Post-training weight quantization using approximate second-order information, layer by layer. Works to 3-4 bits.

Grouped-query attention. Fewer K/V heads than Q heads, with each K/V head shared by a group of query heads. Shrinks the KV cache proportionally.

The width of one attention head. Usually d_model / n_heads, though it can be set independently.

One d_model-length slice of the residual stream, at one position and one layer.

Memory reserved inside a sequence allocation that the sequence never uses. The dominant waste in naive allocators.

Combining adjacent operations into one kernel so intermediates never round-trip to HBM.

The batch size beyond which throughput gains become small relative to the latency cost. The right default operating point.

The stored key and value tensors for all previous positions, kept so they are not recomputed each step. Per-sequence, and it grows with every token.

Replacing softmax with a kernel feature map so attention can be reassociated into O(N d^2). Constant state, weaker exact recall.

concurrency = arrival rate x latency. Tells you how many requests must be in flight to hit a throughput target.

Local-inference engine optimized for the weight-dominated regime: aggressive weight quantization, CPU and partial-offload support.

The unnormalized [.., vocab_size] scores produced by the LM head, before any softmax or sampling.

Self-speculation with multiple independent decoding heads predicting several positions ahead, verified as a tree.

Keep tokens with probability at least min_p times the top token's probability. Fixed ratio, adaptive to confidence.

Multi-head latent attention. Compresses K and V into a shared low-rank latent per token; the up-projections are absorbed into neighbouring matrices.

Accept with probability min(1, p/q); on rejection sample from normalized max(0, p - q). Provably yields p.

Multi-query attention: a single KV head shared by all query heads. Maximum cache savings, largest quality cost.

Drafting by searching the context for the current suffix and proposing what followed it. Free, and excellent on repetitive text.

High-bandwidth GPU-to-GPU interconnect, ~900 GB/s on H100. Roughly 14x faster than PCIe Gen5.

Computing softmax incrementally with a running max and sum, rescaling earlier partials when a larger value appears. Exact, not approximate.

Arrivals follow an external schedule (usually Poisson) regardless of server state. The only way to observe overload.

Fixed-size block allocation for the KV cache with per-sequence block tables, borrowed from OS virtual memory. Cuts fragmentation waste from 60-80% to under 4%.

A separate scale for each channel of a tensor. Necessary for keys, which have strong per-channel outliers.

Idle stage time while the pipeline fills and drains. Fraction is (N-1)/(M+N-1) for M microbatches and N stages.

Assigning different layers to different GPUs. One activation hand-off per stage boundary. Tolerates slow links.

Normalizing the input to each block rather than its output, leaving the residual path unbroken. Required in practice for deep transformers.

Reclaiming a running sequence's KV memory, either by swapping it to host memory or discarding and recomputing it.

The forward pass over the entire prompt, done in one shot. Compute-bound; sets TTFT.

A long compute-bound prefill blocking every decoding sequence, spiking their inter-token latency.

Reusing cached KV state across requests that share a leading token sequence. Exact, not approximate.

SGLang's radix-tree index over cached prefixes, enabling automatic longest-prefix reuse across arbitrary sharing patterns.

Requests that generate thousands of hidden chain-of-thought tokens, shifting nearly all wall-clock into memory-bound decode.

Divides logits of already-seen tokens by r > 1. Blunt, and harmful for code and structured output.

The normalized positive part of p - q. The mass the target wanted that the draft undersupplied.

The [batch, seq, d_model] tensor that every block reads from and adds back into. Its width is constant through the network.

The arithmetic intensity at which a machine transitions from memory-bound to compute-bound. About 295 FLOP/byte for an H100.

Normalization by root-mean-square only, with no mean subtraction and no bias. Cheaper than LayerNorm and works as well.

attainable = min(peak FLOP/s, intensity x bandwidth). A two-line model that identifies your bottleneck.

Rotary position embedding. Rotates Q and K by an angle proportional to absolute position so their dot product depends only on relative distance.

A long-context benchmark testing multi-needle retrieval, aggregation and tracing. Far more informative than perplexity for evaluating KV compression.

Batch the position-independent operations across sequences while handling attention per-sequence. What makes continuous batching implementable.

Generating drafts from the target model itself via extra heads or a small feature-level head, avoiding a separate draft model.

Splitting the sequence dimension across devices, as in Ring Attention. Used when one sequence exceeds a single GPU.

Serving engine organized around structured, prefix-sharing workloads: RadixAttention plus a frontend programming language.

On-chip memory, ~228 KB per SM on Hopper, with roughly 6x the bandwidth of HBM. The tier FlashAttention keeps its tiles in.

Migrates activation outliers into the weights via per-channel scaling, enabling int8 for both operands.

Partitioning a reduction across parallel workers and combining their partial results afterward.

State space model. Maintains a fixed-size recurrent state, so there is no KV cache; Mamba is the leading example.

Fixed batch that runs to completion. Utilization collapses when sequence lengths vary, and gets worse as batch grows.

The gated feed-forward block used by Llama-family models: down(silu(gate(x)) * up(x)). Three weight matrices instead of two.

Training on the known target sequence so all positions are computed in one parallel pass. The reason training is far more hardware-efficient than generation.

Divisor applied to logits before softmax. Below 1 sharpens, above 1 flattens.

Splitting weight matrices within a layer across GPUs. Two all-reduces per transformer layer. Needs NVLink.

NVIDIA serving engine that compiles a model into a hardware-specific engine ahead of time. Fastest and least flexible.

The cap on tokens processed per scheduler iteration. Decode tokens are admitted first, prefill chunks fill the rest.

Keep the k highest-probability tokens. Fixed count, does not adapt to the distribution.

Keep the smallest set whose cumulative probability reaches p. Fixed mass, adaptive count.

Time per output token / inter-token latency. The gap between successive tokens once streaming starts.

Verifying many candidate continuations in one pass using a mask where each node attends only to its ancestors.

Time to first token. How long the user waits before anything appears.

Serving engine organized around memory management: PagedAttention, continuous batching, broad model support.

Assigning different warps within a block to different roles (producer/consumer) so data movement overlaps computation. Central to FlashAttention-3.

GPU memory reserved for activations, temporary buffers, and communication. Typically 10-20 GB, and easy to forget when budgeting.