LLM INFERENCE SCORE000000 CLEARED00/11

REFERENCE

TIMELINE

The papers and systems that actually changed how inference is done — each one an answer to the same question about bytes moved from memory.

Read this as a single argument rather than a list. In 2017 nobody was thinking about inference cost, because models were small enough that it did not matter. By 2020 prompts had got long. By 2022 the models had got big enough that the memory wall became the only thing that mattered, and every year since has produced another attack on it — architectural, algorithmic, systems-level, or hardware.

Dates are the arXiv preprint month where one exists, since that is usually when the idea entered circulation rather than when a venue accepted it.

  1. 2017-06

    Attention Is All You Need

    The transformer. Encoder-decoder, post-norm, learned positional encodings, and no thought given to generation cost.

    Establishes the architecture whose weights you will spend the next decade trying to move less often. Also establishes causal masking, which is what makes a KV cache legal at all.

  2. 2019-04

    The Curious Case of Neural Text Degeneration

    Nucleus (top-p) sampling, and the demonstration that maximizing likelihood produces degenerate text.

    Kills beam search for open-ended generation — which incidentally removes a k-fold multiplier on KV cache and compute that nobody could have afforded at scale.

  3. 2019-09

    Megatron-LM

    Tensor parallelism: split each layer across GPUs with column-parallel then row-parallel matmuls, costing two all-reduces per layer.

    The mechanism by which a model too large for one GPU becomes servable. It also multiplies aggregate bandwidth, which is why TP helps decode latency and not just capacity.

  4. 2019-11

    Fast Transformer Decoding: One Write-Head is All You Need

    Multi-query attention. One KV head shared by all query heads, shrinking the cache by the head count.

    The first paper to name the KV cache as *the* inference bottleneck and attack it architecturally. Four years ahead of the field, and largely ignored until the field caught up.

  5. 2020-05

    GPT-3 / in-context learning

    Few-shot prompting as the standard interface, which made prompts 10–100x longer overnight.

    Turns prefill from a rounding error into a distinct, compute-bound workload with its own latency budget. Everything about TTFT starts here.

  6. 2022-05

    FlashAttention

    Tiling plus online softmax, computing exact attention without ever materializing the N×N score matrix in HBM.

    The clearest single demonstration of the thesis: same arithmetic, far fewer bytes moved, 2–4x faster. Memory traffic drops from O(N²) to O(N²d²/M) where M is SRAM size.

  7. 2022-07

    Orca (OSDI ’22)

    Continuous / iteration-level batching: evict finished sequences and admit new ones at every decode step rather than every request.

    Turns the batch dial from a static setting into a live one. Since decode arithmetic intensity equals batch size, keeping the batch full is the single largest throughput lever in serving.

  8. 2022-10

    GPTQ

    Accurate post-training weight quantization to 3–4 bits using approximate second-order information.

    Weights are most of the bytes you move at small batch, so 4-bit weights are close to a 4x decode speedup in the weight-dominated regime. No retraining required.

  9. 2022-11

    Efficiently Scaling Transformer Inference

    A rigorous cost model for inference, partitioning strategies derived from it, and the latency/throughput Pareto frontier made explicit.

    The paper that made roofline reasoning standard practice for inference. If you internalize one analytical framework, take it from here.

  10. 2022-11

    Speculative decoding

    Draft several tokens with a cheap model, verify them in one parallel pass of the target model, and accept via modified rejection sampling.

    The only technique that shortens the sequential dependency chain rather than making each link cheaper — and it provably preserves the output distribution exactly.

  11. 2023-05

    GQA

    Grouped-query attention: an interpolation between MHA and MQA, plus a recipe for converting existing MHA checkpoints cheaply.

    The compromise the whole field adopted. Cuts the KV cache 8x on a 70B model with minimal quality loss, and is what makes long-context serving economically possible.

  12. 2023-07

    FlashAttention-2

    Better work partitioning across warps and thread blocks, and fewer non-matmul FLOPs. Roughly 2x over FlashAttention-1.

    A reminder that on modern GPUs, non-tensor-core arithmetic is comparatively expensive and scheduling is as important as the algorithm.

  13. 2023-09

    vLLM / PagedAttention (SOSP ’23)

    Virtual-memory paging applied to the KV cache: fixed-size blocks, per-sequence block tables, non-contiguous physical storage, copy-on-write sharing.

    Measured KV memory utilization in prior systems at 20.4–38.2%. Paging pushes waste below 4%, and the recovered memory converts directly into batch size and therefore throughput.

  14. 2023-11

    Splitwise

    Run prefill and decode on separate machine pools, transferring the KV cache between them.

    Takes the prefill/decode asymmetry to its logical conclusion: two workloads with opposite bottlenecks should not share a scheduler, let alone a GPU.

  15. 2023-12

    SGLang / RadixAttention

    A radix tree over cached KV prefixes, enabling automatic longest-prefix reuse across arbitrary request patterns, plus a frontend language for structured generation.

    Turns prefix sharing from a special case (one fixed system prompt) into a general mechanism. Decisive for agentic and multi-turn workloads, where history is re-sent every step.

  16. 2023-12

    Mamba

    A selective state-space model with constant-size recurrent state, so there is no KV cache that grows with sequence length.

    The alternative bet: rather than compress the cache, remove it. Pure SSMs lost ground on exact recall, but hybrid attention/SSM stacks took the idea mainstream.

  17. 2024-01

    Medusa and EAGLE

    Self-speculation: extra decoding heads (Medusa) or feature-level autoregression (EAGLE) generate drafts without a separate draft model.

    Removes the main practical objection to speculative decoding — that you need a second well-aligned model with its own memory footprint and its own weights to stream.

  18. 2024-01

    DistServe (OSDI ’24)

    Prefill/decode disaggregation with per-phase parallelism, optimizing goodput under explicit TTFT and TPOT SLOs.

    Makes goodput — throughput that actually meets its latency target — the metric, and shows that co-locating the two phases costs you more than the KV transfer does.

  19. 2024-05

    DeepSeek-V2 / Multi-head Latent Attention

    Compress K and V into a shared low-rank latent vector per token, cached in place of the full per-head keys and values.

    The most aggressive architectural attack on the cache to date — a reported ~93% KV reduction versus MHA — while claiming *better* quality than MHA rather than a trade.

  20. 2024-07

    FlashAttention-3

    Hopper-specific: warp specialization, asynchronous TMA copies, and interleaved matmul/softmax to hide the non-tensor-core work. Adds FP8 support.

    Shows that extracting the last factor of two now requires exploiting specific hardware asynchrony, not just better asymptotics.

  21. 2025-01

    Reasoning models change the workload shape

    Long chain-of-thought models that spend thousands of tokens thinking before answering.

    Shifts the decode-to-prefill ratio dramatically toward decode — the memory-bound half. A workload that was 60% prefill by FLOPs can become almost entirely decode by wall-clock.