SERVING ENGINES

Feature tables go stale in weeks and tell you nothing about why a system is shaped the way it is. What follows is the design philosophy of each engine — the thing that stays true between releases.

vLLM — memory management as the organizing principle

The bet: if you fix KV cache fragmentation, everything else follows. Recovered memory becomes batch size, batch size becomes arithmetic intensity, arithmetic intensity becomes throughput.

PagedAttention is the mechanism, and it drove the whole design. Blocks of 16 tokens, per-sequence block tables, non-contiguous physical storage, copy-on-write for shared prefixes. Continuous batching sits on top and depends on it — you cannot cheaply admit and evict sequences mid-flight if each one owns a contiguous reservation.

What it gives up: peak single-request latency. vLLM is built around keeping many sequences in flight, and the scheduler, block manager and Python-side machinery cost you something at batch 1. It has also historically trailed TensorRT-LLM on raw kernel performance for specific model/hardware pairs, because it optimizes for generality across a very large model zoo.

Pick it when: you are serving many concurrent users, you want broad model support, and you want to deploy today without a compilation step.

How vLLM works

Everything above is downstream of one data structure: the block table. The KV cache is carved into fixed-size blocks — 16 tokens each, in the original design — drawn from a single pool shared by every sequence resident on the GPU. Each sequence gets a block table, a small array mapping its logical block indices (0, 1, 2, …) to physical block indices anywhere in that pool, the same indirection an OS uses for virtual-memory pages. A sequence claims a new physical block only when its current one fills, so allocation tracks actual usage rather than a worst-case reservation. The paged-attention kernel reads through this indirection directly — given a block table and a slot mapping, it gathers the right K/V vectors from scattered memory instead of assuming one contiguous buffer per sequence.

The scheduler runs once per token, not once per request. Continuous batching — the technique Orca introduced as iteration-level scheduling — makes the forward pass, not the sequence, the unit of scheduling. Every step, the scheduler looks at its running, waiting and swapped queues and decides who is in this iteration's batch: running sequences advance by one token, waiting sequences are admitted if blocks are free, and chunked prefill slices a long prompt into pieces so it interleaves with other sequences' decode steps instead of blocking them for the length of a full prefill pass. A sequence can enter or leave the batch on any step — nothing waits on a fixed-size batch's slowest member, which is the entire point of scheduling at the iteration level instead of the request level.

Preemption is what happens when the pool runs dry. If every physical block is spoken for and a running sequence needs one more, the scheduler evicts the lowest-priority sequence rather than failing the request, by one of two means: swap its blocks out to host memory and back in later, or recompute — drop the blocks and rerun prefill over its tokens-so-far once it is rescheduled. Recompute tends to win for short sequences, where redoing the prefill is cheap; swap wins once a sequence is long enough that its cache is expensive to regenerate but comparatively cheap to move over PCIe.

Copy-on-write turns a shared prefix into shared physical blocks. Two sequences that share tokens — parallel samples from one prompt, or two unrelated requests behind the same system prompt — can point their block tables at identical physical blocks instead of duplicating them, tracked by a reference count per block. A write forks only the block being written: the instant one sequence's continuation diverges from the other's, that single block is copied and the two block tables split there, not before. Automatic prefix caching generalizes this to requests that never asked to share anything — each block is hashed on its own tokens chained with its prefix's hash, so a new request reproducing a previously seen prefix resolves to existing physical blocks on lookup instead of allocating and recomputing them.

Multi-GPU is a driver plus workers. Under tensor or pipeline parallelism, one driver process owns the scheduler and the authoritative block tables; it broadcasts each step's batch to one worker process per GPU, and workers exchange activations through NCCL collectives at the points the parallelism strategy requires them. The block manager stays single-writer on the driver — workers execute against the block tables they are handed rather than negotiating allocation among themselves.

SGLang — the workload is not one request at a time

The bet: real traffic is structured and repetitive. Requests share system prompts, few-shot examples, documents, and conversation history. An engine that treats each request as independent is leaving most of the work on the table.

RadixAttention is the mechanism: a radix tree over cached KV prefixes with automatic longest-prefix matching and LRU eviction. Where vLLM's prefix caching handles the common case, SGLang treats sharing as the default structure of the workload. Its frontend language makes that explicit, letting you express branching, parallel calls and constrained generation as a program rather than a sequence of opaque API calls. Structured output is a first-class concern rather than a wrapper.

What it gives up: simplicity, and some generality. The programming model is more opinionated, and the benefits are largest for workloads that genuinely share prefixes.

Pick it when: agentic loops, multi-turn chat with long histories, RAG over a shared corpus, or heavy structured/JSON output — anywhere the prefix reuse rate is high.

TensorRT-LLM — compile ahead of time, run with no surprises

The bet: the last 20–30% of performance lives in kernel selection, fusion and layout decisions that are only correct for a specific model on specific hardware at a specific batch size. So make those decisions ahead of time, at build time, with full knowledge of the deployment.

You compile a model into an engine artifact for a target GPU, precision, batch size range and sequence length range. The result is typically the fastest option on NVIDIA hardware, with the tightest and most predictable latency.

What it gives up: flexibility, and a great deal of operational convenience. The engine is tied to the GPU it was built for. Changing max batch size or context length means recompiling. Build times are long, and new model architectures take longer to land than in the Python-first engines.

Pick it when: a fixed model on fixed NVIDIA hardware at scale, where a 25% efficiency gain is worth a compilation pipeline in your deploy process.

llama.cpp — the weight-dominated regime, taken seriously

The bet: most people running a model locally have one user, a short context, and not enough memory. That is the weight-dominated regime from Level 02, where weight bytes are essentially the entire cost of a decode step — so weight quantization is not one optimization among many, it is the optimization.

Hence the k-quant family and an obsessive focus on getting good quality out of 2–5 bits per weight. Hence also CPU inference, GPU offloading of a subset of layers, Metal and Vulkan backends, and a zero-dependency C++ build. Different regime, different correct answers.

What it gives up: throughput at scale. Continuous batching and paged KV arrived late and are not the design centre. Serving hundreds of concurrent users is not what it is for.

Pick it when: local or edge inference, consumer hardware, one or few users, or you need the model to fit in memory it does not obviously fit in.

How to choose, in one table

your situation	engine
many users, many models, deploy today	vLLM
high prefix reuse: agents, chat, RAG	SGLang
heavy structured / JSON output	SGLang
one model, NVIDIA, squeeze the last 25%	TensorRT-LLM
strict and predictable p99 latency	TensorRT-LLM
laptop, desktop, edge, single user	llama.cpp
CPU-only or partial GPU offload	llama.cpp

What they all agree on

The convergence is as informative as the differences. Every serious engine now implements continuous batching, paged KV cache, some form of prefix caching, chunked prefill, CUDA graphs, fused kernels, FlashAttention-family kernels, quantization (weights, and increasingly KV), and speculative decoding.

Those are no longer differentiators. They are table stakes, and the fact that four independently designed systems converged on the same list is decent evidence that the list is correct.

Version numbers and specific feature availability move fast. Treat the philosophies above as durable and check current documentation for anything you are about to depend on.