REFERENCE
SERVING ENGINES
Not a feature table. An argument about what each engine optimizes for, and what it gives up to get there.
Feature tables go stale in weeks and tell you nothing about why a system is shaped the way it is. What follows is the design philosophy of each engine — the thing that stays true between releases.
vLLM — memory management as the organizing principle
The bet: if you fix KV cache fragmentation, everything else follows. Recovered memory becomes batch size, batch size becomes arithmetic intensity, arithmetic intensity becomes throughput.
PagedAttention is the mechanism, and it drove the whole design. Blocks of 16 tokens, per-sequence block tables, non-contiguous physical storage, copy-on-write for shared prefixes. Continuous batching sits on top and depends on it — you cannot cheaply admit and evict sequences mid-flight if each one owns a contiguous reservation.
What it gives up: peak single-request latency. vLLM is built around keeping many sequences in flight, and the scheduler, block manager and Python-side machinery cost you something at batch 1. It has also historically trailed TensorRT-LLM on raw kernel performance for specific model/hardware pairs, because it optimizes for generality across a very large model zoo.
Pick it when: you are serving many concurrent users, you want broad model support, and you want to deploy today without a compilation step.
SGLang — the workload is not one request at a time
The bet: real traffic is structured and repetitive. Requests share system prompts, few-shot examples, documents, and conversation history. An engine that treats each request as independent is leaving most of the work on the table.
RadixAttention is the mechanism: a radix tree over cached KV prefixes with automatic longest-prefix matching and LRU eviction. Where vLLM's prefix caching handles the common case, SGLang treats sharing as the default structure of the workload. Its frontend language makes that explicit, letting you express branching, parallel calls and constrained generation as a program rather than a sequence of opaque API calls. Structured output is a first-class concern rather than a wrapper.
What it gives up: simplicity, and some generality. The programming model is more opinionated, and the benefits are largest for workloads that genuinely share prefixes.
Pick it when: agentic loops, multi-turn chat with long histories, RAG over a shared corpus, or heavy structured/JSON output — anywhere the prefix reuse rate is high.
TensorRT-LLM — compile ahead of time, run with no surprises
The bet: the last 20–30% of performance lives in kernel selection, fusion and layout decisions that are only correct for a specific model on specific hardware at a specific batch size. So make those decisions ahead of time, at build time, with full knowledge of the deployment.
You compile a model into an engine artifact for a target GPU, precision, batch size range and sequence length range. The result is typically the fastest option on NVIDIA hardware, with the tightest and most predictable latency.
What it gives up: flexibility, and a great deal of operational convenience. The engine is tied to the GPU it was built for. Changing max batch size or context length means recompiling. Build times are long, and new model architectures take longer to land than in the Python-first engines.
Pick it when: a fixed model on fixed NVIDIA hardware at scale, where a 25% efficiency gain is worth a compilation pipeline in your deploy process.
llama.cpp — the weight-dominated regime, taken seriously
The bet: most people running a model locally have one user, a short context, and not enough memory. That is the weight-dominated regime from Level 02, where weight bytes are essentially the entire cost of a decode step — so weight quantization is not one optimization among many, it is the optimization.
Hence the k-quant family and an obsessive focus on getting good quality out of 2–5 bits per weight. Hence also CPU inference, GPU offloading of a subset of layers, Metal and Vulkan backends, and a zero-dependency C++ build. Different regime, different correct answers.
What it gives up: throughput at scale. Continuous batching and paged KV arrived late and are not the design centre. Serving hundreds of concurrent users is not what it is for.
Pick it when: local or edge inference, consumer hardware, one or few users, or you need the model to fit in memory it does not obviously fit in.
How to choose, in one table
| your situation | engine |
|---|---|
| many users, many models, deploy today | vLLM |
| high prefix reuse: agents, chat, RAG | SGLang |
| heavy structured / JSON output | SGLang |
| one model, NVIDIA, squeeze the last 25% | TensorRT-LLM |
| strict and predictable p99 latency | TensorRT-LLM |
| laptop, desktop, edge, single user | llama.cpp |
| CPU-only or partial GPU offload | llama.cpp |
What they all agree on
The convergence is as informative as the differences. Every serious engine now implements continuous batching, paged KV cache, some form of prefix caching, chunked prefill, CUDA graphs, fused kernels, FlashAttention-family kernels, quantization (weights, and increasingly KV), and speculative decoding.
Those are no longer differentiators. They are table stakes, and the fact that four independently designed systems converged on the same list is decent evidence that the list is correct.
Version numbers and specific feature availability move fast. Treat the philosophies above as durable and check current documentation for anything you are about to depend on.