MODEL ARCHITECTURES

Module 1 established that a decode step reads the weights once to produce one token. Module 2 added the cache that grows with context. Module 4 turned both into time. Put the three together and one line prices any architecture:

bytes per decode step  ~=  W_active  +  B x S x kv_per_token
step time              ~=  bytes / achieved bandwidth

W_active is the weight bytes actually read, B is the number of sequences in flight, S is their mean length. The asymmetry between the two terms is the thing to internalise: the weight term is paid once per step no matter how many sequences you are serving, and the cache term is paid once per sequence. That is the entire reason batching works, and it is why the two terms have to be tracked separately rather than added into a single "model size".

For Llama-3-8B at fp16, the two numbers are 15.0 GB and 128 KiB. At batch 1 and 8k context:

weights  15.0 GB
cache     8192 x 131072 = 1.07 GB
                          ---------
                          16.1 GB  ->  4.8 ms on an H100  ->  208 tok/s

At batch 64 the weight term does not move and the cache term multiplies by 64, to 68.7 GB. The model has not changed. Which term you are paying has.

Here is the whole level in one table — four families, four different bets:

Two warnings before the details. First, the axis an architecture does not attack tends to get worse, because designers spend the savings: MoE models are enormous, MLA models are dense, and hybrids that keep a few attention layers keep a cache for them. Second, quantization scales both terms and is not on this list, because it is a deployment choice rather than an architectural one. Module 6 covers it, and the two compose — a fp8 MoE moves half of a small number.

Three shapes came out of the original transformer, and only one of them is what you serve today.

Encoder-only — BERT and its descendants. Bidirectional attention over a fixed input, no generation. Still the right tool for classification, retrieval embeddings and reranking, and still deployed everywhere in the retrieval half of a RAG system. It has no decode loop, so none of these chapters apply to it.

Encoder-decoder — T5, BART, Whisper. The encoder reads the input bidirectionally once. The decoder generates, and each decoder layer has two attention blocks: self-attention over what it has generated so far, and cross-attention over the encoder's output.

The inference-relevant fact about that shape is one people usually get backwards. The cross-attention K and V are projections of the encoder output, so they are computed once per request and never grow. Only the decoder's self-attention cache grows, and it grows with the output length rather than the input length. For a task with a long input and a short output — summarization, translation, transcription — an encoder-decoder has a smaller and flatter cache profile than a decoder-only model doing the same job. Whisper is the surviving example most people have running: its encoder consumes a fixed 30-second window of mel spectrogram, so its cross-attention cache is a constant.

So encoder-decoder is not the expensive shape. It is the inflexible one: two parameter sets to hold, an input that must be fully encoded before any output appears, and no way to append to the encoded region without recomputing it.

Decoder-only won, and it is worth being accurate about why. The engineering conveniences are real — one set of weights, one code path for prefill and decode, and a cache that stays valid as the sequence grows (Module 0's causality argument). But the published evidence is about quality: Wang et al. compared architectures and objectives head to head and found causal decoder-only models trained with a plain autoregressive objective had the strongest zero-shot generalization after unsupervised pretraining. The systems properties are why nobody has been tempted back.

Prefix-LM is the middle ground that keeps coming up and never quite arrives: bidirectional attention over the prompt, causal attention over the continuation. UL2's S-denoising is exactly this, and PaliGemma uses it so that image and prompt tokens see each other in both directions. The serving consequence is the one that keeps it niche — a bidirectional region cannot be extended without recomputing it, so multi-turn chat, where each turn appends to the previous prompt, loses the prefix caching from Module 6.

Module 2 derived the formula and Module 6 spent the savings. Here it is as a family tree, because the variants are usually presented as a list of tricks when they are a single idea applied at three different points:

cache bytes per token  =  2 x n_layers x n_kv_heads x head_dim x dtype_bytes
                          ^        ^          ^          ^         ^
                          |        |          |          |         └─ quantization (Module 6)
                          |        |          |          └─ MLA compresses this jointly
                          |        |          └─ MQA and GQA cut this
                          |        └─ cross-layer sharing cuts this
                          └─ MLA removes the factor of 2

Hold the Llama-3-70B shape fixed — 80 layers, head_dim 128, fp16 — and vary only the attention design:

MQA is not the winner despite the best number, because a single shared KV head costs more quality than most models can afford; GQA at 4 to 8 groups is where the field settled. Module 6 has the mechanism for MLA and the DeepSeek result — a 93% reduction against MHA with benchmark quality that went up rather than down.

The term this level adds is the one on the left. Cross-layer KV sharing lets neighbouring layers reuse a single set of K and V, cutting n_layers in the formula rather than the head count. Character.AI reported stacking three of these attacks in production: MQA, a 5:1 pattern of local to global layers with a 1024-token horizon, and cross-layer sharing worth a further 2–3×, for a combined reduction of more than 20× with no quality regression they could measure. That is the shape of the modern answer — not one clever attention variant, but three multiplicative ones chosen because they compose.

And the constraint that governs all of it, restated because it is the reason this level exists: these are training-time decisions. GQA has an uptraining recipe that costs about 5% of original pre-training compute. MLA does not — it must be trained from scratch. Cross-layer sharing changes what the model is. When you pick a model to serve you are picking a row of that table, and the row is not negotiable.

Module 9 covered how to serve a mixture of experts — expert parallelism, all-to-all, load imbalance. This is the architecture, and the arithmetic underneath the headline numbers.

An MoE layer replaces the single MLP with E expert MLPs and a router — a small linear layer producing one score per expert — that sends each token to the top k of them. Routing is per token and per layer, so a token can use different experts at every depth.

Start with the reconciliation everyone skips. Mixtral-8x7B is not 56B parameters, and it does not activate 14B. Only the MLP is replicated. Attention, embeddings and norms are shared by all experts:

per layer, attention (GQA-8, d_model 4096):        41,943,040
per layer, one expert (SwiGLU, ffn 14336):        176,160,768
per layer, eight experts:                       1,409,286,144
per layer, total:                               1,451,229,184
x 32 layers:                                   46,439,333,888
+ embedding and LM head (32000 x 4096, x2):       262,144,000
                                               --------------
                                               46,701,477,888   = 46.7B, the published figure

active per token:  41,943,040 + 2 x 176,160,768 = 394,264,576
x 32 layers + LM head:                           12,878,610,432   = 12.9B

The activation ratio — active over total — is the number to read off a model card. Mixtral is 12.9/46.7 = 28%. Qwen3-235B-A22B is 22/235 = 9%, with 128 experts and top-8 and no shared expert at all. DeepSeek-V3 is 37/671 = 5.5%, with 256 routed experts plus one shared expert and top-8 routing, the first three layers left dense. The trend is unmistakable: more experts, each smaller, a smaller fraction active. That is DeepSeekMoE's contribution — fine-grained expert segmentation, so routing decisions are finer, plus an isolated shared expert that every token reads and that therefore absorbs the knowledge which would otherwise be duplicated into all 256.

Routing details worth knowing because they change behaviour under load: Switch Transformer routes to exactly one expert, GShard to at most two, and DeepSeek-V3 replaced softmax gating and auxiliary balancing losses with sigmoid affinities and a bias term adjusted during training to keep experts balanced without a loss that fights the objective.

Now the part that the model card does not tell you. The activation ratio is a per-token number, and a batch does not read one token's experts — it reads the union of the batch's experts. Under uniform routing, a batch of B tokens touches

E x (1 - (1 - k/E)^B)   distinct experts

For Mixtral, that is 2 experts at batch 1, 7.2 at batch 8, and all 8 by batch 32. For DeepSeek-V3 it is 8 at batch 1, 163 at batch 32, and 252 of 256 by batch 128. The consequence, stated as carefully as it deserves:

• Bytes per step rise with batch for an MoE and are flat for a dense model. At a large enough

batch the step reads every expert, so it moves the bytes of its total size, not its active size.

• Bytes per token still fall, because those bytes are amortized over B tokens. MoE does not

stop working at scale.

• What erodes is the latency advantage. An MoE at batch 1 decodes like a small model. The same

model at batch 512 has a step time set by its full weight footprint, and what it keeps is the FLOP saving — which matters on the other side of the ridge point, where Module 4 says you are compute-bound anyway.

So the honest summary is that MoE is a bandwidth optimization at small batch and an arithmetic optimization at large batch, and it is worth knowing which one you are buying. Module 9's note that larger batches smooth routing imbalance is the same phenomenon seen from the scheduler's side: by the time the batch is big enough to balance, it is big enough to touch everything.

One last practical fact. You can convert a dense checkpoint into an MoE — sparse upcycling copies the dense MLP into every expert, initialises a fresh router, and continues training, at roughly half the sunk cost of the original pretraining. It is the only entry in this level that is even partly retrofittable, and "half of a pretraining run" is not a deployment-day decision.

The cache term has one more factor that no attention variant touches: S, the number of tokens you keep. Bound it and the cache stops growing.

Sliding-window attention lets each position attend only to the last w tokens. Mistral 7B shipped with w = 4096 inside an 8192-token context; later Mistral models dropped it and Ministral reintroduced it in interleaved form. The immediate objection — that the model then cannot use anything older than the window — is wrong, and worth working through. Attention composes across depth: a token at layer 2 attends to positions that themselves attended w tokens further back, so the receptive field grows roughly as layers × w. What a window costs is not connectivity but exact recall, because information from far away arrives blurred through many layers rather than retrieved directly. That is the same trade Module 7 describes for state space models, applied per layer instead of per model.

Local-global interleaving buys the exact recall back cheaply. Make most layers local and a few global. Gemma 2 alternated 1:1 with a 4096-token window. Gemma 3 moved to 5:1 with a 1024-token window, and only the global layers carry the long-context RoPE scaling. Character.AI's production stack used the same 5:1 shape with a 1024 horizon.

The arithmetic is the reason. Take 48 layers, GQA-8, head_dim 128, fp16 — 4 KiB per token per layer — and a 32k context:

all-global:      48 layers x 32768 tokens x 4 KiB        =  6.0 GiB
5:1 local-global: 8 global x 32768 x 4 KiB               =  1.0 GiB
                 40 local  x  1024 x 4 KiB               =  0.16 GiB
                                                            ---------
                                                            1.15 GiB   5.2x smaller

Note where the saving comes from and where it stops. The local layers are already saturated at 1024 tokens, so their contribution is constant from 1k of context onward — doubling the context doubles only the global part. As S grows the ratio approaches L / (L/6) = 6×, and it gets there quickly. This is the cheapest long-context lever there is, which is why every recent model family has adopted some version of it.

Two things to watch when serving one. Prefix caching (Module 6) has to know the window, because a cached block older than w is useful to the global layers and dead weight for the local ones — engines that page uniformly across layers will happily store both. And the attention-sink result from Module 6 is the reason naive windowing was worse than it should have been: keep the first few tokens pinned alongside the window and quality recovers, because those positions absorb attention mass regardless of their content.

The last move is to delete the cache. A state space model or linear-attention layer carries a fixed-size recurrent state instead: every token updates it, and nothing accumulates. Module 7 covers the mechanism and the reason pure versions lost — a fixed state cannot hold an arbitrary amount of detail, so exact recall degrades in a way that is information-theoretic rather than an implementation shortfall. Here the question is only what it costs to serve.

For Mamba-2 the state is d_inner × d_state values per layer, plus a small convolution cache a few timesteps wide. Take the 2.7B checkpoint — d_model 2560, expand 2, d_state 128, 64 layers:

per layer:   5120 x 128            =    655,360 values
x 64 layers:                       = 41,943,040 values
at bf16:                           =      83.9 MB per sequence

83.9 MB at 1k tokens. 83.9 MB at 1M tokens. The state does not know how long the sequence is. Compare that to a KV cache, which at 128k context and GQA-8 is measured in gigabytes per sequence, and the shape of the trade is clear: a recurrent model's memory scales with concurrency only, where a transformer's scales with concurrency times context.

Nobody ships the pure version. Hybrids interleave a few full-attention layers among many recurrent ones, which is enough to restore exact retrieval while keeping most of the memory saving. The published ratios vary more than the folklore suggests:

Jamba's own table is the cleanest statement of the payoff: at 256k context with a 16-bit cache it needs about 4 GB, against 32 GB for Mixtral and 128 GB for Llama-2-70B at the same context. That is the entire argument for hybrids in one row.

What it does to a serving stack is less advertised. You now have two allocators: a paged KV cache for the attention layers, and a constant-size state block per sequence for the rest. The state is allocated when the sequence starts and freed when it ends, which makes admission control simpler and preemption harder — you cannot evict half a state the way you can drop and recompute KV blocks. Prefix caching works only on the attention layers unless the engine also snapshots the recurrent state at block boundaries, and a snapshot costs the same fixed bytes whether the prefix is 100 tokens or 100,000.

Three parts of a config file have nothing to do with the interesting architecture debates and quite a lot to do with your latency.

Depth. Under tensor parallelism a decode step costs two all-reduces per layer, one after attention and one after the MLP, so an 80-layer model pays 160 collectives per token. Those collectives carry small messages, so their cost is mostly fixed latency that does not shrink when you add GPUs — Module 9 measures it. The same is true of kernel launches, though CUDA graphs remove most of that particular tax. A deep-narrow model and a shallow-wide model with equal parameter counts do not decode equally, and the deep one loses. It often wins on quality per parameter, which is why the trade keeps being made.

Head counts. Tensor parallelism splits attention by heads, so n_heads must divide by the TP degree. The one that surprises people is the KV side. When the TP degree exceeds n_kv_heads, engines do not fail — they replicate. A GQA-8 model at TP=16 gives every rank one KV head, held twice across the group, so your KV cache costs 2× what the formula says. The cache arithmetic from Module 2 silently stops being right at the moment you scale past 8 ways, and nothing in the logs tells you.

Vocabulary. The LM head is a d_model × vocab matrix and it is read on every decode step. It has grown: GPT-2 at 50,257, Llama-2 at 32,000, Llama-3 at 128,256, Qwen3 at 151,936, Gemma 3 at 262,208. On Llama-3-8B that is 4096 × 128256 × 2 = 1.05 GB of the 15.0 GB read per token — 7%, noticeable but not decisive. On a 0.6B model with a 151,936-token vocabulary it is 311 MB of about 1.2 GB, a quarter of the entire decode cost spent on the output projection. This is why small models tie their embeddings: Gemma ties at every size, Qwen3 ties at 4B and below, and Llama does not tie at all. Tying saves one vocab × d_model matrix, which is rounding error on a 70B model and a design constraint on a 600M one.

The general lesson is that the config fields nobody argues about on release day are the ones that decide whether the model shards cleanly, and a model that does not shard cleanly is a model you serve at 2× the cache cost forever.

A vision-language model is usually three parts: a vision encoder (a ViT such as CLIP or SigLIP), a projector that maps its output into the language model's embedding space, and the language model itself. LLaVA-1.5 uses CLIP ViT-L/14 at 336px with a two-layer MLP projector; PaliGemma pairs SigLIP with Gemma; Qwen2-VL uses a ViT with an MLP token merger.

The arithmetic that matters is how many positions an image becomes. A ViT at 336px with patch size 14 produces (336/14)² = 24 × 24 = 576 patches, and LLaVA-1.5 passes all of them through, so one image is 576 tokens. Higher-resolution schemes tile: LLaVA-NeXT's grid reaches 2,880 tokens per image, and Qwen2-VL's dynamic resolution maps any input size to a variable count bounded in the thousands.

Then the point of this level: those are ordinary positions. They cost the same prefill FLOPs per token, and the same cache bytes per token, as text. On a Llama-3-8B-shaped model at 128 KiB per token:

one 576-token image:    576 x 131072   =  72 MiB of KV
one 2880-token image:  2880 x 131072   = 360 MiB of KV
a 40-token text prompt:   40 x 131072  = 5.0 MiB

A single image is worth fourteen to seventy text prompts. So in an image-heavy request with a short question, prefill is dominated by pixels, the cache is dominated by pixels, and every scheduling decision from Module 5 — chunked prefill, admission control, the prefill/decode interference trade — is being made about image tokens whether or not the scheduler knows it.

The exception is the cross-attention family: Flamingo and Llama 3.2 Vision feed image features into the language model through dedicated cross-attention layers instead of inserting them into the sequence. Image content then never enters the self-attention cache at all; it adds a separate, fixed-size K and V, exactly the encoder-decoder structure from earlier in this level. It costs extra parameters and buys a cache that does not grow with image count — the same trade, made in a different place, for the third time in this module.

Two practical notes. The vision encoder is a second model resident on the same GPU, with its own weights and its own batch behaviour, and it runs in the prefill phase only. And prefix caching over images requires hashing the image content rather than the token ids, which most engines now do — worth checking, because in a multi-turn conversation about one image the image is precisely the part that repeats.