FINE-TUNING

Pretraining optimizes exactly one thing: predict the next token, over trillions of tokens of ordinary text. Nothing in that objective knows about "questions" or "answers" or "being helpful." Hand a raw base model the prompt How do I boil an egg? and it is just as likely to continue with How long does it take? What about a soft-boiled one? — more plausible internet text — as it is to answer you. The model is not being difficult. It is doing exactly what it was trained to do: continue the distribution it saw.

Supervised fine-tuning (SFT) — what most people mean when they say "fine-tuning" without qualification — closes that gap with no architecture change at all. Same transformer, same next-token cross-entropy loss, same optimizer. The only thing that changes is the data: instead of "the internet," the model trains on curated (instruction, response) pairs, wrapped in a chat template that marks who is speaking. A few thousand to a few hundred thousand well-written examples are usually enough to shift a model from "completes anything" to "answers you" — SFT is teaching a format and a role, which turns out to need far less data than teaching new knowledge does (a distinction the pitfalls below return to).

One detail that trips people up the first time they write an SFT loop by hand: you do not compute loss on the whole sequence. The prompt tokens are masked out — the model is not being trained to predict your question, only the assistant's response to it. Get this wrong and the model partially learns to imitate users instead of answering them, and burns capacity re-deriving prompts it will never need to generate at inference time.

Base model and instruction-tuned model are usually the exact same architecture at the exact same size; Llama-3-8B and Llama-3-8B-Instruct differ only in what was done to the weights after pretraining finished. Everything else in this module is either a cheaper way to do SFT, or a second stage that runs after it.

Fine-tuning the entire model looks simple: unfreeze every parameter, run SFT (or any of the objectives later in this module), backprop, step the optimizer. The reason it is not the default is memory, and the amount is easy to underestimate if you only count the weights.

Training almost universally uses mixed-precision Adam: forward and backward passes run in fp16/bf16 for speed, but a full-precision "master copy" of the weights is kept in fp32 so tiny gradient updates do not get rounded away, and Adam's own running statistics are kept in fp32 too. Per trainable parameter:

2 bytes   fp16/bf16 weight    (used in the forward/backward matmuls)
2 bytes   fp16/bf16 gradient
4 bytes   fp32 master weight  (what the optimizer actually updates)
4 bytes   fp32 Adam moment 1  (m, the running mean of the gradient)
4 bytes   fp32 Adam moment 2  (v, the running mean of the squared gradient)
--------
16 bytes  per parameter, before a single activation is stored

For Llama-3-8B that is 8.03e9 x 16 = 128.5 GB — before you have loaded a single training example, and on a single 80 GB H100 or A100 you are already out of memory. This is not an inefficiency to be optimized away; it is the honest cost of Adam remembering enough about every parameter's gradient history to take a stable step. ZeRO/FSDP-style sharding spreads this 16 bytes/parameter across many GPUs rather than reducing it — the same "spread the state, don't shrink it" idea Module 9 used for inference-time tensor parallelism, applied to training instead.

Multiply by 70B and the picture gets uglier before it gets better — the math lab works both scales by hand. Every method in the rest of this module is, one way or another, an answer to this number: LoRA answers it by making almost all of a model's parameters permanently ineligible for gradients or optimizer state; QLoRA answers it again by shrinking what is left; DPO and GRPO answer a related but distinct memory bill, the cost of keeping several models resident at once for alignment.

Whatever fine-tuning teaches a weight matrix W0 [d_out x d_in], the update delta W it learns is itself a matrix of that same shape. LoRA's premise — argued empirically in the original paper on GPT-3 and RoBERTa — is that delta W during task adaptation does not need the full d_out x d_in degrees of freedom; it has low intrinsic rank. So instead of learning delta W directly, parameterize it as a product of two much smaller matrices:

delta W  =  (alpha / r) * B @ A

  A : [r, d_in]      B : [d_out, r]      r << min(d_out, d_in)
  W0 stays frozen — no gradient, no optimizer state
  only A and B are trained

The forward pass becomes h = W0 x + (alpha/r) B(Ax) — one extra small matmul through the r-dimensional bottleneck, added to the frozen path. A is initialized to small random values and B to exactly zero, so at training step zero the LoRA path contributes nothing at all and the model starts training from the pretrained model's exact behavior — a nice property full fine-tuning does not give you for free. alpha is a fixed scaling constant that keeps the update's effective magnitude roughly stable as you change r, so you are not re-tuning the learning rate every time you try a different rank; alpha = 2r is the common starting point, and pushing it much higher tends to hurt rather than help.

Which weight matrices get a LoRA adapter is a real design knob. The original paper found adapting only W_q and W_v in attention sufficient at GPT-3 scale; most current recipes adapt all four attention projections (q, k, v, o) and often the MLP's gate/up/down projections too, trading more trainable parameters for a smaller quality gap to full fine-tuning. On a GQA model like Llama-3-8B (Module 11), k_proj and v_proj are already narrower than q_proj/o_proj — the math lab uses exactly this to compute a real trainable-parameter count.

LoRA stops the optimizer from paying for the base model's weights, but the base weights still have to sit in GPU memory in some precision so the forward pass can read them. QLoRA's move is to apply Module 6's quantization idea at fine-tuning time rather than only at serving time: store the frozen base model in 4-bit NF4 (4-bit NormalFloat) instead of fp16 or bf16.

NF4 is not a generic 4-bit integer format. Its quantization bins are spaced to be information-theoretically optimal for weights that are roughly zero-centered and Gaussian — which pretrained transformer weights empirically are, once you normalize per small block — rather than the uniform bins a plain INT4 scheme would use. Crucially, compute does not happen in 4-bit: every matmul dequantizes the relevant weight block to bf16 on the fly, multiplies, and discards the bf16 copy. Storage shrinks four-fold; the arithmetic itself still runs at bf16 precision throughout training. Two more tricks round out the recipe: double quantization, which quantizes the per-block scale factors themselves, saving a further fraction of a bit per parameter; and paged optimizers, which use unified GPU/CPU memory so an activation-memory spike (gradient checkpointing, in particular, causes these) pages optimizer state out to host RAM instead of crashing the run.

The headline result — fine-tuning a 65B-parameter model on a single 48 GB GPU while matching 16-bit full fine-tuning quality — is the demonstration that these three tricks compose cleanly with LoRA's already-small optimizer footprint. The honest cost is speed, not just an efficiency footnote: independent benchmarks report QLoRA runs on the order of 30-40% slower in wall-clock time than plain bf16 LoRA at comparable memory-unconstrained settings, because every forward and backward pass now pays for a dequantize step the bf16 version does not. QLoRA buys memory, not speed — you reach for it specifically when the model would not fit any other way.

LoRA is the dominant parameter-efficient fine-tuning (PEFT) method today, but it was not the first, and it is worth knowing what it beat and why.

Adapters (Houlsby et al., 2019) insert a small bottleneck feed-forward block — down-project to a few dozen dimensions, a nonlinearity, up-project back — inline, after the attention and FFN sublayers, with everything else frozen. On GLUE, adapters land within 0.4% of full fine-tuning while training only about 3.6% of the parameters per task, robust across a wide range of bottleneck sizes. The catch is architectural: an adapter sits in the middle of the residual stream with a nonlinearity inside it, so there is no algebraic move that folds it back into the surrounding frozen weights. Every inference request pays its extra forward-pass compute, forever.

Prefix-tuning and prompt-tuning (2021) don't touch any weight at all. They prepend a short sequence of trainable "virtual token" embeddings — a soft prompt — to the input, at every layer (prefix-tuning) or just the embedding layer (prompt-tuning), and train only those vectors. This ties directly back to Module 2: those virtual tokens occupy real sequence positions, so they consume context-window budget and KV-cache memory on every single request, exactly as if the prompt were physically longer. Cheap to train, but not free at serving time either — just charged in a different currency than adapters.

BitFit trains only the bias terms — tiny parameter count, correspondingly weaker quality, rarely the first choice today. (IA)^3 learns per-channel rescaling vectors that multiply key, value, and hidden activations rather than adding a low-rank detour; fewer parameters than LoRA in many configurations, and in several cases those vectors can also be absorbed into an adjacent weight matrix.

DoRA — Weight-Decomposed LoRA (Liu et al., 2024) is the modern refinement most current recipes have converged toward. It decomposes each frozen weight into a magnitude vector and a direction matrix (W0 = m * V / ||V||), trains a standard LoRA update on the direction and the magnitude vector directly, and — crucially — still merges back into a single weight matrix for zero added latency, the same property LoRA has. Separating "how far" from "which way" lets DoRA's updates resemble full fine-tuning's more closely than plain LoRA's additive delta can, at the same rank; it beats vanilla LoRA by a small, consistent margin on standard benchmarks and is close to a drop-in swap in current PEFT libraries.

The dividing line, restated: anything that is additive to an existing weight matrix — LoRA, DoRA — produces a delta of the same shape as the weight it modifies, so W0 + delta W is again just a matrix, substitutable with zero change to the computation graph. Anything that adds new layers (adapters) or new sequence positions (soft prompts) changes the graph itself, and no amount of algebra removes that cost at inference time.

SFT teaches a model to imitate the responses it is shown. It does not teach the model to prefer a subtly better response over a subtly worse one written in a similar style — that is a comparative signal, and getting a human to say "A is better than B" is far cheaper and more reliable than getting them to write a gold response from scratch. That is what preference data is, and aligning a model to it is the second post-training stage, after SFT.

RLHF, the recipe InstructGPT made standard, runs in three stages. Collect pairs of model outputs and human preferences between them. Train a reward model r_phi(x, y) on those pairs with a Bradley-Terry pairwise loss — maximize the log-sigmoid gap between the chosen and rejected response's scores, the same functional form behind Elo ratings. Then run PPO: treat generation as an RL episode, reward each rollout with r_phi minus a KL penalty beta * KL(pi_theta || pi_ref) that stops the policy drifting so far from the SFT model that it degenerates into reward-hacking the reward model rather than actually improving. The engineering cost is real: PPO needs four models resident simultaneously — the policy being trained, a frozen reference policy for the KL term, the frozen reward model scoring rollouts, and a value/critic network estimating a baseline for the advantage — and the RL loop itself is notoriously fiddly to keep stable.

DPO (Rafailov et al., 2023) starts from an observation about that same KL-regularized objective: for a fixed reward function, the optimal policy has a closed form in terms of the reward and the reference policy. Substitute that relationship into the Bradley-Terry preference loss, and the reward model's normalizing term cancels out algebraically. What is left is a loss computable directly from the policy's own log-probabilities of the chosen and rejected response, compared against the frozen reference's log-probabilities of the same two responses:

loss = -log sigmoid( beta * [ (logpi_theta(y_w|x) - logpi_ref(y_w|x))
                             - (logpi_theta(y_l|x) - logpi_ref(y_l|x)) ] )

  y_w = preferred ("winner") response,  y_l = rejected ("loser") response

Two forward passes per preference pair, no sampling, no reward model, no critic, no RL loop — just a supervised classification loss over data you already collected. Only two models are resident: the policy and the frozen reference. That simplicity is DPO's entire appeal, and it is why most open-source alignment recipes today skip PPO.

The tradeoff is that DPO optimizes entirely against a fixed, already-collected preference dataset — there is no rollout step where it can score fresh, on-policy samples the way an RL loop conceptually can. Several studies since have found DPO trailing a well-tuned reward-model-plus-PPO pipeline in ceiling quality once enough compute and data go into the RL side, which is one reason interest in RL-based alignment did not disappear — it fed directly into the reasoning-focused methods in the next concept. A family of DPO variants has grown up around trimming its remaining rough edges: KTO learns from unpaired desirable/undesirable labels rather than requiring matched pairs; ORPO folds preference optimization into the SFT stage itself, needing no separate reference model at all; SimPO drops the reference model and uses a length-normalized log-probability directly as the implicit reward.

RLHF and DPO both depend on a learned proxy for quality — a reward model, or the implicit reward baked into DPO's loss — because "is this response good" is usually subjective. For a narrower but very important class of tasks it is not subjective at all: does the final boxed answer match the known solution, does the generated code pass its test suite, does a symbolic verifier accept a proof step. RLVR — reinforcement learning with verifiable rewards — uses a deterministic checker instead of a learned reward model wherever one is available: no reward-model training, no reward-hacking-the-reward-model failure mode, and a signal that scales with however many problems you have known answers to.

GRPO (Group Relative Policy Optimization, from DeepSeekMath, and the algorithm behind DeepSeek-R1) is the policy-gradient method built for this setting. It keeps PPO's clipped policy-gradient update but removes the critic network entirely. Instead of training a value function to estimate a baseline, GRPO samples a group of G completions for the same prompt from the current policy, scores each one (with the verifiable reward above, or a reward model — the group-relative trick is separable from RLVR specifically), and sets each completion's advantage as its own reward normalized against the group's mean and standard deviation:

advantage_i  =  (reward_i - mean(reward_1 .. reward_G)) / std(reward_1 .. reward_G)

That is a Monte-Carlo baseline computed for free from rollouts you were generating anyway, so an entire model's worth of memory and compute — the critic — disappears relative to PPO. A frozen reference policy is still kept for a KL penalty, so GRPO needs two models resident, like DPO, not four like full PPO. What it needs instead is samples: G full completions per prompt, every single training step, before a gradient exists to take.

That is where this module closes the loop with the rest of the book. Generating G completions for every prompt in a batch is G x batch_size full decode sequences — at G = 64 and a batch of 256 prompts, that is 16,384 sequences to decode before one gradient step happens. That workload is exactly the large-batch, memory-bound decode this entire book has been about optimizing, and it now sits on the training throughput critical path rather than a user's. It is why modern RL post-training frameworks embed a real serving engine — frequently vLLM itself — as their rollout generator instead of a naive generation loop: the training loop's steps-per-hour ceiling is, quite literally, the inference engine's decode-throughput ceiling from Module 1, Module 5, and Module 7 combined. One more data point worth knowing: DeepSeek's R1-Zero variant skipped the SFT warm-start stage entirely, running GRPO directly against a base model, and still reached strong reasoning behavior — evidence that for verifiable domains, the RL signal alone can carry more of the weight than the earlier RLHF-era pipeline assumed.