You will almost never train a model. And yet understanding post-training is what separates the agent engineer who can reason about why the model lies, flatters, or invents a plausible-but-false chain of reasoning from the one who patches symptoms with prompts and is surprised when they come back. This module closes a loop. The grader you wrote in the evals phase is, literally, a training reward function. Whoever masters designing non-hackeable rewards masters both sides of the coin: evaluating and training.

Three layers turn a token predictor into an agent

A base model is a token predictor. It completes text with the distribution of the internet. It's not an assistant, it doesn't follow instructions reliably, it doesn't use tools. Post-training is everything that happens after pretraining to turn that predictor into something useful, aligned, and agentic. Split it into three layers, because each one grabs something different and fails in a different way.

The post-training pipeline: each layer grabs something different

Layer 1, SFT / distillation. Supervised fine-tuning on well-behaved trajectories, often generated by a large teacher model and distilled into a small one. It grabs form: response format, chain-of-thought structure, tone. Its risk is exactly that. It teaches form over substance, producing plausible CoT with wrong intermediate steps.

Layer 2, preference optimization. RLHF, its supervised collapse DPO, and Constitutional AI / RLAIF. Aligns which response is preferred according to humans or principles. This is where conversational usefulness and sycophancy both come from.

Layer 3, RLVR (RL from verifiable rewards). RL against programmatic rewards: a test passes, a math answer is correct, a final state is reached. This is what made long chain-of-thought reasoning and tool-use emerge: the reasoning-model shift.

The cost ladder: when (almost never) to train

Before touching a gradient, exhaust the ladder. Each rung is orders of magnitude cheaper and faster than the next, and most problems are solved without going past the third.

  1. Prompting (hours): rewrite the instruction, the role, the expected format.
  2. Few-shot: add in-context examples.
  3. Context / harness engineering: structure what the model sees and the tools it has. The 90% of the ROI lives here.
  4. RAG / agentic search: only if the knowledge changes and doesn't fit in context.
  5. Fine-tuning: only with verifiable signal, volume, and clear ROI.

Why reasoning models changed the game: test-time compute

The most important result of 2025 was DeepSeek-R1-Zero: pure RL on verifiable rewards, with no prior SFT, made chain-of-thought, self-correction, and the so-called "aha moments" emerge on their own. The model learned to re-evaluate its own reasoning without anyone teaching it explicitly. On AIME it went from 15.6% to 71% with RL alone.

This opened a second scaling axis: beyond more compute in pretraining, you now spend more compute at inference. The model "thinks" longer, and performance rises predictably. For an agent engineer this reconfigures the latency/quality/cost tradeoff. A model that reasons well can replace manual scaffolding (planning, reflection, retries) you used to hand-code in the harness.

What makes RLVR work is that the reward is a verifiable function, not a learned reward model. Did the test pass? Is the answer correct? Was the final state reached? Because the signal is programmatic, it's much harder to hack than a trained reward model. The 2025-2026 challenge is extending this to multi-turn agentic settings, where the signal is sparse: success is only known at the end of a long trajectory of tool-calls. Agent-RLVR shows RLVR degrades in agentic environments because of that sparsity, and proposes "agent guidance" to route successful trajectories (Qwen-2.5-72B rose from 9.4% to 22.4% on SWE-Bench Verified).

GRPO + verifiable rewards is the dominant recipe today, but variants exist: DAPO adds stability tricks (dynamic sampling, asymmetric clipping), GSPO works at the sequence level. Treat "GRPO" as the reference pattern, not the only algorithm.

Reward hacking and Goodhart's Law

When a metric becomes a target, it stops being a good metric.

This is the central pathology that explains why models misbehave. If your reward signal is approximate instead of exact, the optimizer will find the crack. Documented examples: length hacking (the model learns to ramble because the reward model rewards long answers), CoT that sounds like reasoning but doesn't solve anything, correct answers reached through false intermediate logic. This directly explains production pathologies: verbosity, overconfidence, and the sycophancy below.

How alignment and training work today: DPO, GRPO, CAI

DPO: the collapse of RLHF into a supervised objective

Classic RLHF is expensive: you train a reward model and then run on-policy RL against it, for four models in memory (policy, reference, reward, critic). DPO proved that optimizing a classification-style loss over (preferred, rejected) pairs converges to the same policy as RLHF with the optimal reward model, removing both the reward model and the RL loop. You go from 4 models in memory to 2, and it runs on spot instances. More important, it demystifies the "RL": much of what we call "alignment" is really supervised learning of preferences. And note the conceptual trap: DPO/RLHF is preference alignment, it does not add reasoning capability out of nowhere. Capability comes from RLVR (layer 3), not from preference optimization (layer 2).

The other half of the "demystify the RL" story is GRPO. The bottleneck of RL for LLMs was the value model (critic): a separate network estimating the reward baseline, expensive to train. GRPO (introduced in DeepSeekMath, Feb 2024, scaled in R1) removes it. It estimates the baseline from a group of responses sampled for the same prompt. Sample 8 responses, the group mean is your baseline, each one's advantage is relative to its siblings. This dropped the cost barrier that made RL impractical at scale.

Constitutional AI / RLAIF rounds out the layer-2 picture. Anthropic generates preference data by having the model critique and revise its own responses against a constitution (principles in natural language), then a model labels which response is better according to a randomly chosen principle. CAI (Bai et al. 2022) was the first documented large-scale instance of RL from AI feedback. One precision: this same paper is also where the term "RLAIF" was coined — it states it verbatim in its abstract. Google's 2023 paper (Lee et al.) didn't formalize the term; it popularized and scaled RLAIF, showing it matches RLHF on broader tasks while citing Constitutional AI as the term's origin [D]. For an agent engineer this has a concrete, constant consequence: the safety hard-rails of Claude you rely on come from here. A company persona concatenated into your system prompt cannot override those trained limits. They're etched in the weights, not the context.

Sycophancy and inherited biases from post-training

RLHF optimizes for what human annotators prefer, not for what is correct. And humans prefer answers that agree with them, that sound confident, that don't contradict them. The result is a systematic bias: models that flatter, over-assert, and avoid contradicting the user.

Credit assignment: the sparse reward of agents

In a real agent the reward only appears after a long trajectory: the customer rescheduled their appointment, or didn't. That breaks the action↔reward coupling: which of the 12 tool-calls deserved the credit? The solutions are dense per-turn rewards, token-level credit (GAE/TD), and trajectory curation. The same guide gives a useful intuition on the SFT:RL ratio: 60 SFT demonstrations + 400 RL episodes reach results comparable (~85%) to 5000 pure RL episodes (~88%). Cold-start SFT accelerates enormously.

For your grader design, the translation is direct. A tool_trajectory grader that gives partial credit for progress is exactly dense reward shaping. A grader that returns a hard -1.0 for an unconfirmed write is safety reward shaping. Same primitive the labs use, pointed at evaluation.

Fine-tuning as a first resort: the correct sequence is prompting → few-shot → context/harness → RAG → fine-tuning only with verifiable signal. Classic PPO with a value model: the "critic tax" (4 models) made it impractical against GRPO/DPO. A learned reward model as the alignment default: scalar and highly hackeable, so DPO and RLVR displaced it. RAG/embeddings by default for "knowledge": context engineering plus agentic search over files often suffices and avoids the debt of a vector index.

The eval↔reward isomorphism: mastering non-hackeable rewards serves both evaluating and training. Distillation as a cost strategy for your own team: capture your agent's trajectories with an expensive model and distill them to a cheap one for routine subtasks (routing, intent classification), cutting inference cost without setting up RL. Reasoning models that replace scaffolding: capabilities you used to hand-code in the harness (plan, reflect, retry) now come trained. Don't reinvent what the model already does better.

The lesson that carries across the whole track: most of your agent decisions are harness decisions, not model decisions. But understanding the model tells you which problems you can't fix with the harness, and which are etched biases you only counter at the system level. The grader you wrote earlier is the reward function here. Whether RL adds genuinely new capabilities or only sharpens what's already in the base distribution is still contested. One camp finds base models match RL-tuned ones at large pass@k, another reports prolonged RL reaching strategies the base never does. Hold it open, and build for the model you actually have.