Before you write a single line of your harness, you have to know what the engine actually is. Strip away the marketing and an LLM does three plain things, and only three. It guesses the next word, it remembers nothing on its own, and it can only look at a limited amount of text at once. Not "an assistant with memory." Not "a reasoning machine." Get those three facts in your bones and you stop fighting the model. You start building the right scaffolding around it.
Each plain fact has a proper name you'll meet everywhere. Guessing the next word is being The output isn't fixed: run the model twice on the same input and you can get different answers, because each next token is sampled at random from a probability distribution. This is why one passing run proves little and you measure reliability across many (pass^k), not one. glossary → (the output is sampled from a probability distribution, so the same input can give different answers). Remembering nothing is being The model retains nothing between API calls; a 'conversation' only exists because you resend the prior history on every request. glossary → (it retains nothing between API calls). And the limited view is a finite The finite maximum number of tokens (prompt plus output) the model can attend to in a single call; beyond it, content must be dropped, summarized, or retrieved. glossary → . Almost everything you'll build as an agent engineer (sessions, memory, human-in-the-loop, evals, token budgets) exists to compensate for one of those three.
You don't need to implement attention, gradients, or a tokenizer to ship a good agent. What decides whether your agent survives production is a causal model of why the model fails ( The arithmetic errors that come from how a model chops numbers into tokens: byte-pair encoding can group digits inconsistently (e.g. '1234' as '123'+'4'), so the model never sees clean place value and miscalculates. It's a tokenization artifact, not a reasoning failure. glossary → from BPE, recency, sampling, refusals) so you can design the harness that absorbs it.
Three properties, and what each one forces
The useful move is translation: when the agent misbehaves, don't ask "why is the model dumb?" Ask "which of the three properties am I ignoring?"
Forgot a fact from turn 3? Statelessness, mishandled. Added two numbers wrong? Tokenization. Passed the eval yesterday and fails today? Stochasticity. That symptom-to-cause reflex is the skill that separates an agent engineer from a chatbot user.
The analogy I keep coming back to: the LLM is a CPU with no RAM and no disk. Ferociously fast per cycle, but amnesiac. Your harness is the memory, the storage, and the clock that turn it into a usable computer.
Context is an attention budget, not a bucket
This is the highest-impact idea in this course. Context is not storage. It's a finite attention budget you spend every turn.
Transformer attention relates every pair of tokens, so the compute is O(n²): double the context and you roughly quadruple the prefill cost. That's the physical, non-negotiable root of the token budget.
Worse, recall degrades long before you hit the nominal limit. The documented shape is the U-curve, "lost in the middle." A model recalls what sits at the start and end of context well, and loses what's in the middle.
The beginner stuffs the whole history in because "it fits in 1M." The expert knows every irrelevant token dilutes the signal and quadruples cost. More context is not more intelligence. The right technique is just-in-time retrieval: keep lightweight identifiers (paths, queries) and pull data on demand with tools, instead of dumping everything up front.
The conversation is an illusion you rebuild
The model remembers nothing between requests. The API is stateless. A "conversation" is something you fabricate by resending the history on every call. That's why persistence, sessions, and resume aren't implementation details. They're the central architecture that stands in for the memory the model doesn't have.
In-context learning comes from induction heads, not from "understanding"
Here's the most useful demystification of prompting there is. The ability to learn from examples in the prompt (few-shot, following a format, copying a tool-call pattern) does not come from the model "understanding your instructions." It comes from a concrete mechanism: A two-attention-head circuit that detects a prior 'A→B' pattern and predicts B after seeing A again — the mechanistic engine of in-context (few-shot) learning. glossary → , pairs of attention heads that detect "I saw A→B earlier; now I see A, so I predict B." This circuit shows up in an abrupt phase change during training, and it's the engine of in-context learning.
The design implication is direct: canonical examples work because they activate a learned copy-and-generalize mechanism, not because the model reasons over your rules.
Why this changes how you write the system prompt
If the model generalizes by pattern-copy, two antipatterns get obvious. First, a list of 30 near-identical examples teaches the circuit nothing new; it just burns attention budget. Second, one badly-formatted example teaches the bad format, because the induction head copies exactly what you showed it. The quality and diversity of 3-5 canonical examples beats raw quantity every time.
Tool-use is trained, not prompted
A model deciding when to call a tool, chaining several, and knowing when to stop is not prompt magic. It's a behavior the model was taught by trial and error: shown good examples ( Training a base model on curated input-output examples (often distilled from a larger teacher) to teach response format, tone, and chain-of-thought structure — it grabs form, not new factual knowledge. glossary → on trajectories) and rewarded when its answers actually checked out (RL with a reward at the end of the trajectory). The family of techniques that does this is the same one used to train reasoning: An RL algorithm (DeepSeekMath, 2024) that drops the value/critic network by sampling a group of responses per prompt and using the group's mean reward as the baseline for each response's advantage. glossary → with verifiable rewards ( RL where the reward is a programmatic, checkable function — did the test pass, is the math correct, was the goal state reached — making it much harder to game than a learned reward model and shaping reasoning and tool-use. glossary → ). The clearest public reference for that recipe is DeepSeek-R1 (2025), which used GRPO + RLVR to induce verifiable-answer chain-of-thought, not tool-use: R1 shipped without native function-calling. So read R1 as the reference for the training method, not as a tool-trained model.
Two practical consequences fall out of this. First, the model carries strong priors about what well-designed tools look like (clear schemas, descriptive names, prescriptive descriptions) because that's what it was trained against. Second, the agent's reliability frontier is the distribution of tasks it was trained on, which is why you evaluate by trajectory, not only by final answer.
Open question: does RL add new skills, or just sharpen old ones?
A live debate worth flagging: whether RL only sharpens capabilities already in the base model's distribution, or adds genuinely new ones. One camp (Yue et al., "Limit of RLVR") finds base models match RL-tuned ones at large pass@k is the probability that at least one of k attempts succeeds (a capability/best-case measure); pass^k is the probability that all k independent attempts succeed (reliability, ≈p^k), and production cares about pass^k. glossary → , suggesting RL just samples existing solutions more efficiently. Another (ProRL) reports prolonged RL uncovering strategies the base model never reaches. Treat it as contested, not settled.
BPE decides what the model sees, and where it breaks
The model doesn't see characters or numbers. It sees The model reads and writes in subword chunks ('tokens') produced by byte-pair encoding rather than characters or whole words, which is why digit grouping breaks arithmetic and exact character counting. glossary → . That causes systematic, predictable failures. Arithmetic breaks because digits get grouped into tokens arbitrarily. Character counting breaks for the same reason. And structure (JSON, tool-args) costs real format tokens.
When you budget tokens, count with the provider's real tokenizer. The folklore that "tiktoken undercounts Claude by 15-20%" is overstated. The real gap is smaller (~12% on prose) and can go either way depending on content. Use the provider's count_tokens endpoint instead of guessing.
Sampling and "thinking": determinism was always a myth
The output is sampled from a distribution. Temperature and top-p control its spread. Two consequences break common intuitions.
The first: non-determinism breaks naive caching and forces pass^k evals, since one green run doesn't mean reliable. The second is newer, and it's a model-mental shift:
Extended thinking trades reasoning tokens for quality. It's test-time compute, the inference-time scaling lever, and it's a genuine result: compute-optimal test-time scaling can beat a much larger model on some problems. The fixed budget_tokens knob is deprecated (removed, 400, on 4.7+) in favor of effort (low → max; xhigh arrived in Opus 4.7) and interleaved thinking between tool calls. Interleaved thinking is exactly what lets the agent reason about a tool's result before deciding the next call, instead of planning everything up front.
More thinking isn't free, and it isn't monotonically better. There's a documented "overthinking" regime where accuracy drops at long budgets. Calibrate effort per route.
The Bitter Lesson decides what to build
The last piece is a decision criterion. Rich Sutton's Bitter Lesson: general methods that scale with compute beat hand-engineered domain knowledge over time. Applied to agents, the scaffolding you write today to patch a model limitation is technical debt the next generation erases. So bet on capabilities that scale (agentic search, long context, learned tool-use, reasoning) and minimize the hand-engineered logic the model will eventually absorb. That's the criterion for deciding what to build versus what to leave to the model.