Picture an ER doctor at 3am. They never see directly inside the patient. They get partial signals (what the patient says, a blood panel, a monitor), form a hunch about what's wrong, take one action (order a test, start a drug), and then wait for the real result before deciding the next move. They don't act on what they imagine the test will say. They act on what it actually says.
That is almost exactly what an AI agent is. An agent keeps choosing its next move while it can only ever see part of the world. It acts, watches what really happens, updates its picture, and acts again. "An LLM with tools" is the toy schematic. This loop (deciding under an incomplete view, then correcting against reality) is the real thing you're engineering.
Once you look at an agent this way, the decisions that felt ad-hoc stop being separate problems. What to put in context, when to verify, which tool to expose, when to ask permission: they all collapse into the same three questions every time. What do I believe right now, what am I about to change, and what am I aiming at?
The formal name: a policy over a POMDP
Here is what researchers call this picture, because you'll meet the words everywhere and they're worth being able to read.
The decision-maker is a The function mapping what the agent has observed so far to its next action; in an LLM agent, the reasoner conditioned by its system prompt IS the policy π(action | observation, history). glossary → : a function from everything observed so far to the next action. Written out, that's π(action | observation, history) — and the bar reads as "given." The next action given what the agent has seen. The LLM, steered by its system prompt, is that policy.
The world it acts on is a A decision-making model where an agent never sees the true world state directly and must act on a belief inferred from partial observations — the formal lens for LLM agents. glossary → : a decision process where you never see the full state of the world, only partial observations of it. (A single, self-contained API call is the easy opposite: a one-shot A decision model where the agent sees the full state of the world before each action. It's the fully-observed special case of a POMDP; a single, self-contained LLM API call is a degenerate one-step MDP, while a real agent acting on partial information is a POMDP. glossary → where the model sees everything it needs at once. A real agent almost never lives there.) This is the standard formulation in the agentic-RL literature, used since 2023–2024 (ReAct, ETO, HiAgent) and itself built on the classical POMDP (Åström, 1965); the 2025 agentic-RL survey gives a clean consolidated statement of it, contrasting the single-step MDP of LLM-RL with the POMDP of agentic RL. You don't need the formalism to build. You need the decomposition, because it renames problems you already have into problems you can reason about.
Each letter of POMDP is just a question you were already asking on every feature. The table makes that one-to-one.
| POMDP symbol | In an LLM agent | The engineering question you were already asking |
|---|---|---|
| S, hidden state | The real state of the world (e.g. the actual rows in the database) | (the agent never sees this directly) |
| O, observation | What the agent sees: messages, documents read, tool results | What do I put in context? |
| b, The agent's running estimate of the hidden world state formed from observations; in an LLM agent this is the curated context window, not the raw prompt. glossary → | The context window: the agent's estimate of S | What do I keep, compact, or externalize? |
| A, The complete set of actions an agent can take — in an LLM agent, its available tools; restricting it lowers policy variance and shrinks the attack surface. glossary → | The set of available tools | Which tools do I expose? |
| T, The rules by which an action changes the world and produces the next observation — the harness (workers, database, locks, persistence) that executes and persists effects. glossary → | The harness: worker, DB, locks, persistence | How does the effect execute and persist? |
| R, reward | The task-success eval | How do I know it did the job? |
| π, policy | The LLM conditioned by the system prompt | How does it pick the next action? |
Forget the doctor for a second and notice what the formal version buys you. Partial observability explains the classic failures with one cause instead of a shrug at "the model being dumb." The agent hallucinates (it filled unobserved state with a false belief), repeats actions (its belief never recorded that it already acted), or gets lost (the belief degraded when the window overflowed). All three are pathologies of the belief state, not random model stupidity.
Workflow vs agent: who controls the trajectory
The most misread distinction in the field isn't "does it use an LLM?" It's who decides the trajectory. In a A control-flow distinction: in a workflow, code predefines the trajectory and the LLM fills slots; in an agent, the LLM chooses its own actions and tools at runtime. glossary → , the paths are predefined in code: you fix the flow, and the LLM fills slots inside branches you already drew. In an agent, the LLM directs its own steps and tool use at runtime.
Most of the production systems people called "agentic" in 2025 were workflows with an LLM as a component. That's not an insult, it's the recommendation. Anthropic names five workflow patterns that cover most cases before you need a full agent: prompt chaining (sequential), routing (classify and dispatch), parallelization (sectioning/voting), orchestrator-workers (a lead delegates dynamically), and evaluator-optimizer (generate and critique in a loop). Confusing workflow with agent produces the archetypal over-engineering: a reasoning loop where a 10ms API call would have done.
The loop is a control loop, not a long thought
The The repeating cycle where a model decides an action, the harness executes it against the environment, and the resulting observation is fed back for the next decision until a stopping condition is met. glossary → is gather context → take action → verify → repeat. What separates an agent from a long chain-of-thought is that every step receives a real observation from the environment, not a prediction from the model. The verify step, comparing against An observable external fact injected into the loop — a real tool result, a passing/failing test, a concrete error — as opposed to the model's own unverified self-assessment. glossary → each iteration, is what stops the error from compounding. Treating the model's output as truth instead of executing the action and observing is chain-of-thought wearing an agent costume, and it's the number-one failure mode.
The arithmetic that governs the architecture
Here's the click moment, and it's just multiplication. Say each step succeeds on its own with probability p. Then a task that takes t steps in a row succeeds with probability p^t. Your linear intuition betrays you hard:
| Per-step accuracy (p) | Steps (t) | Task success (p^t) |
|---|---|---|
| 90% | 10 | ≈ 35% (0.90¹⁰ = 0.3487) |
| 99% | 100 | ≈ 37% (cumulative failure ≈ 63%) |
| 99.9% | 100 | ≈ 90% |
An agent that nails 90% of its steps fails two out of three ten-step tasks. That single fact is why The phenomenon where small per-step error rates accumulate over a long task, collapsing success probability (heuristically p^t; rigorously a quadratic εT² bound in imitation learning per Ross & Bagnell). glossary → , not raw model smarts, sets the ceiling on long tasks.
The 2025 finding that matters: near p ≈ 1, tiny improvements in step accuracy produce huge jumps in the reachable horizon. That's why reliability per step dominates raw capability on long-horizon tasks. Going from 99% to 99.9% per step (by engineering reliability: solid tools, verification, poka-yoke) buys you more, over a long horizon, than swapping in a "smarter" model.
The rigorous version: why the error actually grows quadratically (εT²)
p^t is a back-of-envelope gauge, not a theorem. It assumes independent, identically reliable steps, which agents are not: steps correlate, error recovers, the harness intervenes. So read p^t for the shape of the problem. The established imitation-learning result is stronger and different: compounding error grows quadratically with the horizon (on the order of εT², from Ross & Bagnell's Ross, Gordon & Bagnell's 2011 imitation-learning algorithm that mitigates compounding error by training the policy on the states it actually visits, replacing quadratic error growth with a linear O(εT) term. glossary → analysis), because a small per-step error rate ε pushes the policy into states it never trained on, where it errs more. Cite p^t for intuition. Cite εT² when you need the real theory.
Why this forces architecture instead of suggesting it
HITL on writes, checkpoints, an explicit verify step, shorter horizons, task decomposition: none of these are style preferences. They're direct consequences of compounding error. If you can't push p higher, you shorten t. If you can't shorten t, you insert a human or a verifier to reset the effective error each step. The math doesn't care about your taste; it tells you the only three levers you have: raise p, lower t, or break the chain.
Autonomy is a spectrum, set per action
Autonomy isn't a binary, and it isn't an L1–L5 capability ladder. It's a spectrum defined by the role the human takes: operator (directs each action) → collaborator (approves each decision, HITL) → consultant (the agent proposes, the human decides) → approver → observer (the agent acts alone). More autonomy isn't better. You grade it by reversibility and stakes.
Which loops back to the highest-leverage question in the whole field. It's almost never "which agent framework?" It's "do I even need an agent, or does a workflow or a single LLM call solve this cheaper, faster, more predictably?" The decision ladder:
- Rules engine, if every valid output is enumerable in tests. Microseconds, zero cost, no plausible-but-wrong.
- One LLM call (+ in-context examples), you need language understanding but the flow is fixed.
- Workflow, there are known branches; draw them as a flowchart in code.
- Agent, only if the action space is open, the horizon variable, and the trajectory unpredictable.
The default is the simplest thing that works. The agent earns its place only when its performance gain pays for the latency, cost, and compounding error it brings.
Tools are the action space
The agent's action space is its tool set, and the quality of that set shapes the policy as much as the model does. Designing the The design of the tools an agent can call — names, formats, examples, error-proofing — effectively the design of the POMDP's action space and a major reliability lever. glossary → is designing the action space of the POMDP: obvious names, examples, documented edge cases, error-proof formats ( A design principle borrowed from manufacturing: making tools and formats error-proof so the agent structurally cannot make certain mistakes. glossary → ), and, this one gets skipped, restricting the set to cut the policy's variance.
// A wide-open tool set is high policy variance + a bigger attack surface.
// Fewer, sharper tools make a modest model reliable.
const DISALLOWED = ['Bash', 'Write', 'Edit', 'WebFetch', 'Task'];
// The action space is a security and reliability lever,
// not just a capability list. Default-deny, then add what the
// task actually needs, never the reverse.A bad ACI degrades any model. A good one makes a modest model dependable. The action space is an underrated reliability lever next to the endless debate over which model to pick.
What the consensus over- and under-weights
- "More agentic = better." False. In 2025 the deterministic workflows won production.
- Multi-agent by default. It only wins on parallelizable, read-heavy work; on shared state it produces conflicting decisions.
- Heavy frameworks as a starting point. They hide the loop. Direct API calls first: most patterns are a few lines.
- "Agent = LLM + tools." Incomplete. It ignores the loop, the environment, and partial observability, the things that actually define an agent.
- Computing the error budget before designing the horizon. Almost nobody does it; it's why verify-steps and HITL aren't optional.
- Reliability per step ≫ capability. Near p ≈ 1, engineering reliability beats swapping models.
- The context IS the belief state. Curating it is first-order architecture, not a prompting detail.
- The environment is part of the agent. The harness, locks, and persistence are the transition dynamics T.
The Anthropic vs Cognition tension on multi-agent
This isn't a settled consensus, it's a real disagreement between two serious labs. Anthropic reports an A multi-agent pattern where a lead agent decomposes a task, spawns specialized workers with bounded mandates, and synthesizes their results into one answer. glossary → system (an Opus lead plus Sonnet subagents) beating the single agent by 90.2% on its internal research eval: multi-agent wins on read-heavy, parallelizable tasks where each subagent explores an independent branch. Cognition argues the opposite for shared-state tasks: parallel subagents make independent decisions that conflict on recomposition, and they advocate single-threaded plus a compressor LLM. They don't contradict each other so much as bound the domain. Parallelize independent reads; stay single-threaded when there's writing or shared state. (One precision note: Cognition popularizes and defines "context engineering" but didn't coin it; that's usually credited to Tobi Lütke and Andrej Karpathy, mid-2025.)
The point of the lens isn't elegance. It's that once the agent is a policy on a POMDP, the rest of the path stops being a list of disconnected techniques: context engineering is curating the belief, RAG and agentic search extend it, HITL and safety grade autonomy per action, and evals measure the policy rather than the answer.