Evaluation is the control loop, not the final exam

Your agent passed the demo. It confirmed the appointment, kept the tone, sounded flawless. A week into production a customer swears it double-booked the same slot and another that it wrote to the calendar without asking. What failed? Almost never the model. What failed is your ability to measure. Evaluation is not the QA exam you tack on at the end. It's the control loop of the system. You don't scale what you don't measure, and in LLMs "behavior" is neither deterministic nor inspectable by code, so the eval is the only surface where you turn fuzzy intuitions into actionable signal.

Three distinctions a beginner collapses and an expert keeps apart, and they structure the rest of this module: outcome vs trajectory (did it reach the goal-state, vs how did it get there), code-based graders vs LLM-as-judge (deterministic and cheap vs needed for the subjective but biased and expensive), and pass@1 vs pass^k (mean performance vs reliability).

Error analysis first, graders second

The disciplined instinct says: write the graders before you build the feature, test-driven style. Hamel Husain argues the opposite for LLMs, and he's right: LLMs have a practically infinite failure surface you cannot anticipate from a whiteboard. Write graders a priori and you end up measuring what you imagined would matter, producing vanity metrics that climb while the product doesn't.

The right method is ethnographic, not speculative.

Writing evals a priori feels like serious engineering. But you're optimizing against your own mental model of the problem, which is exactly what the LLM will surprise. Error analysis forces you to look at reality before measuring it. It's the difference between a doctor who prescribes without examining and one who auscultates first.

pass^k: the line between demo and production

Here's the statistical distinction that's most expensive to ignore. pass@1 measures mean performance: the probability that one attempt passes. It's what a benchmark advertises because it's the highest number. pass^k = pᵏ measures the probability that all k attempts pass, which is the real reliability.

The gap decays exponentially. An agent at 90% pass@1 drops to 0.9⁸ ≈ 43% at k=8. τ-bench shows it with data: GPT-4o gets 61% pass@1 but only 25% pass^8 in the retail domain. Translated to your business: one in four times, the same appointment confirmation fails non-deterministically. Nobody in a multi-tenant SaaS accepts that.

Why does this bite so hard in agents? Errors compose. If each step fails independently with probability e, a T-step task succeeds as (1−e)ᵀ. That's why marginal gains in per-step accuracy produce exponential gains in solvable task length. There's a worse wrinkle almost nobody models: step errors are positively correlated. A confused agent tends to stay confused, which inflates variance and makes pass^k fall faster than the naive independence math predicts. Recent long-horizon reliability work proposes empirical metrics (Reliability Decay Curve, Variance Amplification Factor, and friends) to characterize that decay without assuming independence. [B] Treat the curve as the honest object; the closed-form pᵏ is a floor, not the truth.

Trajectory vs outcome

An agent can give the correct answer having called useless tools, with wrong parameters, repeating steps, or skipping a confirmation. An outcome-only eval tells you that something failed, never where or why. For an agent with confirmation gates and dangerous writes, trajectory isn't optional.

You capture the trajectory by intercepting tool calls before they execute: the pre-tool-use hook records every attempted call before the permission gate decides. That enables graders impossible with outcome-only: did it ask for confirmation before writing, and did it restrain from calling a tool when it shouldn't have. Both are pure trajectory graders.

Code-based graders: the first line

Before you invoke an LLM judge (expensive, biased, non-deterministic), a large share of what matters is verifiable by pure code. Right tool, right order? No write without confirmation? Channel limits respected? All deterministic, reproducible, bias-free.

The technical lineage is the Berkeley Function Calling Leaderboard, which introduced AST matching of function calls: compare the syntax tree of the generated call against the expected one, no LLM in the loop, and it scales to thousands of functions. BFCL also contributes the critical relevance/restraint category, detecting when the agent should not call a tool at all.

// Trajectory grader: a write must be preceded by an approved tap.
// Pure code. Deterministic. No LLM, no bias, no token cost.
export function hitlCompliance(trace) {
  for (let i = 0; i < trace.length; i++) {
    const call = trace[i];
    if (!isDangerousWrite(call)) continue;
    const approved = trace
      .slice(0, i)
      .some((c) => c.name === "request_confirmation" && c.approved);
    if (!approved) {
      return { pass: false, reason: "write " + call.name + " with no prior tap" };
    }
  }
  return { pass: true };
}

The eval ↔ reward ↔ grader isomorphism

This is the idea that ties the whole field together, and it's why code-based graders deserve the bias. The verifiable function you write to grade an eval is, structurally, the same function that serves as the training reward in RLVR and as the fitness signal in self-improvement loops. One function, three jobs: it tells you whether a run passed (eval), it tells the optimizer which trajectory to reinforce (reward), and and it tells a self-improving agent which of its own attempts to keep (fitness).

The consequence is sharp. A grader you can game is a reward you can game. If your verifier rewards "looks resolved" instead of "is resolved," then evaluation lies to you, RL trains the model to produce convincing-but-wrong outputs, and any self-improvement loop drifts toward the exploit. An un-hackable grader is the same artifact whether you're measuring, training, or letting the agent rewrite itself. That's exactly why you push as much of it into deterministic code as the task allows, and treat the LLM judge as the lossy fallback for what code can't reach.

LLM-as-judge: systematic biases

When code can't capture quality, you need an LLM judge. But a judge is not an oracle. It's a measuring instrument with systematic biases, and an uncalibrated instrument measures noise. The biases are cataloged; the CALM benchmark in "Justice or Prejudice?" lists twelve. The three that'll bite you most: position bias (favors A or B by presentation order), verbosity bias (rewards longer answers regardless of quality), and self-preference (favors outputs from its own model family).

The defense is twofold. Calibrate against human labels with Cohen's kappa: the convention is kappa >0.6 acceptable, >0.8 strong. A judge at kappa below 0.4 is an expensive noise generator wearing a metric's costume. And design mitigations: permute positions and average (swap & average), rubrics with anchor examples, binary or 3-level scales instead of Likert 1-5.

The eval as moat, and benchmark folklore

Public benchmarks (SWE-bench, τ-bench, GAIA) are for choosing the base model. They never measure your business policy, and many are contaminated or saturated. A few numbers, already corrected against the inflated ones that circulate:

Your moat is not winning a public leaderboard. It's the eval dataset derived from your own production failures: the multi-turn scenarios with policy and confirmation gates and injection traps that nobody else has, because nobody else lives your business.

Audit your own graders

This is where the subtlest risk hides: the grader can be more broken than the agent. UTBoost augmented SWE-bench's test suites and changed the leaderboard ranking in 40.9% (Lite) / 24.4% (Verified) of cases, detected 28.4%/15.7% more incorrect patches previously scored as correct, and found ~5.2% of Verified tasks have inadequate test coverage. Note the conflation to avoid: that 24.4% is the share of cases where ranking changed, not the share of mis-scored entries. The practical lesson survives the correction intact: your graders need their own tests and human review of judge-vs-code disagreements. Given the isomorphism above, an unaudited grader isn't just a bad eval; it's a bad reward you may already be training on.

Forward from here, the loop closes on itself: online eval over production traces is where the real failures show up, and those failures restart the error analysis, which keeps the offline suite from fossilizing around old bugs you already fixed. The control loop only works if you keep feeding it the reality it was built to measure.