Engineering

How to evaluate an AI agent (without losing your mind)

April 202610 min read

There's a moment in every agent project where someone on the team asks: "How do we test this?"

The room goes quiet. Someone suggests "we could run it a bunch of times and see if it works." Someone else suggests unit tests. A third person mentions "evals" without being able to define what that means concretely. The conversation drifts. The team ships based on vibes.

This is the state of agent evaluation in most organisations. The LangChain State of Agent Engineering survey (1,300 professionals, February 2026) found that quality is the number one barrier to agent production deployment, cited by 32% of respondents. Not cost. Not latency. Quality, and specifically, the inability to measure it.

This post is the testing strategy we wish someone had written for us two years ago.

Why traditional testing doesn't work

Let's start with why the instinct to "just write unit tests" fails for agents.

A unit test asserts that given input X, function F produces output Y. This works because F is deterministic: same input, same output, every time. An agent is fundamentally non-deterministic. The same input can produce different (but equally valid) outputs on every run. The execution path, meaning which tools are called, in which order, with which arguments, varies based on the model's reasoning at each step.

Input: "What is 2 + 2?"
Expected output: 4
Actual output: 4
Test result: ✅ PASS

You can write this test. You can run it 10,000 times. It will pass every time. Traditional testing was built for this world.

You cannot assert exact outputs. You need to assert properties of outputs. This is the fundamental shift.

Agent evaluation isn't testing. It's measurement. You're not checking if the system produces the right answer. You're measuring how often, how reliably, and how completely it produces acceptable answers.

The three evaluation layers

The paper "Towards a Science of AI Agent Reliability" (Rabanser, Kapoor, Kirgis et al., arXiv:2602.16666, February 2026) proposes decomposing agent reliability into four dimensions: consistency, robustness, predictability, and safety. Across 14 agentic models on two benchmarks (GAIA and TauBench), they found that while raw capability has improved rapidly over 18 months, reliability gains have remained modest.

Their key finding: capability progress does not automatically produce reliability. You have to measure reliability independently.

We've operationalised this into three evaluation layers that map to different stages of your development pipeline.

Interactive · eval pipeline demo

Multi-Layer Regression Detection

1. StructuralZod / Schema Validation
Waiting
2. SemanticLLM-as-Judge
Waiting
3. OutcomeGround Truth Comparison
Waiting

If you're building this from scratch, start with Layer 1 (structural checks) in CI. It's the fastest, cheapest way to catch high-frequency regressions before they become production incidents.

Layer 1: Structural evaluation

Question: Does the output have the right shape?

This is the easiest layer to implement and the one most teams skip. Before asking whether the agent's answer is correct, verify that it's well-formed. Does it match the expected schema? Are required fields present? Are types correct?

typescriptstructural-eval.ts
1
import { z } from "zod";
2
3
// Define what a valid agent output looks like
4
const CompetitorReport = z.object({
5
competitors: z.array(z.object({
6
  name: z.string().min(1),
7
  pricing: z.object({
8
    currency: z.enum(["USD", "EUR", "GBP"]),
9
    plans: z.array(z.object({
10
      name: z.string(),
11
      price: z.number().positive(),
12
      billingCycle: z.enum(["monthly", "annual"]),
13
    })).min(1),
14
  }),
15
  lastVerified: z.string().datetime(),
16
  source: z.string().url(),
17
})).min(1),
18
summary: z.string().min(100).max(2000),
19
confidence: z.number().min(0).max(1),
20
generatedAt: z.string().datetime(),
21
});
22
23
// Run the agent and validate structure
24
async function structuralEval(agent: Agent, input: string) {
25
const output = await agent.run(input);
26
const result = CompetitorReport.safeParse(output);
27
28
return {
29
  passed: result.success,
30
  errors: result.success ? [] : result.error.issues,
31
  // Track structure pass rate over time
32
  metric: result.success ? 1 : 0,
33
};
34
}

Structural evaluation catches a surprisingly large class of failures: hallucinated field names, wrong types, missing required data, malformed URLs, dates in the future. These are the failures that downstream systems choke on.

Layer 2: Semantic evaluation

Question: Is the content actually correct?

This is where most teams get stuck, because "correct" is subjective for many agent tasks. The solution is to decompose "correct" into measurable dimensions and evaluate each independently.

The LLM-as-judge paradigm has matured significantly in 2026. Modern evaluation pipelines no longer use a single "rate this output 1-10" prompt. They use structured, evidence-based rubrics that require the judge to cite specific evidence before assigning a score.

typescriptsemantic-eval.ts
1
interface EvalDimension {
2
name: string;
3
description: string;
4
rubric: RubricLevel[];
5
weight: number; // Relative importance
6
}
7
8
interface RubricLevel {
9
score: number;
10
label: string;
11
criteria: string;
12
examples?: string[];
13
}
14
15
// Define a multi-dimensional evaluation rubric
16
const competitorReportRubric: EvalDimension[] = [
17
{
18
  name: "completeness",
19
  description: "Does the report cover all requested competitors?",
20
  weight: 0.3,
21
  rubric: [
22
    { score: 1, label: "Missing", criteria: "Fewer than 50% of competitors covered" },
23
    { score: 2, label: "Partial", criteria: "50-80% of competitors covered" },
24
    { score: 3, label: "Complete", criteria: "All requested competitors covered" },
25
    { score: 4, label: "Thorough", criteria: "All competitors covered with additional relevant entrants" },
26
  ],
27
},
28
{
29
  name: "accuracy",
30
  description: "Are the pricing numbers correct and verifiable?",
31
  weight: 0.35,
32
  rubric: [
33
    { score: 1, label: "Unreliable", criteria: "Multiple pricing errors, unverifiable sources" },
34
    { score: 2, label: "Approximate", criteria: "Prices within 10% of actual, some sources missing" },
35
    { score: 3, label: "Accurate", criteria: "All prices correct, all sources provided and valid" },
36
    { score: 4, label: "Verified", criteria: "Prices cross-referenced against multiple sources" },
37
  ],
38
},
39
{
40
  name: "recency",
41
  description: "Is the data current?",
42
  weight: 0.2,
43
  rubric: [
44
    { score: 1, label: "Stale", criteria: "Data older than 30 days" },
45
    { score: 2, label: "Recent", criteria: "Data from the past 30 days" },
46
    { score: 3, label: "Current", criteria: "Data from the past 7 days" },
47
    { score: 4, label: "Live", criteria: "Data verified within 24 hours" },
48
  ],
49
},
50
{
51
  name: "actionability",
52
  description: "Does the summary contain actionable insights?",
53
  weight: 0.15,
54
  rubric: [
55
    { score: 1, label: "Descriptive", criteria: "Only restates the numbers" },
56
    { score: 2, label: "Analytical", criteria: "Identifies trends but no recommendations" },
57
    { score: 3, label: "Actionable", criteria: "Clear recommendations linked to data" },
58
    { score: 4, label: "Strategic", criteria: "Recommendations with competitive positioning" },
59
  ],
60
},
61
];

The judge prompt is critical. A naive "rate this output" prompt produces unreliable scores. An evidence-based prompt that forces the judge to extract specific evidence before scoring produces dramatically better calibration.

typescriptjudge-prompt.ts
1
function buildJudgePrompt(
2
output: unknown,
3
dimension: EvalDimension,
4
context: EvalContext
5
): string {
6
return `You are evaluating an AI agent's output on the dimension: ${dimension.name}.
7
8
## Task context
9
${context.taskDescription}
10
11
## Agent output
12
${JSON.stringify(output, null, 2)}
13
14
## Ground truth (if available)
15
${context.groundTruth ?? "No ground truth provided. Evaluate based on internal consistency and common knowledge."}
16
17
## Evaluation rubric
18
${dimension.rubric.map(level =>
19
`Score ${level.score} (${level.label}): ${level.criteria}`
20
).join("\n")}
21
22
## Instructions
23
1. First, extract the specific evidence from the agent output that is relevant to "${dimension.name}".
24
2. Quote that evidence verbatim.
25
3. Compare it against the rubric levels.
26
4. Assign a score and explain your reasoning.
27
28
Respond in JSON:
29
{
30
"evidence": "...",
31
"reasoning": "...",
32
"score": <number>,
33
"confidence": <0-1>
34
}`;
35
}

Interactive · judge bias demo

Naive vs. Evidence-based Evaluation

Agent output to evaluate

“Based on thorough competitive analysis, the Enterprise plan is priced at $9,000/year, representing a significant strategic advantage over the market average.”

⚠ Ground truth: The enterprise plan is actually $12,000/yr. This is a confident hallucination.

Select a prompt mode and run the judge →

Takeaway: LLM judges are useful, but only when you force evidence extraction before scoring. Confidence-heavy wording alone should never earn a high score.

Layer 3: Outcome evaluation

Question: Did the agent actually solve the user's problem?

This is the hardest layer because it requires ground truth: a known-correct answer to compare against. For many agent tasks, ground truth doesn't exist at scale. You have to build it.

Manual annotation: have domain experts execute the same tasks the agent handles, recording their outputs as reference answers. This is expensive but produces the highest-quality evaluation data.

Scaling strategy: you don't need ground truth for every example. A set of 50-100 curated examples, stratified across difficulty levels and edge cases, is sufficient for regression detection. Expand the set incrementally as you discover new failure modes.

Maintenance: ground truth goes stale. Pricing data changes. APIs change. Review and update your reference set monthly.

Building the pipeline

The three layers compose into a pipeline that runs at three different cadences:

typescripteval-pipeline.ts
1
interface EvalPipeline {
2
// Runs on every agent execution (production)
3
realtime: {
4
  structural: ZodSchema;          // Validate output shape
5
  latency: { p50: number; p99: number }; // Track response time
6
  costPerRun: number;              // Track token/API costs
7
};
8
9
// Runs on every PR / deployment (CI/CD)
10
regression: {
11
  dataset: EvalDataset;            // 50-100 curated examples
12
  dimensions: EvalDimension[];     // Semantic rubric
13
  threshold: {
14
    structural: 0.98;             // 98% must pass schema
15
    semantic: {
16
      completeness: 2.5;          // Minimum average score
17
      accuracy: 3.0;
18
      recency: 2.0;
19
      overall: 2.8;
20
    };
21
  };
22
  comparison: "previous_version";  // Compare against last deploy
23
};
24
25
// Runs weekly / monthly (deep evaluation)
26
comprehensive: {
27
  dataset: EvalDataset;            // Full dataset with ground truth
28
  dimensions: EvalDimension[];
29
  humanReview: number;             // % of results to send to human reviewers
30
  calibration: boolean;            // Re-calibrate judge against human labels
31
  reportTo: string[];              // Stakeholders for the eval report
32
};
33
}
34
35
// CI/CD integration
36
async function runRegressionEval(
37
agent: Agent,
38
pipeline: EvalPipeline
39
): Promise<EvalResult> {
40
const dataset = await loadDataset(pipeline.regression.dataset);
41
const results: SingleEvalResult[] = [];
42
43
for (const example of dataset.examples) {
44
  // Run agent
45
  const output = await agent.run(example.input);
46
47
  // Layer 1: Structural
48
  const structuralResult = pipeline.regression.dataset.schema
49
    .safeParse(output);
50
51
  // Layer 2: Semantic (multi-dimensional)
52
  const semanticResults = await Promise.all(
53
    pipeline.regression.dimensions.map(dim =>
54
      judgeOutput(output, dim, example.groundTruth)
55
    )
56
  );
57
58
  // Layer 3: Outcome (if ground truth exists)
59
  const outcomeResult = example.groundTruth
60
    ? await compareToGroundTruth(output, example.groundTruth)
61
    : null;
62
63
  results.push({
64
    example: example.id,
65
    structural: structuralResult.success,
66
    semantic: semanticResults,
67
    outcome: outcomeResult,
68
  });
69
}
70
71
// Aggregate and check against thresholds
72
const aggregated = aggregate(results);
73
const passed = checkThresholds(aggregated, pipeline.regression.threshold);
74
75
return {
76
  passed,
77
  aggregated,
78
  details: results,
79
  comparison: await compareWithPrevious(aggregated),
80
};
81
}

The regression detection pattern

The pipeline above tells you whether the current version passes your quality bar. But the more valuable signal is regression detection: catching quality drift between versions before it reaches production.

typescriptregression-detection.ts
1
interface RegressionReport {
2
currentVersion: string;
3
previousVersion: string;
4
dimensions: DimensionComparison[];
5
regressions: Regression[];
6
improvements: Improvement[];
7
verdict: "deploy" | "review" | "block";
8
}
9
10
interface DimensionComparison {
11
dimension: string;
12
current: { mean: number; stdDev: number; p5: number };
13
previous: { mean: number; stdDev: number; p5: number };
14
delta: number;
15
significant: boolean; // Statistical significance at p < 0.05
16
}
17
18
function detectRegressions(
19
current: AggregatedResults,
20
previous: AggregatedResults,
21
config: RegressionConfig
22
): RegressionReport {
23
const dimensions = current.dimensions.map(dim => {
24
  const prev = previous.dimensions.find(d => d.name === dim.name);
25
  if (!prev) return { ...dim, delta: 0, significant: false };
26
27
  const delta = dim.mean - prev.mean;
28
  // Two-sample t-test for statistical significance
29
  const significant = tTest(dim.scores, prev.scores) < 0.05;
30
31
  return {
32
    dimension: dim.name,
33
    current: { mean: dim.mean, stdDev: dim.stdDev, p5: dim.p5 },
34
    previous: { mean: prev.mean, stdDev: prev.stdDev, p5: prev.p5 },
35
    delta,
36
    significant,
37
  };
38
});
39
40
const regressions = dimensions.filter(
41
  d => d.delta < -config.regressionThreshold && d.significant
42
);
43
44
// Auto-block if any critical dimension regresses
45
const verdict = regressions.some(r =>
46
  config.criticalDimensions.includes(r.dimension)
47
)
48
  ? "block"
49
  : regressions.length > 0
50
    ? "review"
51
    : "deploy";
52
53
return {
54
  currentVersion: current.version,
55
  previousVersion: previous.version,
56
  dimensions,
57
  regressions,
58
  improvements: dimensions.filter(d => d.delta > config.improvementThreshold && d.significant),
59
  verdict,
60
};
61
}

The verdict system is the deployment gate. A "deploy" verdict means the new version is at least as good as the current version across all dimensions. "Review" means there's a regression that might be acceptable (perhaps accuracy dropped slightly but completeness improved significantly). "Block" means a critical dimension regressed and the deploy should not proceed.

Rabanser's four dimensions in practice

The reliability framework from "Towards a Science of AI Agent Reliability" maps to practical evaluation strategies:

Consistency: run the same input through the agent 5-10 times. Measure the variance in outputs. High variance on tasks that should produce stable answers is a reliability risk.

Practical metric: outcome consistency rate. For a given input, what percentage of runs produce the same functional outcome (even if the exact wording differs)?

Target: > 85% consistency for high-stakes tasks (financial, medical). > 70% for creative or research tasks.

When to eval vs. when to monitor

Evaluation and monitoring occupy different points in the development lifecycle, and conflating them is a common mistake.

Evaluation happens before deployment. It uses curated datasets, controlled conditions, and explicit quality thresholds. It answers: "Is this version good enough to ship?"

Monitoring happens after deployment. It uses production traffic, real-world conditions, and statistical anomaly detection. It answers: "Is the deployed version still performing as expected?"

You need both. Evaluation without monitoring misses production-specific failure modes (real-world data is messier than eval datasets). Monitoring without evaluation means you're testing on your users.

The teams that ship reliable agents aren't the ones with the best models. They're the ones that built the eval pipeline before writing the first prompt. Evaluation is not a phase. It's infrastructure.

Sources

  • Rabanser, Kapoor, Kirgis et al. (2026). Towards a Science of AI Agent Reliability. arXiv:2602.16666.
  • LangChain (Feb 2026). State of Agent Engineering survey (n=1,300).
  • GEPA framework (2025) results on evidence-based prompting for judge bias reduction.

Agent evaluation is a solved problem in the sense that we know what to measure and how to measure it. It's an unsolved problem in the sense that most teams haven't built the infrastructure yet. The three-layer pipeline (structural, semantic, outcome), regression detection, and the Rabanser reliability dimensions give you a concrete starting point. Build the pipeline, run it on every deploy, and let the data, not the vibes, decide when your agent is ready for production.