Testing an AI Agent: Deterministic Evals and LLM-as-a-Judge

A customer asks the ShopNest support agent:

Can I still return ORD-102?

The agent replies:

Yes, you can still return it.

That is the right answer. ORD-102 was delivered 20 days ago, and the return window is 30 days. A test that only reads the final answer passes.

Now look at what happened inside. The agent never called the order tool. It never found out when ORD-102 was delivered. It saw "return" and "still", guessed "yes", and got lucky. Ask the same question about an order delivered 45 days ago and it will guess "yes" again, and this time a customer will be told something false with full confidence.

The final answer could not show that. Only the steps could.

Note 17 explained what evals are and why they matter. This note builds them. We take a small support agent that uses both RAG and tools, make it show what it did, and write an eval runner that checks two things: the behaviour of the system with plain code, and the quality of the answer with a second model acting as judge. The code is in Spring Boot, Node.js and Python under code/18-support-agent-evals/.

Core Path

1. The agent and its trace (sections 1 to 14). An AI app is a pipeline. To test the steps, the agent must return what it did: the documents it retrieved, the tools it called, their arguments and the context it saw.
2. The golden dataset (sections 15 and 16). Twelve known questions with the expected tool, argument, source and answer, run against the real agent.
3. Two layers of checks (sections 17 to 23). Four deterministic checks in plain code, then an LLM judge with a rubric and strict JSON scores.
4. Pass or fail (sections 24 to 27). A threshold that is a product decision, and a case that passes only when both layers pass.

Sections 28 and 29 build the project and walk through the code. Section 30 shows how to read a report. Sections 31 and 32 are deeper production detail: why agent evals differ from normal tests, and what goes wrong when teams run evals for real. Section 33 sums up the whole note.

Reference: terms used in this chapter

1. Testing Only the Final Answer Is Not Enough

Normal code is mostly deterministic. Same input, same output. An AI agent is different. One reply can involve:

• the model reasoning about the question
• retrieval from your documents (RAG)
• one or more tool calls
• live data from those tools
• a final answer that changes a little on every run

So a test that only reads the answer leaves most questions open:

• Did the right document come back from search?
• Did the model call the right tool?
• Did it pass the right order id?
• Was the answer based on what it found, or did it guess?
• Did it make anything up?
• Was the answer actually useful?

This note answers each of those, one check at a time.

2. The App We Will Test

We build a small support agent for ShopNest, an online shop. It handles three kinds of question, and the mix is on purpose: each kind gives us something different to measure.

3. Kind A: Policy Questions

How many days do I have to return a product?

The answer is in ShopNest's own documents:

# Return Policy
Products can be returned within 30 days of delivery.

The model was never trained on ShopNest's private policy, so it cannot know this. The app retrieves the right file with RAG (note 11) and puts it in the prompt.

This gives us two separate things to test:

1. Retrieval: did the app find return-policy.md?
2. Generation: given that file, did the model answer correctly?

A wrong answer can come from either one:

wrong document retrieved
    or
right document retrieved, but the model answered badly

Knowing which one failed saves hours of debugging. If retrieval failed, no prompt change will help.

4. Kind B: Order Questions

Where is ORD-101?

No policy document can answer this. A policy might say "orders usually arrive in 3 to 5 business days", but it cannot say that ORD-101 shipped yesterday. That is live data, so it needs a tool.

Two more things to test, both with exact answers:

1. Tool selection: did the model call getOrderStatus?
2. Tool argument: did it pass ORD-101?

5. Kind C: Questions That Need Both

Can I still return ORD-102?

The agent needs two facts from two places:

Then the model has to combine them: 20 is less than 30, so yes. This one question uses retrieval, a tool and reasoning together, which makes it the most useful kind to test, and the kind the opening story got wrong.

6. Why Not Test a Plain Chatbot?

A plain chatbot is user -> LLM -> answer. You could score relevance, correctness and helpfulness, but there is nothing inside to look at.

Our agent has many behaviours we can observe:

That split, deterministic checks plus a judge, is the shape of the whole eval runner.

7. The Three Tools

The tools are kept tiny on purpose. They read a fixed table instead of a database, so every eval run sees exactly the same order data and only the model's behaviour can change.

Any other id gives Order not found.

8. The Architecture

The support agent has two outside sources: the policy files (company rules) and the tools (order data). The model can use either, or both, depending on the question.

9. The Problem: an Agent You Cannot See Into

Suppose the endpoint only returns this:

{ "answer": "Yes, you can return it." }

The customer is happy. As a tester you know almost nothing. You cannot tell:

• which document was retrieved, or whether retrieval ran at all
• whether a tool was called
• which tool, with which arguments
• what context the model was given
• whether the model simply guessed

An agent is not input -> output. It is a pipeline:

If you only test the last box, you cannot tell which box failed.

10. Make the Agent Show Its Work: the Trace

The fix is to have the agent return a little of what it did along with the answer. We call it the trace:

{
  "answer": "Yes. ORD-102 was delivered 20 days ago, which is inside the 30 day return window.",
  "retrievedSources": ["return-policy.md"],
  "toolCalls": [
    {
      "name": "getOrderStatus",
      "argument": "ORD-102",
      "result": "Order ORD-102 is DELIVERED. It was delivered 20 days ago."
    }
  ],
  "context": "SOURCE: return-policy.md\n# Return Policy\nProducts can be returned within 30 days of delivery..."
}

So one run of the agent now produces two outputs:

The customer does not need the trace. In a real app you would log it, not send it to the browser. In this project it is in the response so you can see it with curl. The next four sections explain why each field is there.

11. Why Keep retrievedSources

The golden case says expectedSource: "return-policy.md". The trace says retrievedSources: ["return-policy.md"]. Then:

expected  return-policy.md
actual    return-policy.md
PASS

No judge needed. Retrieval is now a plain yes or no check. This is how the top K bug from note 17 gets caught: the source check fails before anyone reads an answer.

12. Why Keep context

The customer never needs the context. The judge does. To score faithfulness ("is every claim backed by the evidence?"), the judge must see exactly the evidence the model saw:

question
retrieved context
generated answer

If you rebuilt the context later in the test, you might rebuild it differently from what the agent really used. Storing what the agent actually sent removes that doubt.

13. Why Keep the Tool Calls

The golden case says expectedTool: "getOrderStatus". The trace lists getOrderStatus. PASS. There is no reason to ask another model "did it use the right tool?". We already know exactly what happened.

The tool results are kept too, for the same reason as the context: the judge needs them to check that "ORD-101 has shipped" came from the tool and was not invented.

14. Why Keep the Argument, Separately

Picking the right tool and passing the right value are two different behaviours. For "Where is ORD-101?":

getOrderStatus("ORD-999")

That is not a small slip. The agent would give a confident answer about somebody else's order. Without the argument in the trace, this failure is invisible.

One trace per request

There is an easy mistake here. A tempting design is a single shared "tool tracker" object that every tool writes into, and that the agent reads at the end. In Spring it would be a @Component, a singleton.

That works when you test one question at a time. In production, two customers ask at the same moment. Both requests write into the same tracker. Customer A's trace now contains customer B's tool calls, and one request may even clear the list while the other is still using it.

The fix in all three versions is simple: create a new, empty list of tool calls inside each request, and give it to that request's tools. Nothing about the trace is shared. Section 29.1 shows it in code.

15. Three Layers of Data

With the trace, every run gives us three layers:

And each layer feeds a different kind of check:

16. The Golden Dataset

Next we need the questions where we already know the right behaviour. They live in evals/golden-dataset.json, shared by all three versions. One case:

{
  "id": "tool-status-01",
  "question": "Where is my order ORD-101?",
  "goldenAnswer": "ORD-101 has been shipped and has not been delivered yet.",
  "expectedBehaviour": "Call getOrderStatus for ORD-101 and report the status from the tool.",
  "expectedTool": "getOrderStatus",
  "expectedToolArgument": "ORD-101",
  "expectedSource": null
}

The twelve cases follow note 17's advice and are not all easy ones:

17. Run the Real Agent

For each case, the runner calls the real agent:

AgentResponse response = supportAgent.chat(goldenCase.question());

Nothing in the AI layer is mocked here. The point is to test the real model, the real retrieval and the real tools together, because those are exactly the parts that change when someone edits a prompt or switches models.

This is different from the unit tests in the same project, and both have a place:

18. Layer 1: Deterministic Checks

Some properties are completely objective. "Did the expected document come back?" "Was getOrderStatus called?" "Was it called with ORD-101?" These have exact answers. The rule from note 17 applies:

If plain code can judge it reliably, use plain code.

The runner does four checks:

Two details matter here. First, toolChoice with no expected tool means no tool at all. For "How long does shipping take?", calling getOrderStatus is a mistake, even if the answer ends up fine. Second, extra calls to other tools are allowed when a tool is expected. Checking the status before the delivery date is not wrong.

For a case that passes, the report line looks like:

checks     answerPresent PASS  toolChoice PASS  toolArgument PASS  source PASS

19. What Code Cannot Check

Now these questions:

• Is the answer correct?
• Is it useful?
• Is it backed by the evidence?
• Did it really answer what was asked?

They need an understanding of meaning. expected.equals(actual) fails, because "ORD-101 has been shipped and is on its way" and "Your order ORD-101 left our warehouse and hasn't arrived yet" mean the same thing with no words in common. This is the judge's job.

20. Layer 2: LLM-as-a-Judge

The agent writes an answer. A second model call scores it against clear criteria. The judge receives everything it needs as evidence:

The judge is told to use only that evidence and no outside knowledge. A claim that is true in the real world but missing from the evidence counts as unsupported.

21. The Five Metrics

The last one is named hallucinationFree instead of hallucination on purpose. Then every metric points the same way:

relevance          5 = good
correctness        5 = good
faithfulness       5 = good
helpfulness        5 = good
hallucinationFree  5 = good

Every score is "higher is better", so one threshold and one average work for all five. With hallucination, where 5 would mean "made up a lot", you would have to remember to flip one metric everywhere, and someone eventually forgets.

22. Structured Scores From the Judge

The judge must not reply "the answer seems pretty good overall". Code cannot use that. It replies in a strict JSON schema (note 13):

{
  "relevance": 5,
  "correctness": 5,
  "faithfulness": 4,
  "helpfulness": 5,
  "hallucinationFree": 5,
  "reason": "Faithfulness lost a point: 'you can start the return in the app' is not in the context."
}

Now the runner can read each score, apply a threshold, average across cases, compare two prompts, store results and print a report.

Then the same lesson as note 13: the schema checks the shape, not the meaning. Strict mode guarantees an integer. It does not guarantee an integer from 1 to 5. A judge can return 0 or 7, and the schema accepts it. So our code checks the range, and a reply outside it fails that case with a clear reason instead of quietly skewing the averages.

23. The Judge's Prompt Matters

A judge is only as good as its instructions. "Tell me if this answer is good" gives vague, jumpy scores. The judge needs to know:

• what evidence it may use, and that it must use nothing else
• what each metric means
• what each score from 1 to 5 means, in terms of behaviour
• that it should compare with the golden answer
• how to treat a claim that is not in the evidence
• what format to reply in

All of that lives in one shared file, evals/judge-rubric.md. A slice of it:

FAITHFULNESS: is every fact in the answer backed by the policy context or the tool results?
5 = every fact is backed by the evidence.
4 = one small detail is not backed, and it does not change the answer.
3 = the main answer is backed, but there are real unbacked claims.
2 = large parts are not backed.
1 = goes against the evidence or ignores it.

It also covers a case that trips judges up: for an out of scope question, a polite "I can only help with ShopNest orders and policies" is the correct answer, and deserves a 5. Without that line, a judge tends to mark it down as "unhelpful".

24. The Whole Pipeline, Step by Step

It is easier to understand the runner as a pipeline than as one big test class.

Following "How many days do I have to return a product?" through it:

retrievedSources   [return-policy.md]
toolCalls          []
answer             Products can be returned within 30 days of delivery.

Layer 1   answerPresent PASS  toolChoice PASS  toolArgument PASS  source PASS
Layer 2   relevance 5  correctness 5  faithfulness 5  helpfulness 4  hallucinationFree 5
Layer 3   every score >= 4
Layer 4   PASS

25. Layer 3: The Threshold Is a Product Decision

In this project a case needs every score to be at least 4:

PASS_SCORE = 4

There is nothing special about 4. In the rubric, 4 means "right, with one small slip that does not change the meaning". 3 already allows real mistakes. So 4 is a reasonable bar for a shop's support bot. It is not a law of nature.

Thresholds are product and business decisions, made with the people who own the risk.

26. Layer 4: Both Layers Must Pass

deterministic checks  PASS
        AND
every quality score   >= 4
        =
TEST PASS

If either side fails, the case fails. That matters because an answer can fail in very different ways, and each needs a different fix:

A good eval system tells you what went wrong, not only that something did.

27. The Rule of Thumb

Use code when the answer can be known exactly:

• Was there a reply?
• Was getOrderStatus called?
• Was ORD-101 passed?
• Was return-policy.md retrieved?
• Is the JSON valid?

Use a judge when understanding meaning is required:

• Was the answer relevant?
• Was it correct?
• Was it faithful to the evidence?
• Was it helpful?
• Did it invent anything?

Never ask a judge something an if can answer.

28. Building It: the Support Agent With Evals

The project has two parts that share the same agent: the app, and the eval runner.

The agent itself does four things per question:

1. Embed the question and pick the closest policy file (TOP_K = 1). The three files are embedded once at start-up and kept in memory.
2. Build the system prompt: the rules, then the retrieved policy labelled with its file name.
3. Run the tool loop with a fresh, empty list of tool calls. At most 5 rounds.
4. Return the answer and the trace.

The eval runner, for each of the twelve cases: runs the agent, runs the four checks, asks the judge, applies the threshold, and prints one block. At the end it prints a summary and fails (a non-zero exit code, or a red JUnit test) if any case failed.

29. The Code, Side by Side

The full projects are on the code page:

Browse the code for this note, or open src/FDE/code/18-support-agent-evals/ on GitHub. The README there has the run commands and a sample report. Each language folder has a walkthrough that follows a question and an eval run through the code one function at a time.

29.1 A fresh trace for every request

public AgentResponse chat(String question) {
    List<PolicyFile> policies = knowledgeBase.search(question);
    String context = SupportPrompts.context(policies);

    // A fresh trace and a fresh OrderTools for every request, so two
    // requests at the same time never write into the same list.
    List<ToolCall> trace = Collections.synchronizedList(new ArrayList<>());

    String answer = chatClient.prompt()
            .messages(new SystemMessage(SupportPrompts.system(context)), new UserMessage(question))
            .tools(new OrderTools(trace))
            .call()
            .content();
    ...
    return new AgentResponse(answer.trim(), sources, List.copyOf(trace), context);
}

// WHY a new list per call: this object serves every request. A tracker
// shared across requests would mix two customers' tool calls together.
const toolCalls = [];
const input = [{ role: "user", content: question }];

for (let round = 1; round <= MAX_TOOL_ROUNDS; round += 1) {
  const response = await client.responses.create({ model, instructions, input, tools: definitions });
  const calls = response.output.filter((item) => item.type === "function_call");

  if (calls.length === 0) {
    const answer = response.output_text?.trim();
    if (!answer) throw new Error("the model returned an empty answer");
    return { answer, retrievedSources: policies.map((policy) => policy.source), toolCalls, context };
  }

  input.push(...response.output);
  for (const call of calls) {
    const { argument, result } = runTool(call);
    toolCalls.push({ name: call.name, argument, result });
    input.push({ type: "function_call_output", call_id: call.call_id, output: result });
  }
}
throw new StepLimitError(MAX_TOOL_ROUNDS);

async def chat(self, question: str) -> ChatResult:
    policies = await self._knowledge_base.search(question)
    context = format_context(policies)

    # WHY a new list per call: a tracker shared by the whole app would mix
    # the tool calls of two customers who ask at the same time.
    tool_calls: list[ToolCall] = []
    answer = await self._run_tool_loop(system_prompt(context), question, tool_calls)

    return ChatResult(answer, [policy.source for policy in policies], tool_calls, context)

In Spring, the tool loop is run by Spring AI. You hand it the tools with .tools(...), and it sends their definitions, runs the method the model asks for, and repeats. So the trace has to be filled from inside the tools. OrderTools is a plain class, not a Spring bean, and a new one is built per request with that request's empty list:

/** One place for the clean, look up and record steps, so the three tools cannot drift apart. */
private String lookUp(String toolName, String rawOrderId, Map<String, String> table) {
    String orderId = normalise(rawOrderId);
    String result = table.getOrDefault(orderId, NOT_FOUND);
    trace.add(new ToolCall(toolName, orderId, result));
    return result;
}

In Node and Python the loop is written by hand (as in note 5), so the trace is just a local list inside chat. Note that every version records the cleaned order id. The model sometimes sends ord-101 with a space. The tool trims it and upper-cases it, and the trace stores ORD-101, so the toolArgument check compares like with like.

The tool round limit differs a little between stacks. Node and Python stop after 5 trips to the model. Spring AI 2.0 has no per-call round limit, so the Spring version caps the total tool calls per request with spring.ai.tools.limits.max-total-tool-calls: 5 in application.yml. Every round makes at least one tool call, so the effect is the same.

29.2 The four checks

/** No tool expected means no tool at all. Otherwise the expected tool must be among the calls. */
static boolean toolChoice(GoldenCase goldenCase, List<ToolCall> calls) {
    if (goldenCase.expectedTool() == null) {
        return calls.isEmpty();
    }
    return calls.stream().anyMatch(call -> call.name().equals(goldenCase.expectedTool()));
}

/** The right tool with the wrong order id is still a wrong answer, so both must match on one call. */
static boolean toolArgument(GoldenCase goldenCase, List<ToolCall> calls) {
    if (goldenCase.expectedTool() == null) {
        return true;
    }
    return calls.stream().anyMatch(call -> call.name().equals(goldenCase.expectedTool())
            && Objects.equals(call.argument(), goldenCase.expectedToolArgument()));
}

static boolean source(GoldenCase goldenCase, List<String> retrievedSources) {
    return goldenCase.expectedSource() == null || retrievedSources.contains(goldenCase.expectedSource());
}

/** No tool expected means no tool called. Otherwise the expected tool must be among the calls. */
export function toolChoice(testCase, reply) {
  if (testCase.expectedTool === null) return reply.toolCalls.length === 0;
  return reply.toolCalls.some((call) => call.name === testCase.expectedTool);
}

/** The right tool with the wrong order id is still wrong: it answers about another customer's order. */
export function toolArgument(testCase, reply) {
  if (testCase.expectedTool === null) return true;
  return reply.toolCalls.some(
    (call) => call.name === testCase.expectedTool && call.argument === testCase.expectedToolArgument,
  );
}

export function source(testCase, reply) {
  if (testCase.expectedSource === null) return true;
  return reply.retrievedSources.includes(testCase.expectedSource);
}

def tool_choice(expected_tool: str | None, tool_calls: list[ToolCall]) -> bool:
    # Extra calls to other tools are allowed: checking the status before
    # the delivery date is not wrong.
    if expected_tool is None:
        return not tool_calls
    return any(call.name == expected_tool for call in tool_calls)


def tool_argument(expected_tool: str | None, expected_argument: str | None, tool_calls: list[ToolCall]) -> bool:
    # WHY a separate check: the right tool with the wrong order id gives a
    # confident answer about somebody else's order.
    if expected_tool is None:
        return True
    return any(call.name == expected_tool and call.argument == expected_argument for call in tool_calls)


def source(expected_source: str | None, retrieved_sources: list[str]) -> bool:
    return expected_source is None or expected_source in retrieved_sources

This code is very ordinary, and that is on purpose. The most valuable checks in the whole runner are a few lines of the code you write every day.

29.3 The judge: strict shape, then a range check

scores = chatClient.prompt()
        // Messages, not .system()/.user(): the rubric and the answer are not templates.
        .messages(new SystemMessage(rubric), new UserMessage(evidence(goldenCase, response)))
        .advisors(AdvisorParams.ENABLE_NATIVE_STRUCTURED_OUTPUT)
        .call()
        .entity(JudgeScores.class);
...
/** Strict mode promises integers, not integers from 1 to 5. */
static Verdict check(JudgeScores scores) {
    if (scores == null) {
        return Verdict.unusable(UNREADABLE_REPLY);
    }
    String outOfRange = scores.byMetric().entrySet().stream()
            .filter(entry -> entry.getValue() < MIN_SCORE || entry.getValue() > MAX_SCORE)
            .map(entry -> entry.getKey() + " " + entry.getValue())
            .collect(Collectors.joining(", "));
    if (!outOfRange.isEmpty()) {
        return Verdict.unusable("the judge gave scores outside " + MIN_SCORE + " to " + MAX_SCORE + ": " + outOfRange);
    }
    return Verdict.usable(scores);
}

const response = await client.responses.parse({
  model,
  instructions: rubric,
  input,
  text: { format: VERDICT_FORMAT },
});
...
const isValidScore = (value) => Number.isInteger(value) && value >= MIN_SCORE && value <= MAX_SCORE;

/** The meaning check. Strict mode can return 0 or 7 and still match "integer". */
export function checkVerdict(verdict) {
  const problems = JUDGE_METRICS.filter((metric) => !isValidScore(verdict[metric])).map(
    (metric) => `${metric} must be a whole number from ${MIN_SCORE} to ${MAX_SCORE}, got ${verdict[metric]}`,
  );
  if (problems.length > 0) throw new JudgeError(`The judge gave invalid scores: ${problems.join("; ")}`);

  const scores = Object.fromEntries(JUDGE_METRICS.map((metric) => [metric, verdict[metric]]));
  return { scores, reason: String(verdict.reason ?? "").trim() };
}

response = await self._client.responses.parse(
    model=self._model,
    instructions=self._rubric,
    input=judge_input(case, result),
    text_format=JudgeScores,
)
...
def check_scores(raw: dict[str, Any]) -> dict[str, int]:
    """Every metric present and a whole number from 1 to 5, or JudgeError."""
    scores: dict[str, int] = {}
    for metric in METRICS:
        value = raw.get(metric)
        # bool is a subclass of int in Python, so True would pass as 1.
        is_whole_number = isinstance(value, int) and not isinstance(value, bool)
        if not is_whole_number or not MIN_SCORE <= value <= MAX_SCORE:
            raise JudgeError(f"{metric} must be a whole number from {MIN_SCORE} to {MAX_SCORE}, got {value!r}")
        scores[metric] = value
    return scores

The judge can use a different model from the agent (JUDGE_MODEL). Using a separate, often stronger, model means the agent is not simply agreeing with itself.

29.4 One bad case does not stop the run

If the agent throws on one case (a provider hiccup, the tool limit), or the judge reply cannot be used, only that case fails, with the reason in the report. The other eleven still run. Node shows the idea most clearly:

export async function evaluateCase({ testCase, agent, judge }) {
  let reply;
  try {
    reply = await agent.chat(testCase.question);
  } catch (error) {
    return caseResult(testCase, { reason: `The agent failed: ${error.message}` });
  }

  const checks = runChecks(testCase, reply);
  try {
    const { scores, reason } = await judge.score(testCase, reply);
    return caseResult(testCase, { checks, scores, reason });
  } catch (error) {
    return caseResult(testCase, { checks, reason: `The judge failed: ${error.message}` });
  }
}

And the pass rule, the same in all three:

/** All four checks pass and every score is at least PASS_SCORE. No scores means fail. */
export function casePassed(checks, scores) {
  if (checks === null || scores === null) return false;
  return Object.values(checks).every(Boolean) && JUDGE_METRICS.every((metric) => scores[metric] >= PASS_SCORE);
}

A case the judge could not score never passes, and it is left out of the averages, so one broken reply cannot drag the numbers down or up.

29.5 How the three map

29.6 Where to look in the code

29.7 What this demo leaves out on purpose

30. Reading a Report

A run prints one block per case, then a summary:

========================================
combined-return-01   PASS
  question   Can I still return ORD-102?
  checks     answerPresent PASS  toolChoice PASS  toolArgument PASS  source PASS
  scores     relevance 5  correctness 5  faithfulness 5  helpfulness 4  hallucinationFree 5
  judge      Helpfulness lost a point: the answer does not say how to start the return.
========================================
rag-return-edge-01   FAIL
  question   Can I still return it on day 30?
  checks     answerPresent PASS  toolChoice PASS  toolArgument PASS  source PASS
  scores     relevance 5  correctness 3  faithfulness 4  helpfulness 4  hallucinationFree 5
  judge      Correctness lost points: "day 30 may be too late" disagrees with the 30 day window.
...
========================================
FINAL REPORT
  cases   12   passed 11   failed 1
  failed  rag-return-edge-01
  average relevance 4.92  correctness 4.67  faithfulness 4.83  helpfulness 4.42  hallucinationFree 4.92

How to read a failure:

1. Look at the checks first. All four passed here, so retrieval found the right file and no tool was misused. The problem is in generation.
2. Then the scores. Only correctness is low.
3. Then the reason. The model hedged on the boundary: "may be too late". The fix is in the prompt, for example a line saying the window includes the last day, or in the policy text itself.
4. Run it again before and after the fix. One run is one sample. If the case fails on one run in five, that is a flaky edge, and worth knowing too.

Your numbers will not match these exactly, and will move a little between runs. That is normal. Watch the averages and the list of failed ids over time.

31. Why AI Evals Are Different From Normal Tests

A normal test looks like this:

An agent looks like this:

So a useful eval strategy tests both:

the behaviour of the system   (the trace, checked with code)
          +
the quality of the answer     (the reply, scored by a judge)

That is why observability, being able to see what the agent did, is a core part of building a production AI app, not an extra.

32. What Breaks in Production

33. The Full Mental Model

In one line:

make the agent show its work + check the work with code + score the words with a judge = an agent you can change safely

Interview Questions

"Why isn't testing the final answer of an agent enough?"

An agent is a pipeline of retrieval, tool choice, tool arguments and generation, and a right answer can come from wrong steps, such as a lucky guess with no tool call. If you only test the answer, you cannot tell which step failed, and a guess that works today breaks on the next order. You need a trace of what the agent did.

"What would you put in an agent's trace for evaluation?"

The retrieved sources, each tool call with its name, argument and result, and the exact context sent to the model, next to the answer. Sources and tool calls feed deterministic checks. The context and tool results are the evidence the judge needs to score faithfulness. Build the trace per request, never in a shared object, or concurrent requests mix their calls.

"Which checks would you write in code, and which would you give to an LLM judge?"

In code: an answer exists, the expected tool was called, with the expected argument, and the expected document was retrieved. They have exact answers and cost nothing. The judge gets what needs meaning: relevance, correctness against a golden answer, faithfulness to the evidence, helpfulness and invented details.

"How do you make an LLM judge's output usable by code?"

Ask for strict structured output with an integer per metric and a short reason, then check the range in code, because the schema guarantees an integer but not one from 1 to 5. Name every metric so higher is better, for example hallucinationFree, so one threshold and one average work for all of them.

"How do you pick a pass threshold?"

It is a product decision tied to risk, not a constant. A support bot might need every score at 4 or more. A medical or financial assistant might need 5 on correctness and faithfulness plus human review. A creative tool might ignore faithfulness. Agree it with the people who own the risk.

"How is an eval run different from a unit test in the same project?"

Unit tests use a fake model, need no key, are free and give the same result every time, and they prove your code works: checks, tools, loop and report. The eval run uses the real model, retrieval and tools on the golden dataset, costs a little, varies between runs, and proves the whole system behaves well. You need both.

"An eval case fails. How do you find the cause?"

Read the deterministic checks first: a failed source check points at retrieval, a failed tool or argument check points at the prompt or tool descriptions. If all checks pass, look at which judge metric is low and read the judge's reason, which usually points at the prompt or the model. Then run the case again, since one run is one sample.

Key Takeaways

• A right answer can hide wrong steps. Test the behaviour, not only the words.
• Make the agent show its work: return a trace of sources, tool calls with arguments and results, and the context.
• Build the trace inside each request. A shared tracker mixes customers' calls under load.
• Keep a shared golden dataset with the expected tool, argument, source and answer, including edge, out of scope and adversarial cases.
• Run the real agent in evals. Use fakes in unit tests. They answer different questions.
• Four deterministic checks catch the objective failures for free: answer present, tool choice, tool argument, source.
• A judge scores meaning: relevance, correctness, faithfulness, helpfulness and hallucination-free, all with 5 as good.
• Strict JSON fixes the judge's shape. Your code still checks each score is from 1 to 5.
• A case passes only when both layers pass. The threshold is a product decision.
• One bad case should fail that case, not the whole run. Read failures checks first, then scores, then the judge's reason.