Testing an AI Agent: Deterministic Evals and LLM-as-a-Judge

This topic builds directly on Topic 17, GenAI Evals: How to Know Your AI App Works. Last time, we learned what evaluations (evals) are. Now we’ll build a testing approach for an AI agent that uses RAG + tools + an LLM.

We’ll use the ShopNest online store example from your notes and focus on JavaScript/Node.js, with practical code snippets.

1. The problem: a correct answer can still hide a bug

Imagine a customer asks:

Can I still return ORD-102?

ShopNest has two rules and facts:

• Return policy: a product can be returned within 30 days of delivery.
• Order information: ORD-102 was delivered 20 days ago.

The agent replies:

Yes, you can still return it.

The answer is correct. But what if the agent never checked the order and simply guessed?

The main lesson: testing only the final answer is not enough. We must test both the agent’s behaviour and the quality of its answer. Source: Topic 18, sections 1–2.

2. First, understand the system we are testing

Our ShopNest agent handles three kinds of questions.

Here is the architecture:

RAG supplies company knowledge; tools supply data that changes, such as a particular order’s status. The LLM combines the relevant information into an answer.

The testing challenge is that each stage can fail independently.

• RAG retrieves the wrong document.
• The agent chooses the wrong tool.
• The correct tool receives the wrong order ID.
• The model receives correct evidence but generates an incorrect answer.
• The model adds details that were never supported by the evidence.

That is why the notes introduce two complementary testing methods: deterministic checks and an LLM judge.

3. What is a trace, and why do we need one?

A trace is a record of what the AI agent did while answering one request.

Without a trace, the API might return only:

{
  "answer": "Yes, you can return it."
}

As a tester, you cannot tell whether the agent checked the policy, called the order tool, or guessed.

Instead, return the answer alongside internal debugging information:

{
  "answer": "Yes. ORD-102 was delivered 20 days ago, within the 30-day return window.",
  "retrievedSources": ["return-policy.md"],
  "toolCalls": [
    {
      "name": "getOrderStatus",
      "argument": "ORD-102",
      "result": "Delivered 20 days ago"
    }
  ],
  "context": "Products can be returned within 30 days of delivery."
}

Let's understand each field.

A trace is like a backend request log

Imagine a normal API request:

POST /orders/ORD-102
    ↓
Controller
    ↓
Service
    ↓
Database
    ↓
Response

If the API fails, you investigate the steps. You don't just look at the final response.

AI agents need the same idea, except some decisions are made probabilistically by an LLM.

Important production rule: keep traces for debugging and evaluation, but don't expose internal context or tool results to customers by default. They may contain private information. Log them with appropriate access controls and redaction.

4. Deterministic evals: use normal code for objective checks

A deterministic eval is a check that ordinary code can reliably answer with a pass or fail.

For example:

• Was the answer empty?
• Did the agent call getOrderStatus?
• Did it pass ORD-101?
• Did RAG retrieve return-policy.md?

We don't need another AI model to answer these questions. JavaScript can do it exactly.

The four checks in Topic 18

Let's implement them.

Check 1: Is the answer present?

function answerPresent(reply) {
  return typeof reply.answer === "string" &&
    reply.answer.trim().length > 0;
}

If the agent returns an empty string, this test fails.

Check 2: Did the agent call the correct tool?

function toolChoice(testCase, reply) {
  if (testCase.expectedTool === null) {
    return reply.toolCalls.length === 0;
  }

  return reply.toolCalls.some(
    (call) => call.name === testCase.expectedTool
 );
}

Suppose the question is:

Where is ORD-101?

The expected tool is getOrderStatus.

If the trace contains that tool call, the check passes.

One detail matters: if no tool is expected, no tool should be called. For a question about the general shipping policy, calling an order lookup tool is unnecessary and counts as incorrect behaviour in this evaluation.

Check 3: Did the tool receive the correct argument?

function toolArgument(testCase, reply) {
  if (testCase.expectedTool === null) {
    return true;
  }

  return reply.toolCalls.some(
    (call) =>
      call.name === testCase.expectedTool &&
      call.argument === testCase.expectedToolArgument
 );
}

Suppose the agent calls:

getOrderStatus("ORD-999");

But the customer asked about ORD-101.

The tool selection is correct, but the argument is wrong. The agent might accidentally return information about a different order.

That is why tool selection and tool arguments must be tested separately.

Check 4: Did RAG retrieve the expected document?

function source(testCase, reply) {
  if (testCase.expectedSource === null) {
    return true;
  }

  return reply.retrievedSources.includes(
    testCase.expectedSource
 );
}

If the expected source is return-policy.md but the agent retrieves shipping-policy.md, the check fails.

Changing the prompt may not fix this. The real problem could be retrieval configuration, chunking, embeddings, or the number of results retrieved.

Combine the four checks

function runChecks(testCase, reply) {
  return {
    answerPresent: answerPresent(reply),
    toolChoice: toolChoice(testCase, reply),
    toolArgument: toolArgument(testCase, reply),
    source: source(testCase, reply)
  };
}

Example result:

{
  answerPresent: true,
  toolChoice: true,
  toolArgument: true,
  source: true
}

All four checks pass.

Remember: these checks establish that certain behaviours occurred. They do not prove that the final answer is semantically correct.

5. The golden dataset: questions with known expected behaviour

A golden dataset is a collection of test cases that describes how the agent should behave.

Think of it as a test suite designed specifically for an AI application.

In a regular function, you might write:

expect(add(2, 3)).toBe(5);

For an AI agent, you need more than an expected string. You may need the expected tool, argument, source, answer, and behaviour.

Example from the ShopNest notes:

{
  "id": "tool-status-01",
  "question": "Where is my order ORD-101?",
  "goldenAnswer": "ORD-101 has been shipped and has not been delivered yet.",
  "expectedBehaviour": "Call getOrderStatus for ORD-101 and report the status from the tool.",
  "expectedTool": "getOrderStatus",
  "expectedToolArgument": "ORD-101",
  "expectedSource": null
}

Here's what each field does:

• id: identifies the test case in the report.
• question: input given to the real agent.
• goldenAnswer: reference answer used to evaluate correctness.
• expectedBehaviour: describes the behaviour expected from the agent.
• expectedTool: tool that should be called; null means no tool should be called.
• expectedToolArgument: expected argument passed to that tool.
• expectedSource: document that should be retrieved; null means the source is not checked for this case.

A useful dataset should include more than easy, happy-path questions.

For example:

The adversarial case is particularly important: the agent should follow the actual policy rather than obeying a customer's attempt to override it.

6. LLM-as-a-Judge: use a second model to evaluate meaning

Deterministic checks are good at checking exact behaviours. But consider these two answers:

• “ORD-101 has been shipped and is on its way.”
• “Your order ORD-101 left our warehouse and hasn't arrived yet.”

The wording differs, but the meaning is similar.

A simple string comparison cannot reliably evaluate this. This is where LLM-as-a-Judge comes in.

An LLM-as-a-Judge system uses a second model to evaluate the agent's response against a defined rubric.

The judge should receive:

1. The original question.
2. The golden answer and expected behaviour.
3. The agent's generated answer.
4. The retrieved policy context.
5. The tool results.
6. The scoring rubric.

Why give it the context and tool results? Because the judge needs to determine whether the answer is supported by the evidence the agent actually saw.

For example, if the agent says:

You can start your return in the ShopNest mobile app.

But neither the policy nor the tool results mention a mobile app, the claim is unsupported, even if the app really exists outside the information provided to the judge.

Rule: the judge should evaluate the answer using the specified evidence, not invent facts from its own general knowledge.

7. The five metrics used by the judge

The notes use five metrics, each scored from 1 to 5. A higher score is always better.

Let's use the same question:

Can I still return ORD-102?

Expected evidence: delivered 20 days ago; return window is 30 days.

Answer B may receive a high relevance score because it answers the question, but a lower faithfulness and hallucinationFree score because it adds unsupported details.

That distinction is valuable: a response can be relevant yet still be untrustworthy.

Why name the metric hallucinationFree?

The notes deliberately use hallucinationFree, where 5 means good, rather than a metric called hallucination, where 5 might mean bad.

This keeps all metrics aligned:

relevance          5 = good
correctness        5 = good
faithfulness       5 = good
helpfulness        5 = good
hallucinationFree  5 = good

Now one threshold can be used consistently across all five metrics.

8. The judge needs a rubric, not a vague instruction

A weak judge prompt might say:

Tell me whether this answer is good.

The problem is that “good” is vague. One evaluation might score an answer 4, while another scores the same answer 2.

Instead, define the criteria and what each score means.

For example, the Topic 18 notes define faithfulness like this:

A complete rubric should explain:

• Which evidence the judge can use.
• What each metric measures.
• What scores 1–5 mean for each metric.
• How to compare the answer with the golden answer.
• How to treat unsupported claims.
• The required output format.

There is also an important subtlety from the notes: a polite refusal to answer an out-of-scope question can be the correct answer. A judge must not penalize it just because it doesn't directly answer an unrelated question.

9. Structured output: make the judge's response usable by code

Suppose the judge returns:

The answer is mostly correct, but it includes one unsupported claim.

That may be understandable to a person, but it is difficult for an automated test runner to process consistently.

Instead, ask the judge for structured JSON:

{
  "relevance": 5,
  "correctness": 5,
  "faithfulness": 4,
  "helpfulness": 5,
  "hallucinationFree": 4,
  "reason": "The answer includes a return-app instruction that is not supported by the evidence."
}

Your code can now read the scores and decide whether the case passes.

But there's a subtle bug to watch for.

Valid JSON does not necessarily mean valid scores.

For example:

{
  "relevance": 7,
  "correctness": 0,
  "faithfulness": 4,
  "helpfulness": 5,
  "hallucinationFree": 5,
  "reason": "Example invalid scores"
}

This is valid JSON. The scores are integers, but 7 and 0 are outside the allowed range.

So validate the output in your own code:

const METRICS = [
  "relevance",
  "correctness",
  "faithfulness",
  "helpfulness",
  "hallucinationFree"
];

function validateScores(scores) {
  for (const metric of METRICS) {
    const score = scores[metric];

    if (
      !Number.isInteger(score) ||
      score < 1 ||
      score > 5
   ) {
      throw new Error(
        `${metric} must be an integer from 1 to 5`
     );
    }
  }

  return scores;
}

This checks that each score is an integer in the range 1–5. In production, also validate the required fields and types of reason, and reject missing or malformed judge responses.

The general principle is:

• Schema validation checks the response's structure.
• Application validation checks that its values obey your rules.
• The rubric guides the judge's evaluation of meaning.

All three are needed.

10. The pass threshold: when is an answer good enough?

The ShopNest example uses:

const PASS_SCORE = 4;

A test case passes only if every judge score is at least 4.

For example:

All scores meet the threshold, so the judge layer passes.

But if correctness is 3, the case fails, even if the average score is high.

Why? Because a strong score for helpfulness cannot compensate for an incorrect answer.

The threshold depends on the product

These are product decisions, not universal laws. The appropriate threshold depends on the risks of getting an answer wrong.

11. Put both layers together

Now we can define the complete rule for a passing case:

Here is a simple Node.js implementation:

const PASS_SCORE = 4;

function casePassed(checks, scores) {
  // Missing checks or unusable judge scores must fail.
  if (checks === null || scores === null) {
    return false;
  }

  const allChecksPassed =
    Object.values(checks).every(Boolean);

  const allScoresPassed =
    METRICS.every((metric) =>
      scores[metric] >= PASS_SCORE
   );

  return allChecksPassed && allScoresPassed;
}

Notice the use of &&, not ||.

• && means both conditions must be true.
• || means either condition could be true.

We want && because a correct-looking answer must not pass if the agent skipped a required tool call.

Common failure patterns and what to fix

This is one of the most useful benefits of the approach: the report helps identify where the problem happened, rather than merely reporting that a test failed.

12. How the complete eval runner works

The eval runner executes the test cases one by one.

Unit tests versus AI evals

The notes make an important distinction between the two.

You need both.

For example, a unit test can prove your toolArgument() function correctly detects a wrong order ID. It cannot prove that the real LLM will choose the correct tool when a customer phrases the question in an unexpected way.

What a report might look like

combined-return-01   PASS
  checks: all four passed
  scores: relevance 5, correctness 5,
          faithfulness 5, helpfulness 4,
          hallucinationFree 5

rag-return-edge-01   FAIL
  checks: all four passed
  scores: relevance 5, correctness 3,
          faithfulness 4, helpfulness 4,
          hallucinationFree 5
  reason: The answer incorrectly suggests
          day 30 may be too late.

FINAL REPORT
  cases: 12
  passed: 11
  failed: 1

How would you debug the failed case?

1. Check the deterministic checks first. All passed, so the expected document and tool behaviour were correct.
2. Look at the scores. Correctness is low.
3. Read the judge's reason. The model mishandled the 30-day boundary.
4. Update the relevant prompt or policy wording, then rerun the case.

One run is only one sample. If the same case passes four times and fails once, that variability is worth investigating.

13. Production lessons from Topic 18

The example is small, but its production lessons are important.

A. Never share a mutable trace between requests

Imagine two customers asking questions simultaneously:

Customer A → ORD-101
Customer B → ORD-102

If both requests write to the same global toolCalls array, their traces can become mixed.

That can cause misleading evaluations and potentially expose one customer's data to another request.

Instead, create a new trace for each request:

async function chat(question) {
  // Every request gets its own trace.
  const toolCalls = [];

  // Retrieval, tool loop and generation use this trace.
  // Return the answer and this request's trace.
}

Keep the trace request-scoped, not in a shared singleton or global mutable variable.

B. Run evals automatically

A team may build an eval runner and then forget to run it.

A better approach is to run a fast subset in continuous integration (CI) for pull requests that modify prompts, tools or retrieval. Run the full suite before a release.

C. Expect some variability

The same question may receive slightly different answers across runs. Don't simply remove a failing test because it is inconvenient. Repeat the case and investigate whether the behaviour is genuinely unstable.

D. Keep the judge consistent

If you upgrade the judge model, its scores may change even when the agent itself hasn't changed.

Pin the judge model version where possible, and periodically compare its scores with human evaluations.

E. Maintain the golden dataset

If the return policy changes from 30 days to 45 days, the expected answers must change too. Otherwise, the test suite will incorrectly mark valid answers as failures.

Keep the dataset aligned with the actual policy and give it a clear owner.

F. Watch individual failures, not only averages

An average correctness score of 4.7 may look excellent, while every refund-related case is failing.

Break down results by category (policy, tools, combined questions, edge cases and adversarial questions) and track the individual failed case IDs.

G. Control cost

A full suite that calls the agent and judge for every case can become expensive. Consider a cheaper judge for routine runs, a stronger judge before releases, and caching agent responses when only the judge configuration changes.

One more production distinction: the demo exposes traces for easy inspection, but a real application should generally log them securely with request IDs rather than return internal traces directly to the customer.

14. Interview revision notes

These are the main questions you should be able to answer after this topic.

"Why isn't testing only the final answer enough?"

An agent can produce the correct answer by guessing, without retrieving the correct document or calling the required tool. A trace lets us test the internal behaviour as well as the answer.

"What should an agent trace contain?"

The retrieved sources, tool calls with their names, arguments and results, the context supplied to the model, and the final answer.

"What is the difference between deterministic evals and LLM-as-a-Judge?"

Deterministic evals use ordinary code for objective checks, such as whether the expected tool was called. An LLM judge evaluates meaning, such as correctness, relevance, faithfulness and helpfulness.

"What is a golden dataset?"

A collection of known test questions with expected answers and behaviours, including expected tools, arguments and retrieval sources where applicable.

"Why use structured JSON for the judge?"

It makes the scores machine-readable. But the application must still validate the fields and ensure every score is an integer between 1 and 5.

"When does an eval case pass?"

In this example, all four deterministic checks must pass, and every judge score must be at least 4. If either layer fails, the case fails.

"How is an AI eval different from a unit test?"

A unit test usually uses a fake model to test your code predictably. An AI eval uses the real model, retrieval and tools to test the behaviour of the complete system.

"Why should a trace be created per request?"

To prevent concurrent requests from mixing their tool calls and potentially leaking information across customers.

15. Test your understanding

1. The agent gives the correct answer but never calls the required order tool. What should happen?

   • Pass because the final answer is correct
   • Fail the deterministic tool-choice check
   • Skip the judge and pass

2. The correct tool is called, but with ORD-999 instead of ORD-101. Which check catches this?

   • answerPresent
   • source
   • toolArgument

3. The expected document is retrieved, but the model gives an incorrect answer. Where should you investigate?

   • Generation, prompt or model behaviour
   • Only the tool argument check
   • The answer-present check

4. The judge returns valid JSON with a correctness score of 7. What should your runner do?

   • Accept it because the JSON is valid
   • Clamp it to 5 automatically
   • Reject the invalid score

5. Why should each request have its own trace?

   • To make the LLM more deterministic
   • To avoid mixing tool calls between concurrent requests
   • To avoid creating a golden dataset

Final mental model

Remember this formula:

The key lesson from Topic 18 is that reliable AI testing means checking both what the agent did and what the agent said.