GenAI Evals: How to Know Your AI App Works

On a Friday evening, a developer at ShopNest, an online shop, makes a small change to its support bot. The bot is a RAG bot like the one in note 11. Retrieval was sending the top 5 chunks to the model. They change it to the top 3. Replies get about 300 ms faster and the token bill drops. They try three questions by hand. All three answers look fine. They ship it.

On Monday there are 40 angry tickets. Customers with damaged parcels are being told they cannot return them. The rule for damaged items lives in a chunk that used to come back fourth. With the top 3, the model never sees it. Nothing crashed. No error was logged. The bot just started giving a confident wrong answer to one kind of question.

Nobody did anything silly here. The team simply had no way to measure the bot. "I tried three questions and they looked fine" was the whole test.

So far this series has been about making the AI do things: chat, call tools, search documents, return JSON, use MCP servers, read images. This note asks the next question, the one a team has to answer before it ships anything: how do we know the system is doing its job? The answer is evaluation, or evals. This note is the theory. Note 18 builds a real eval runner for a support agent in Spring Boot, Node.js and Python.

Core Path

If you only have an hour, read it in this order:

1. Why normal tests break (sections 1 to 5). The same question can get many correct answers, so expected == actual stops working. We test behaviour instead of exact text.
2. What an eval is (sections 6 to 11). A test input, the behaviour we expect, and something that judges it. A fixed set of these is your eval dataset.
3. Two kinds of judging (sections 12 to 21). If plain code can decide it (valid JSON, the right tool, a status code), use code. If it needs understanding (is this answer correct, faithful, helpful), use a quality metric.
4. LLM-as-a-judge (sections 22 to 28). A second model scores the answer against a rubric and returns structured scores. It is useful and it is not perfect.
5. Where evals run (sections 29 to 35). Humans, RAG split into two parts, offline evals before you ship, online evals after, and the loop that joins them.

Section 36 is deeper production detail: what goes wrong with eval setups in real teams. Section 37 sums up the whole note.

Reference: terms used in this chapter

1. Testing Normal Code Is Easy

Here is a function every developer has tested:

int add(int a, int b) {
    return a + b;
}

The test is one line:

assertEquals(5, add(2, 3));

This works because the function is deterministic. The same input goes through the same logic and gives the same output, every single time.

Most of the software you have tested so far behaves like this. A login check, a tax calculation, a sort. One right answer, and a test that compares against it.

2. An LLM Does Not Give One Right Answer

Now a customer asks the ShopNest bot:

What is the return policy?

The policy document says:

Products can be returned within 30 days of delivery.

Run the bot three times and you might get:

You can return your product within 30 days of delivery.

Our return window is 30 days after the product is delivered.

Customers have up to 30 days from delivery to start a return.

All three are fine. A customer would be happy with any of them. But try writing the test:

assertEquals("You can return your product within 30 days of delivery.", answer);

It passes on run one and fails on runs two and three, even though nothing is wrong. The old assumption, expected output == actual output, breaks. There is no single correct sentence to compare against.

3. Test the Behaviour, Not the Sentence

So the question changes. We stop asking "did it return this exact string?" and start asking:

Did the system show the behaviour we wanted?

For the return question, the behaviour we want is: "tells the customer the window is 30 days from delivery, and does not invent anything else". All three answers above show that behaviour. An answer that says "90 days" does not, however nicely it is written.

The variation is not a bug. It is how generative models work: they pick each next token from a probability list (note 2). Our job is to check the behaviour stays right across that variation.

4. "Looks Good to Me" Is Not Testing

In the first weeks of a project, testing usually looks like this:

With five questions, that is fine. Now count what can change in a RAG chatbot:

• the system prompt
• the model, or the model version
• temperature
• chunk size and chunk overlap
• how many chunks are retrieved (top K)
• metadata filters
• tool descriptions
• agent instructions
• the embedding model

Say you have 500 questions you care about. After each of those changes, would you read all 500 answers again? Nobody does. So people read five, feel good, and ship. That is how the Friday story happens.

5. AI Apps Have Regression Bugs Too

The top K change is worth drawing, because it is the most common kind of AI bug in real teams.

The change made most answers faster and cheaper. It made one kind of answer wrong. Three hand-picked questions would almost never find it.

That is a regression: something that worked and broke after a change. Normal software has regression tests for exactly this. AI apps need the same thing. The problem is not only "LLMs sometimes hallucinate". The problem is:

How do we measure, in a repeatable way, how a system that is not deterministic behaves?

6. What an Eval Is

An eval is a repeatable way to measure whether an AI system behaves the way we want. Think of it as a unit test with a wider idea of "pass".

The big difference is in the last row. An eval often measures properties of an answer instead of comparing exact text.

Every eval, however fancy, has three parts:

1. A test input. The thing you send to the app. "Can I return my shoes after 20 days?"
2. The expected behaviour. What a good system should do. "Say that 20 days is inside the 30 day window." Depending on the app, this can also be "should call weatherTool" or "should return valid JSON".
3. An evaluator. Something that decides if the behaviour happened. It can be plain code, a person, or another LLM.

Keep two jobs apart in your head. The application produces the answer. The evaluator judges it. If the same code does both, you are marking your own homework.

7. Test the Whole System, Not Just the Model

An AI app is much more than one model call. Here is a RAG bot again:

When the answer is wrong, any box could be the cause:

So when we say "evaluate the AI", we mean the whole system: retrieval, prompt, tools and model together. This matters even more for agents, where one wrong tool choice early can spoil every step after it.

8. The Eval Dataset

To compare two versions fairly, both must answer the same questions. If you make up new questions each time, you cannot tell whether the score moved because of your change or because of the questions.

So you keep a fixed list. That list is your eval dataset. For the ShopNest bot, it might start like this:

1. What is the return period?
2. Can I exchange damaged shoes?
3. Can I return something after 60 days?
4. How long does a refund take?
5. Can I return a product without the original box?

Now every version runs the same five:

Same questions, two versions, two numbers you can compare. The Friday bug shows up as "Q2 failed" before anyone ships.

9. The Golden Dataset

A stronger version stores a known good answer next to each question. That is a golden dataset. A domain expert, such as someone from the support team, writes or approves each golden answer.

Now the evaluator has something to compare against. It still does not compare word for word. It checks whether the generated answer means the same thing as the golden one.

What goes in a golden case depends on what the app does:

The RAG row is the useful one. Because it stores the expected document, you can test retrieval on its own, separate from the answer. Section 31 comes back to this.

10. Where Eval Data Comes From

On day one, the cases come from people who know the business:

• developers, who know the edge cases in the code
• domain experts, who know what customers really ask and what the right answer is
• the product team, who know what the feature must never do

After launch, the best source is production. Real users find the failures you never thought of.

So the dataset grows over time:

small eval dataset  ->  production failures  ->  more test cases  ->  a better eval dataset

A mature eval dataset is never finished. Every bad answer a customer reports should end up as a test case, so that exact mistake can never come back unnoticed.

11. What a Good Eval Dataset Contains

A common mistake is a dataset full of easy questions:

What is the return policy?
What is the refund policy?
What is the exchange policy?

All three are the textbook question, worded the textbook way. A bot can pass all three and still fail half of real traffic. A useful dataset mixes these kinds of cases:

The rule is simple:

If the dataset has no hard cases, you have no idea whether the app can handle hard cases.

That is not new. Good unit tests for normal code also cover empty input, the boundary value and the bad input, not only the happy path.

12. Two Kinds of Evaluation

Every check you write falls into one of two groups:

The next sections cover deterministic evals first, because they are cheaper, faster and more reliable. Then quality evals.

13. Deterministic Evals: When Code Can Decide

The most important rule in this whole note:

If plain code can judge it correctly, use plain code.

Do not ask another LLM a question that a normal if can answer perfectly. Many properties of an AI app are objective:

The refusal row shows something nice. If your app returns structured output (note 13) instead of free text, many more of its behaviours become things code can check. A structured app is easier to test.

Here is the "right tool" check in each language. It is the kind of code you already write every day:

// No expected tool means the model should not have called any tool.
static boolean toolChoicePasses(String expectedTool, List<String> toolsCalled) {
    if (expectedTool == null) {
        return toolsCalled.isEmpty();
    }
    return toolsCalled.contains(expectedTool);
}

// No expected tool means the model should not have called any tool.
export function toolChoicePasses(expectedTool, toolsCalled) {
  if (expectedTool === null) return toolsCalled.length === 0;
  return toolsCalled.includes(expectedTool);
}

def tool_choice_passes(expected_tool: str | None, tools_called: list[str]) -> bool:
    # No expected tool means the model should not have called any tool.
    if expected_tool is None:
        return not tools_called
    return expected_tool in tools_called

No model, no cost, no randomness. It gives the same verdict every run.

14. Performance Evals

Evals are not only about whether the answer is good. An AI app also has the same limits as any other service:

response time       < 5 seconds
tokens per request  < 4,000
cost per request    < INR 0.50
tool calls          <= 2

All of these are deterministic. You read them from the response and the usage numbers (note 3) and compare.

The last one matters most for agents. Picture an agent that answers "Where is ORD-101?" correctly, but only after 12 model calls: it looked up the order, then the delivery date, then the order again, then a refund it did not need. The answer is right. The system is still bad. It is slow, it costs twelve times what it should, and any one of those extra calls could have gone wrong. A "tool calls <= 2" check catches that, and a check of only the final answer never would.

15. Why Prefer Code Over a Model

Using an LLM to judge something code could judge adds four costs:

So:

Use the simplest evaluator that can judge the behaviour correctly.

Code first. A model only when meaning is involved.

16. Quality Is Not One Number

Some things code cannot decide:

• Is this answer helpful?
• Is it relevant to the question?
• Is it faithful to the documents we gave the model?

These need someone, or something, that understands language. That is a human, or another LLM acting as a judge.

Before we pick the judge, we need to say what "good" means. When someone says "that's a good answer", they might mean:

• it answered the question
• it was true
• it did not make anything up
• it was easy to understand
• it stuck to the documents it was given
• it had enough detail to act on

Those are different things. An answer can be strong on some and weak on others. A single GOOD = true hides which one failed. So AI quality is measured on several separate scales. Five cover most apps:

17. Relevance

Did the answer address what the user actually asked?

Question: "What is the return period?"

Products can be returned within 30 days.

That is very relevant. Now this one:

ShopNest was founded in 2018 and sells shoes, clothing and accessories.

It may be perfectly true. It does not answer the question. So:

correct information  is not the same as  relevant information

A simple scale:

18. Correctness

Relevance alone is not enough. The policy says 30 days. Compare:

Products may be returned within 30 days.    relevant and correct
Products may be returned within 90 days.    relevant and completely wrong

The second answer is on topic, well written and wrong. Correctness asks:

Is the information true, compared with our reference?

The reference can be a golden answer, a database value, a known fact, an answer a human labelled, or any other data you trust. Without a reference, nobody can score correctness, which is one more reason to build a golden dataset.

19. Faithfulness (Groundedness)

This one matters most for RAG. Suppose retrieval gave the model this:

Return Policy:
Products can be returned within 30 days of delivery.

And the model answered:

Products can be returned within 30 days.
Refunds are processed within 24 hours.

The first sentence is backed by the context. The second is not in the context at all. Even if ShopNest really does refund in 24 hours, a grounded RAG system should not say things its sources do not say. Today it happened to be right. Tomorrow the same habit invents a wrong fact.

Faithfulness asks:

Is every claim in the answer supported by the context the model was given?

Correctness compares with the truth. Faithfulness compares with the evidence the model had. They are different checks, and section 21 shows an answer that passes one and fails the other.

20. Helpfulness and Hallucination

Helpfulness. Question: "Can I return damaged shoes?" Answer:

Yes.

Relevant, correct and faithful. Also not much use. A more helpful answer, assuming the details are in the policy:

Yes. Damaged shoes can be returned within 30 days of delivery.
Open the order in the app and choose Return to start.

Helpfulness looks at clarity, enough detail, whether the customer knows what to do next, and not being longer than needed.

Hallucination. The model states something no source supports:

Context:  Products can be returned within 30 days.
Answer:   Products can be returned within 30 days, and a 15% restocking fee applies.

The restocking fee was invented. Many teams score this as hallucination-free instead of "hallucination", so that 5 means good on every metric. Then all five scales point the same way and you can average them without flipping one. Note 18 does exactly that.

21. The Metrics Are Independent

Here is a good answer scored on all five:

Context:   Products can be returned within 30 days of delivery.
           Refunds take 5 to 7 business days.
Question:  Can I return my headphones after 20 days?
Answer:    Yes. Products can be returned within 30 days of delivery,
           so 20 days is fine.

Now some answers that look fine on one scale and fail on another:

This table is why one number is not enough. Each failure needs a different fix: retrieval, the prompt, or the model.

22. LLM-as-a-Judge

Scoring meaning needs something that reads language. People are slow and expensive, so a practical option is a second LLM, the judge.

The first model does the task. The second model grades it. A tiny example:

Question:          Can I return shoes after 20 days?
Context:           Products can be returned within 30 days.
Generated answer:  Yes. Products can be returned within 30 days.

Judge instruction: Is the answer faithful to the context? Score 1 to 5.

Judge result:      Score: 5
                   Reason: Every claim in the answer is in the context.

That is LLM-as-a-judge. It is just another model call, with a very specific job.

23. What the Judge Needs to See

A judge can only judge what you show it. Each metric needs different inputs:

A common mistake is to ask for faithfulness without passing the context. The judge then scores from its own knowledge, which is exactly what we told the app not to do.

24. Make the Judge Return Structured Scores

Do not let the judge reply like this:

This answer seems pretty good, it covers the main point...

Your code cannot add that up. Ask for structured output (note 13):

{
  "relevance": 5,
  "correctness": 4,
  "faithfulness": 5,
  "helpfulness": 4,
  "hallucinationFree": 5,
  "reason": "Correct and grounded. It does not say how to start the return."
}

Now the result is data. Your code can read each score, apply a threshold, average over a dataset, store the run, and compare it with last week's run. The reason field is for you, the human, when you look at a failure.

And as always with structured output: the schema checks the shape, not the meaning. Your code should still check that each score is a whole number from 1 to 5.

25. Scoring a Whole Dataset

For 100 questions, the loop is:

At the end you get numbers like:

Average relevance      4.6 / 5
Average faithfulness   4.2 / 5
Average correctness    4.4 / 5
Passed                 91 of 100

That is a benchmark. Change the prompt, run it again, and you know whether things got better or worse, and on which metric. Faithfulness dropping from 4.2 to 3.6 after a prompt change tells you the new prompt made the model wander from its sources, before any customer sees it.

26. Pointwise vs Pairwise

There are two ways to ask a judge.

Pointwise: score one answer on its own.

Easy to understand, easy to average, and what most teams start with.

Pairwise: show two answers to the same question and ask which is better.

Pairwise is useful when you compare two prompts, two models, two RAG setups or two agent versions. Models are often better at "which of these two is better?" than at "is this a 3 or a 4?".

One known trap with pairwise: judges tend to prefer whichever answer comes first (or second, depending on the model). Run each pair twice with the order swapped, and only count a win when both runs agree.

27. Rubrics

A rubric says what each score means. This one is too vague:

5 = Excellent
4 = Good
3 = Acceptable
2 = Poor
1 = Incorrect

What makes an answer "good" rather than "excellent"? Two judges, or the same judge on two runs, will read that differently. A better rubric describes behaviour you can point at:

Faithfulness
5 = Every factual claim is directly supported by the context.
4 = Almost every claim is supported. One small unsupported detail
    that does not change the answer.
3 = The main answer is supported, but there are real unsupported claims.
2 = Large parts of the answer are unsupported.
1 = The answer contradicts the context or ignores it.

Now "it added a 15% restocking fee" clearly lands on 3 or below. A good rubric makes both human and LLM judges more consistent. Writing it is real work, and it is worth doing with the domain expert who owns the answers.

28. One Judge per Metric, or One for All?

You can ask for every metric in one call, or make one call per metric.

For learning and for most small teams, one call that returns all five scores is enough. Note 18 does that. If one metric matters a lot, such as faithfulness in a medical or finance bot, give it its own focused judge.

Where LLM judges go wrong

A judge is still an LLM. Known problems:

• It may use its own knowledge when it should only use the context you gave it.
• It can give a different score when you run it again.
• A weak judge model gives weak judgements.
• A vague rubric gives scores all over the place.
• It can have biases: preferring longer answers, preferring answers in its own style, or preferring the first answer in a pair.

So an LLM judge is not the final truth. A simple habit keeps it honest: every few weeks, have a person score 30 to 50 cases by hand and compare with the judge. If they agree most of the time, keep trusting the judge. If they do not, fix the rubric or change the judge model.

29. Human Evaluation

People are still the best judges of many things:

Humans are good at:

• subtle intent, such as a customer who is really asking for a refund while asking about delivery
• business rules nobody wrote down
• tone and style
• whether an answer would really help a real customer

But human review is expensive, slow, hard to scale, and sometimes inconsistent. Give the same answer to two reviewers:

Reviewer A: 4 / 5
Reviewer B: 2 / 5

So humans need good rubrics too. In practice, teams use people for the hardest cases, for building the golden answers, and for checking the LLM judge, not for scoring every run.

30. Who Should Judge What

A good eval setup uses all three, each where it is strongest.

31. Evaluating RAG in Two Parts

A RAG answer can go wrong in two very different places, so test them separately:

Retrieval evaluation. For "Can I return my product after 20 days?" the right source is return-policy.md. Did search return it? That is a deterministic check if your golden case stores the expected source. If retrieval fails, the model never had a chance. No prompt change will fix it. (Note 12 covers retrieval metrics such as recall at K.)

Response evaluation. When the right context did come back, score the answer with relevance, correctness, faithfulness, helpfulness and hallucination.

The split tells you where to look:

The Friday bug from the start would show up as "retrieval: fail" on the damaged-items case. That points straight at top K.

32. Offline Evaluation

Most evaluation during development is offline:

"Offline" means no real users are involved. You run stored test cases. Run them after:

• changing a prompt or system instructions
• switching models or model versions
• changing chunk size, overlap or top K
• changing tool descriptions
• changing the embedding model

Offline evals are controlled, repeatable, safe and good for comparing versions. They are the regression test layer of an AI app. Many teams run a small, fast set on every pull request and the full set before each release.

Their limit: a prepared dataset never fully looks like real users. You write 200 careful questions, and then on day one a customer types:

bro bought this yday but box gone can return???

None of your 200 questions looked like that.

33. Online Evaluation

Online evaluation measures the live system with real traffic.

Three kinds of signal:

1. Explicit feedback. Thumbs up and down, or "Was this helpful?". Clear, but few people click.
2. Implicit feedback. What users do next. Someone who asks the same question again right away probably did not get a useful answer. Someone who gives up halfway through a return, or asks for a human agent, is another signal. Noisier, but there is a lot of it.
3. Sampled judging. Send a small slice of real conversations, say 1%, through the same LLM judge and track the scores over time.

Online evals bring new concerns: user privacy, personal data in the transcripts, the cost of the judge, storage, compliance rules, and not slowing down the live reply. That is why production judging is usually sampled and runs after the reply is sent, not on every request in the request path.

34. Offline and Online Work as One Loop

Neither is enough alone. Together they form a loop:

Every production failure becomes a future regression test. Six months in, your eval dataset is full of the exact ways your app has failed real customers, which is far more valuable than anything you could have written on day one.

35. The Whole Evaluation Setup

Putting every piece together:

And a workflow you can follow every time you change the app:

1. Keep a fixed eval dataset.
2. Run the new version of the app on it.
3. Check the deterministic behaviour with code.
4. Score the meaning with an LLM judge, and with people where it matters.
5. Add up the results.
6. Compare them with the last version.
7. Ship only if quality is still good enough.
8. Watch real production traffic.
9. Add the important failures back into the dataset.

That turns AI work from "looks good to me" into "we measured it".

36. What Goes Wrong With Evals in Real Teams

37. The Full Mental Model

Traditional software engineering asks:

Does the code produce the expected result?

AI engineering asks:

Does the system keep showing the behaviour and quality we expect, across runs, across changes, and across the strange things real users type?

In one line:

fixed dataset + code for facts + judge for meaning + a loop from production = an AI app you can change without fear

Interview Questions

"Why can't you test an LLM app with assertEquals?"

Because the same question can get many different correct answers, worded differently on every run. Comparing exact text fails on good answers. Instead you test behaviour: did the answer contain the right fact, call the right tool, return valid JSON, stay within its sources.

"What is the difference between an eval dataset and a golden dataset?"

An eval dataset is a fixed list of test inputs you run on every change, so versions are compared on the same questions. A golden dataset also stores a known good answer or expected behaviour for each input, such as the expected tool and arguments or the expected source document, which gives the evaluator a reference to compare against.

"When would you use code to evaluate, and when an LLM judge?"

Use code for anything objective: valid JSON, the tool that was called, its arguments, the status code, latency, token count. Use an LLM judge only when meaning is involved, such as relevance, correctness or faithfulness. Code is free, fast and gives the same verdict every time, so never spend a model call on something an if statement can decide.

"What is the difference between correctness and faithfulness?"

Correctness compares the answer with the truth, usually a golden answer. Faithfulness compares it with the context the model was given. An answer can be true but unfaithful, when it states a fact that was not in the retrieved documents, and that habit will eventually produce false claims.

"How do you make an LLM judge reliable?"

Give it a rubric that describes behaviour for each score, give it all the evidence it needs and tell it to use nothing else, ask for structured scores with a reason, and check the scores in code. Then compare it with human scores on a sample every so often, and treat it as a measurement, not the final truth.

"How would you evaluate a RAG system?"

In two parts. Retrieval: did the expected document come back, which is a deterministic check if the golden case stores the source. Response: given the retrieved context, is the answer relevant, correct, faithful and helpful. The split tells you whether to fix retrieval or the prompt and model.

"What are offline and online evals, and why do you need both?"

Offline evals run a stored dataset before you ship, so they are repeatable and act as regression tests. Online evals measure live traffic through feedback, user behaviour and a sampled judge, which catches the questions your dataset never imagined. Failures found online go back into the offline dataset.

"What is pairwise evaluation and when is it useful?"

Instead of scoring one answer, the judge sees two answers to the same question and picks the better one. It is useful for comparing two prompts, models or RAG setups. Run each pair twice with the order swapped, because judges tend to favour one position.

Key Takeaways

• expected == actual breaks on LLMs. Test behaviour, not exact sentences.
• Every eval has three parts: an input, the expected behaviour, and an evaluator.
• Evaluate the whole system. Retrieval, prompt, tools and model can each be the cause.
• Keep a fixed eval dataset so versions are compared on the same questions. Store golden answers, expected tools and expected sources where you can.
• Fill the dataset with hard cases: paraphrases, edges, out of scope, missing information, attacks. Production failures are the best source.
• If code can judge it, use code. It is free, fast and the same every run.
• Quality has several dimensions: relevance, correctness, faithfulness, helpfulness and hallucination. One number hides which one failed.
• An LLM judge needs the evidence, a behaviour-based rubric and structured output. It is a measurement, not the truth. Check it against people.
• Split RAG evals into retrieval and response, so you know what to fix.
• Offline evals guard every change. Online evals find what you missed. Join them in one loop.