Advanced RAG: Chunking, Hybrid Search and Reranking

A customer bought headphones. They arrived with a cracked band. Ten days later she asked the shop's support bot: "Can I return my damaged headphones after 10 days?" The bot said yes, because the return window is 30 days. She packed the box and booked the pickup. The warehouse rejected the return two days later. Headphones are electronics, electronics have a 7 day window, and damage has to be reported within 48 hours. Now there is an angry customer, a wasted pickup that cost the shop about 120 rupees, and a support ticket that starts with "your own bot told me".

Nothing in that story is a model problem. The bot was fluent and polite. It answered from the text it was given. It was given the wrong text. The 30 day rule reached the model. The two rules that mattered did not.

Note 11 built a basic RAG bot: chunk the policies, embed them, find the closest chunks for each question and paste them into the prompt with citations. This note makes that bot hold up on harder questions. It covers:

• how documents are cut into chunks
• how many chunks to fetch, and when to fetch none
• how to filter by metadata
• how to find exact product codes that embeddings blur
• how to merge two kinds of search
• how to reorder results with a reranker
• how to lay out the final context so the model follows the right rule

The code is a small policy desk for a made-up shop called Kartly, written in Spring Boot, Node.js and Python under code/12-advanced-rag-retrieval/.

Core Path

Start with the failure, then add one repair at a time. For the product-code example, the steps are:

1. Exact code: a code such as AX-204 carries little meaning for an embedding. Vector search can pick a similar but wrong model.
2. BM25: add keyword search (it matches the actual words), so exact codes and rare words count strongly.
3. Hybrid search: run semantic vector search and BM25 together.
4. Reciprocal rank fusion (RRF): merge the two ranked lists by position, because their scores are on different scales.
5. Reranking: a slower model reads the question with a small set of candidates and reorders them. It cannot recover a chunk that retrieval never found, so fix retrieval first.

Sections 1 to 16 cover chunking, threshold, top K and metadata problems. Sections 17 to 24 walk through the steps above. Sections 25 to 30 go deeper: context layout, testing, provider choices and the runnable code. Do not add every upgrade by default. Look at the retrieved chunks, find the first stage that fails, and measure the repair.

Reference: terms used in this chapter

Use this table when a term first appears; the walkthrough defines each one in context.

1. Where Note 11 Left Us

Basic RAG is a straight line. A question comes in, it is turned into a vector with an embedding model (note 7), the vector store finds the closest chunk vectors (notes 9 and 10), and those chunks are pasted into the prompt next to the question.

Every box in this line can be good or bad on its own. The answer can only be as good as the worst box before it.

This works well for easy questions. "What is your return policy?" has one obvious chunk. The closest vector is the right one, the model reads it, and the answer is fine.

It quietly assumes the chunks that come back hold everything the answer needs. That breaks when a question needs two or three rules at once. Those are the questions that turn into complaints.

2. One Question, Three Rules

Kartly has these three rules, in three different files:

Now the question: "Can I return my damaged headphones after 10 days?"

A correct answer needs all three. Headphones are electronics, so the 7 day rule replaces the 30 day one. The product is damaged, so the 48 hour rule applies. Ten days have passed, so the damage report is late. The damage policy treats a late report as a normal return, and the 7 day electronics window is closed too.

The model did nothing wrong with what it saw. It was shown one rule out of three.

Given only the general chunk, the model reads "30 days", sees 10 days and says yes. Reasonable, and wrong. A model cannot follow a rule that never reached its prompt, and no system prompt fixes that.

3. Two Ways RAG Goes Wrong

When a RAG answer is wrong, there are only two places the mistake can live.

These need different fixes, and mixing them up wastes weeks. A sharper system prompt cannot recover a document that was not retrieved. Fetching twenty more chunks does not help a model that already had the right one and ignored it.

So the first debugging step is always the same: look at what was retrieved before you look at the answer.

Two branches, two different toolboxes. Most of this note is the left branch. The last part is the right one.

In practice the left branch is far more common. When a team says "the model is hallucinating", the retrieved chunks usually did not hold the answer.

4. Advanced RAG Is Upgrades, Not a New System

"Advanced RAG" sounds like a different architecture. It is not. It is the same line from section 1, with better versions of some boxes.

Read left to right, each box fixes one kind of miss. You do not need every box on day one.

Here is which box fixes which problem:

Start with the problem you can see in your own logs, not with the longest list of techniques.

5. Look Before You Fix

Before changing anything, get a baseline. Take twenty real questions whose answers you know are in the documents. For each one, print the chunks that came back, before the model is called.

With a Spring AI vector store, the retrieval call is a few lines, and printing the result is a few more:

List<Document> found = vectorStore.similaritySearch(
        SearchRequest.builder().query(question).topK(4).build());

for (int i = 0; i < found.size(); i++) {
    // Log the id and a short preview. Full chunk text in production logs
    // can leak customer data from the documents.
    log.info("#{} {} {}", i + 1, found.get(i).getId(), preview(found.get(i).getText()));
}

Do this once and you will find most of your problems in an afternoon.

The project for this note makes it an endpoint. POST /api/search returns every stage (semantic hits, keyword hits, the fused list, the reranked context) without calling the answer model. POST /api/ask runs the same retrieval and then answers. While you tune retrieval, you mostly use the first one.

A useful habit from production: log the chunk ids that went into every answer. When a complaint comes in a week later, you can see in one query whether the bot was shown the right rule.

6. Why Documents Get Chunked at All

Say Kartly has one 100 page policy document covering returns, refunds, shipping, warranty, exchanges and cancellations. Why not embed the whole document as one vector?

Embedding models accept only so much text at once, often around 8,000 tokens, and 100 pages is about 50,000. Worse, even if it fit, one vector would have to stand for every topic at once.

A question about the electronics return window is close to the electronics section. It is not very close to the blurry average of 100 pages.

So we cut documents into chunks: pieces small enough to be about one thing, each with its own vector. The real design question is not whether to chunk. It is where to cut, without cutting a rule away from the condition that changes it.

7. Fixed-Size Chunking

The simplest way is to cut every N tokens or characters. Spring AI ships a TokenTextSplitter that does this by token count, and most RAG libraries have one. It is quick, predictable, and blind to meaning.

Here is the electronics policy, shortened:

Electronics Return Policy

Electronic products can be returned within 7 days of delivery.
Products must be in their original packaging. Items with physical
damage caused by the customer are not eligible for return.
Customers must share the original purchase invoice.

A fixed-size cut can land anywhere. Say it lands after the second sentence:

The rule is in one chunk and its exception is in the other. The question needs both.

Chunk 2 does not even say "electronics". Its text starts with "Items with physical damage". Its vector knows it is about damage and returns, but not which product line. For a question about phones it may score below a general return chunk and never come back.

Fixed-size chunking is fine for flat text like call transcripts. For policies, where one sentence changes the previous one, it is the main source of retrieval failures.

8. Overlap

The usual patch for blind cuts is overlap: repeat the last part of each chunk at the start of the next one. If a condition sits right at a cut, it appears in both chunks.

Each new chunk starts 400 tokens after the last one, not 500. The 100 token overlap is stored twice.

Overlap is not free. With 500 token chunks and 100 tokens of overlap, each chunk moves forward only 400 tokens, so you get 500 / 400 = 1.25 times as many chunks. That is 25% more vectors to store, 25% more embedding cost at ingest, and a real chance that the top 4 results include two chunks that share a paragraph, which wastes prompt space.

A reasonable starting point is chunks of 400 to 600 tokens with 10 to 20% overlap. Then measure, and expect to move both numbers.

9. Structural Chunking: Cut Where the Author Did

Most business documents already have boundaries: titles, headings, numbered clauses, tables. The person who wrote the policy already grouped each rule with its exceptions under one heading. Structural chunking (also called document-aware chunking) cuts at those boundaries instead of at a token count.

Each chunk is one section, and the rule stays with its exceptions because the author put them together.

One detail matters more than it looks: keep the heading path in the chunk text. The heading path is the chain of headings above the text.

Return Policy > Electronics
Electronic products can be returned within 7 days of delivery. ...

Without the heading, "within 7 days" could be a return, refund, warranty or exchange window. With it, both the embedding and the keyword index know what the 7 days belong to. In the project, this line is also what lets keyword search match "return" against a body that only says "returned".

The rule that works in practice: keep the meaning together first, limit the size second. Cut at headings. If one section is still too long, cut it again by size, at sentence ends, with a little overlap. Note 11's chunker works this way, and so does this project's.

10. Semantic Chunking

Some documents have no headings at all: a long FAQ pasted as one block, an email thread, a scanned letter. Semantic chunking tries to find topic changes from the meaning itself.

Customers can return eligible products within 30 days.
Electronic products must be returned within 7 days.
Returned products are inspected before approval.
We ship to every serviceable PIN code in India.
Most orders are dispatched within 24 hours.
Express delivery is available in selected cities.

The idea: embed each sentence (or each small group), then compare each one with the next. Where the similarity drops sharply, the topic probably changed.

Two chunks come out: S1 to S3 about returns, S4 to S6 about shipping. The scores here are made up to show the shape.

It sounds clever, and sometimes it is. The costs are real though. Every sentence needs an embedding at ingest, so a 100 page document with about 3,000 sentences means 3,000 extra embeddings. In money that is tiny (around 60,000 tokens), but ingest gets slower and the "sharp drop" threshold is one more number to tune per document type. It also does not know about tables, "see section 4.2" references, or an exception that is phrased very differently from its rule.

Use it when text has no structure and fixed-size cuts are clearly hurting. If your documents have headings, use the headings.

11. Choosing a Chunk Size

There is no correct chunk size. There is a trade-off:

Put numbers on the last row. With 4 chunks of 400 tokens, the context is about 1,600 tokens per question. With 4 chunks of 1,000 tokens, it is 4,000. At 100,000 questions a day and roughly $0.25 per million input tokens for a small model, the difference is 240 million tokens, about $60 a day or around 1.5 lakh rupees a month, before you count the extra latency of a longer prompt. Check the current price list for your model; the shape of the sum does not change.

The project uses 120 words (about 160 tokens) because each Kartly section is one short rule. A contract or a manual would want bigger chunks. Pick the size by testing on your own documents and real questions.

12. Re-indexing Without Mixing Versions

A practical trap when you experiment with chunking: you change the chunk size, run the ingest again, and now the index holds both the old chunks and the new ones. Search returns a mix, results look strangely repetitive, and you draw the wrong conclusion from the comparison.

Build the new version next to the old one, compare, then switch. Never write the experiment into the live index.

Use a separate namespace in Pinecone, a collection in Qdrant or a table in pgvector. Put the chunking version in every chunk's metadata too, so a stray old chunk is easy to find.

13. Top K: How Many to Fetch

Once chunks are stored, three settings decide which ones are allowed to reach the model. The first is top K: the most results the search may return.

Here are made-up scores for "Can I return my headphones?":

With K = 2 you lose the damage rule. With K = 7 you send shipping and payment rules that have nothing to do with the question.

Bigger is not safer. Every extra chunk costs prompt tokens, adds a little latency, and gives the model one more thing that might distract it. Smaller is not safer either: an exception sitting at rank 4 is gone at K = 3.

The boring approach works. Try K = 3, 5 and 8 on your twenty baseline questions and look at what comes back. There is usually a K where the needed chunks are almost always in and more only adds noise. Reranking (section 21) changes this: fetch a wide K, keep a narrow one.

14. Similarity Threshold: Closest Is Not the Same as Relevant

A customer types "Who won the IPL in 2024?" Kartly's documents say nothing about cricket. Nearest neighbour search does not know that. It always returns the K closest vectors, and something is always closest, even if it is not close at all. Maybe the shipping policy, because it mentions cities.

Without the threshold, those four chunks would go to the model as "context" for a cricket question.

A similarity threshold is a floor. Results below it are dropped, even if that leaves nothing. With a Spring AI vector store it is one more line on the request, .similarityThreshold(0.25), and the project reads it from MIN_SIMILARITY.

Note 11 introduced the threshold. Two things catch teams out:

• A similarity score is not a probability. A cosine of 0.9 does not mean a 90% chance the answer is right. Scores change with the embedding model, the distance metric and the data. With text-embedding-3-small, a good match between a short question and a policy paragraph often scores between 0.4 and 0.6, nowhere near 1. A threshold copied from a tutorial that used a different model can drop every result, or drop none.
• Passing the threshold means "about the same topic", not "contains the answer". The answer instructions still have to tell the model to say "I could not find it" when the extracts do not answer the question.

When nothing passes, handle it in code, before the model. The project returns a fixed answer and makes no chat call at all:

if (context.length === 0) {
  return { answer: NOT_FOUND_ANSWER, sources };
}

That saves a model call on every off-topic question, and the bot never answers a cricket question from memory under your brand name.

15. Metadata Filters: Search the Right Shelf

Every chunk can carry plain fields next to its vector: category, policy type, source file, language, the date it was last updated. These are metadata, the same idea as the filters in note 10. A filter limits search to chunks whose fields match.

If the customer is asking about an order that contained headphones, there is no reason to let the clothing chunk compete.

The easy mistake is a filter that is too strict.

The damage rule applies to every category, so it is tagged general. An "electronics only" filter throws it away.

That is why the project widens every category filter to include general. A category rule often only makes sense next to the general rule it changes.

Where does the category come from? Not from a second LLM call, if you can avoid it. If the customer opened the chat from an order page, the app already knows the product category from the order. Use that structured value directly. It is free, instant and never wrong. Asking a model to guess the category from the question costs another round trip of maybe 300 to 800 ms and is wrong some of the time.

Where the metadata comes from matters too. Reading it from file names (if name contains "electronics") breaks the first time someone uploads Elec_Returns_FINAL_v2.pdf. The project keeps a small manifest.json instead, a stand-in for metadata from the content system itself.

16. The Three Dials Together

Top K, threshold and filter answer three different questions:

With a Spring AI vector store (Pinecone, pgvector, Qdrant and others all sit behind the same interface), all three go on one request:

List<Document> chunks = vectorStore.similaritySearch(
        SearchRequest.builder()
                .query(question)
                .topK(5)
                .similarityThreshold(0.25)
                // Spring AI turns this into the store's own filter syntax.
                .filterExpression("category in ['electronics', 'general']")
                .build());

The filter string is portable. Spring AI translates it into Pinecone's, Qdrant's or pgvector's filter format.

Filter first, then score, then cut. The order matters. Say K is 5 and you filter after the cut. If 3 of the 5 results fail the filter, you get 2 results, not 5. Note 10 calls this the post filter problem.

17. When Meaning Is Not Enough: Exact Codes

Now a different kind of question: "Does the AX-204 wireless headphone support fast charging?"

Kartly sells two headphones:

Embeddings capture meaning. The meaning of this question is "wireless headphones" and "fast charging", and the BX-900 chunk is full of exactly that. So the BX-900 chunk can score higher than the AX-204 one. Then the model reads "supports fast charging" and tells the customer yes, about the wrong product.

To an embedding model, "AX-204" and "BX-900" are both just "some product code". The difference is small next to the shared words around them. The same goes for order numbers, error codes like E-4012, SKUs and names.

Plain keyword search, an old idea, handles this well.

Each one fails where the other is strong. That is the whole case for using both.

18. BM25 in Plain Words

Keyword search is also called lexical search: it looks at the words themselves. The standard way to rank keyword results is a formula called BM25. You do not need to memorise it. Know its three ideas, because each one shows up when you tune search.

One score per chunk. Chunks that share no word with the query score zero and are left out.

The three ideas with Kartly's numbers, using the project's 17 chunks:

1. Rarity. "ax-204" appears in 2 of the 17 chunks. "charging" also appears in 2. "headphones" appears in 5. The rarer a word is, the more it is worth. This weight is called IDF (inverse document frequency). "ax-204" gets about 1.97, "headphones" about 1.19. Words like "the" and "can" appear everywhere and count for almost nothing, so the project drops them before scoring.
2. Frequency with a cap. A chunk that says "AX-204" three times scores higher than one that says it once, but not three times higher. The setting k1 = 1.2 makes each repeat count less. Without the cap, a page that repeats a keyword fifty times would win every search.
3. Length. A long chunk has more words, so it matches more by chance. The setting b = 0.75 marks long chunks down. A short, focused chunk then is not beaten by a long one that mentions everything.

For the AX-204 question, the AX-204 charging chunk wins on BM25 because it contains "ax-204" three times and the BX-900 chunk contains it zero times. The BX-900 charging chunk still comes second, because it shares "wireless", "fast" and "charging" with the question. The semantic search may put BX-900 first. Now we have two lists that disagree.

One detail matters: the project's tokenizer (the code that splits text into words) keeps "AX-204" as one word, ax-204. A tokenizer that splits on every non-letter turns the code into two common fragments, "ax" and "204", and loses the exact match.

19. Hybrid Search

Hybrid search runs both searches on the same question and merges what they find.

Semantic search brings the topic. Keyword search brings the exact code. The merge needs a rule, which is the next section.

In production you rarely write BM25 yourself. Elasticsearch and OpenSearch have it next to vector search, Weaviate, Qdrant and Pinecone offer hybrid modes, and Postgres pairs pgvector with full text search. The project writes BM25 by hand, so you can see it is just counting.

20. Reciprocal Rank Fusion

The two lists use different scales. Cosine similarity lives between -1 and 1, and for these chunks sits around 0.3 to 0.6. BM25 scores have no upper limit; here they run from about 1 to 13 (a rough figure that depends on the query and the files). Adding them makes no sense: a BM25 score of 7 would drown any cosine score.

Reciprocal rank fusion (RRF) ignores the scores and uses only the positions. Each list gives each document these points:

points = 1 / (k + rank)        rank starts at 1, k is usually 60

A document gets points from every list it appears in. A list it is missing from gives it nothing. The total decides the new order.

Here is the AX-204 question with k = 60. The keyword ranks are what BM25 gives on the project's files. The semantic ranks are a plausible guess:

Two things happened. Chunks found by both searches pulled well clear of chunks found by only one. So a document that is good in both lists beats one that is best in just one. And the AX-204 chunk, which semantic search alone put second, has caught up with the BX-900 one.

It has only caught up, though. The two charging chunks tie exactly, because each is first in one list and second in the other. The project breaks ties by id, which happens to put ax-204-manual#1 first. That is luck, not design.

The k in RRF is not the top K from section 13. It is a smoothing number that keeps the gap between ranks small. With k = 60, rank 1 is worth 1/61 and rank 2 is worth 1/62, almost the same. That is on purpose: one list cannot win the merge just by putting a document first. With k = 1, rank 1 would be worth 1/2 and rank 2 only 1/3, and whichever search was more confident would decide everything.

Hybrid search moves the right chunk up, but it does not always move it far enough to decide the question. That is the gap the next stage fills.

21. Reranking: Found, but Ranked Too Low

Back to the damaged headphones. Say the merged search returns:

The damage rule was found. It is at rank 4. With K = 3 it is thrown away.

On the project's own ten files, BM25 already lifts the damage rule high, because "damaged" is a rare word there. In a knowledge base with fifty documents that mention damage, it stops being rare, and an order like the one above is normal.

This is a different problem from section 2. There, the rule was never found. Here it was found, but ranked too low. You could raise K to 5, but then refunds and exchanges come along too, and you pay for them on every question.

A reranker fixes the order. It takes a short list of candidates, reads each one together with the question, and gives each a new relevance score. Then you keep the top few by the new score.

Made-up scores on the code's scale, whole numbers from 0 to 10; the real shape. The damage rule moves from fourth to first.

A reranker cannot invent a chunk that was not retrieved. It only reorders what it was given. If the damage rule had not been in the candidates, no reranker would bring it back. Fix recall first, then order.

22. Bi-Encoders and Cross-Encoders

Why can a reranker see what vector search missed? Because it reads differently.

Vector search uses a bi-encoder. The question and each chunk are embedded separately, by the same model, and compared only afterwards as two vectors. That makes it fast. Chunk vectors are computed once at ingest. A query needs just one new embedding and a nearest neighbour lookup.

A cross-encoder reads the question and one chunk together, as one input, and outputs a single relevance score. It can see that "damaged" in the question and "report within 48 hours" in the chunk are directly about each other. A bi-encoder only sees that two vectors point in roughly the same direction.

The bi-encoder's chunk side is paid once. The cross-encoder pays for every question and chunk pair, every time.

That table is the whole reason for two stages. You cannot run a cross-encoder over a million chunks per question. You can run it over 20.

23. Two-Stage Retrieval

Put the two together: a fast, wide first stage, then a slow, careful second stage.

Wide recall from cheap search, then precision from the expensive model, on a list small enough to afford it.

A rough latency budget for one question with a hosted reranker:

The answer dominates, so a cross-encoder rerank is one of the cheapest quality wins in RAG.

Common sizes are 10 to 30 candidates in, 3 to 5 out. The project fetches up to 10 from each search, fuses them, sends the top 8 to the reranker and keeps topK (default 4).

24. Choosing a Reranker (Provider-Specific Examples)

Provider-specific and time-sensitive: Product names, hosted options, latency ranges, token prices, and model behavior in this section are illustrative. Verify current documentation and benchmark with your own candidate sizes before making a production choice.

There are three common ways to get a reranker:

The project uses the third option on purpose. It lets the whole project run with one OpenAI key. The reranker's interface (question + candidates in, scores out) is also the same as a hosted reranker's, so swapping in a cross-encoder later changes one class.

For production, a hosted cross-encoder is the better default. An LLM reranker over 8 candidates of 160 tokens each is about 1,500 input tokens per question. At 100,000 questions a day that is 150 million extra tokens, about $37 a day at $0.25 per million, before the output tokens. A reasoning model like gpt-5-mini also spends hidden thinking tokens before it replies, and you pay for them as output. The call adds a second or more to every answer.

When not to rerank: if your knowledge base is 30 short pages, top 4 from hybrid search is probably already right, and a reranker adds cost and latency for nothing. Add it when your baseline shows the right chunk often sitting at rank 5 to 15.

In the project, if the reranker fails or returns nonsense, the fused order is used and the response says "reranked": false. A worse order beats a 502 for a step that only improves things.

25. Context Engineering: The Model Still Has to Read It

Say retrieval worked. For "Can I return my headphones after 10 days?" the model receives:

And the model replies: "Yes, you can return it, since the return window is 30 days."

Both rules were there. The model picked the general one. That is a generation failure, and no amount of retrieval tuning will fix it. Context engineering means choosing and laying out what goes into the prompt, so the model reads it correctly. It matters most when the rules in the context compete, repeat or do not apply.

Each step removes one way for the model to misread the extracts.

What the project does, and why:

• Labels. Each chunk goes in as [S1] Return Policy > Electronics (applies to: electronics). The model can tell which rule covers which product, and it can cite [S1], so a person can check the answer.
• Precedence in plain words. "If a category rule and a general rule disagree, the category rule wins." Policies are written this way, but the model does not know that unless told.
• Check every condition. "Check every detail in the question, such as product type, damage and days since delivery, against every extract." This one line catches most of the "followed the first rule it saw" answers.
• A way out. "If the extracts do not contain the answer, say so." The threshold handles the empty case; this handles the "related but not enough" case.
• Data is not instructions. The extracts go in the user message as quoted material, and the rules go in the system instructions. Say a policy document contains the line "ignore previous instructions". The model should read it as text, not obey it. Keeping the two apart is not a complete defence, but mixing them in one block makes things much worse.

On bigger contexts, two more things help. Models use the start and end of a long context better than the middle, so put the strongest chunks first. And drop near duplicates: overlapping chunks waste tokens and make one rule look twice as important.

This is only the start. Context engineering grows into its own subject once history, tool results and agents join the prompt.

26. When Not to Bother

Every upgrade has a cost. Here is when each one is not worth it:

And one more: if the knowledge base is small enough to fit in the prompt, say 20 pages of policies (about 10,000 tokens), and traffic is low, you may not need retrieval at all. Send everything, use prompt caching if your provider offers it, and spend your time on evaluation instead.

27. Measuring It

Every number in this note (chunk size, overlap, K, threshold, candidate pool) is a guess until you measure it. The measurement does not need to be fancy.

Build a small test set: 50 to 200 real questions from support logs, each with the chunk ids a correct answer needs. A support lead can label 100 in an afternoon. Then, for every change, run the questions through /api/search and count.

Recall at K matters most for multi-rule questions like the damaged headphones, where one hit out of three is still a wrong answer.

Change one dial at a time and keep the test set fixed, or you will not know which change helped.

Measure answers separately. Sample them and check against the retrieved chunks: right rule, cited, nothing added. That is the generation side from section 3, and it needs its own checks.

28. Building It: the Kartly Policy Desk

The knowledge base is ten short markdown files in knowledge/ (returns, damage, refunds, exchanges, shipping, warranty and two headphone manuals) plus a manifest.json with each file's category. All three versions read the same folder.

Both return 400 for an empty or over-long question, an unknown category or a topK outside 1 to 8, and 502 when the OpenAI API fails.

One /api/ask request moves through the parts like this:

The HTTP layer knows nothing about OpenAI, the retriever nothing about HTTP, and the answerer only sees the final context.

Decisions worth copying:

29. The Code, Side by Side

The full projects are on the code page:

Browse the code for this note, or open src/FDE/code/12-advanced-rag-retrieval/ on GitHub. The README there has the run commands, the environment variables and sample requests and responses.

29.1 The chunker: cut at headings, then by size

The size step is the same loop in all three languages. It only runs on a section that is already too long. It adds sentences to current one by one. When the next sentence would push the piece past maxWords, it saves the piece and starts a new one that begins with the last sentence of the old one (the overlap):

for (String sentence : SENTENCE_END.split(body)) {
    if (!current.isEmpty() && wordCount(join(current, sentence)) > maxWords) {
        pieces.add(String.join(" ", current));
        // Carry the last sentence into the next piece: that is the overlap.
        int keepFrom = Math.max(0, current.size() - overlap);
        current = new ArrayList<>(current.subList(keepFrom, current.size()));
        // Carry plus new sentence already too long: drop the carry, or
        // one long sentence would keep producing oversized pieces.
        if (wordCount(join(current, sentence)) > maxWords) {
            current.clear();
        }
    }
    current.add(sentence);
}

for (const sentence of body.split(SENTENCE_END)) {
  if (current.length > 0 && wordCount([...current, sentence].join(" ")) > maxWords) {
    pieces.push(current.join(" "));
    // Carry the last sentence into the next piece: that is the overlap.
    // slice(-0) would copy the whole list, so zero overlap needs its own branch.
    current = overlap > 0 ? current.slice(-overlap) : [];
    // Carry plus new sentence already too long: drop the carry, or one long
    // sentence would keep producing oversized pieces.
    if (wordCount([...current, sentence].join(" ")) > maxWords) current = [];
  }
  current.push(sentence);
}

for sentence in SENTENCE_END.split(body):
    if current and _word_count(" ".join([*current, sentence])) > max_words:
        pieces.append(" ".join(current))
        # Carry the last sentence into the next piece: that is the overlap.
        # current[-0:] would copy the whole list, so zero overlap needs its own branch.
        current = current[-overlap:] if overlap > 0 else []
        # Carry plus new sentence already too long: drop the carry, or one long
        # sentence would keep producing oversized pieces.
        if _word_count(" ".join([*current, sentence])) > max_words:
            current = []
    current.append(sentence)

On Kartly's files this loop runs once: the warranty section is 177 words, so it becomes two pieces of 99 and 94 words, which share one sentence. That is how ten files become 17 chunks.

29.2 BM25: three ideas in one line

The scoring line holds all of section 18. inverseDocumentFrequency(term) is rarity. The fraction is capped frequency. lengthFactor is length: above 1 for a chunk longer than average, below 1 for a shorter one.

for (const term of terms) {
  const frequency = counts.get(term);
  if (!frequency) continue;
  const lengthFactor = 1 - B + B * (length / averageLength);
  score += inverseDocumentFrequency(term) * ((frequency * (K1 + 1)) / (frequency + K1 * lengthFactor));
}

The Java and Python versions are the same arithmetic. The tokenizer regex, [a-z0-9]+(?:-[a-z0-9]+)*, is what keeps ax-204 in one piece.

29.3 RRF: positions, not scores

for (List<Hit> list : lists) {
    for (int i = 0; i < list.size(); i++) {
        Hit hit = list.get(i);
        titles.putIfAbsent(hit.id(), hit.title());
        // Rank starts at 1, so position i is worth 1 / (RRF_K + i + 1).
        scores.merge(hit.id(), 1.0 / (RRF_K + i + 1), Double::sum);
    }
}

for (const list of lists) {
  list.forEach((hit, index) => {
    const entry = fused.get(hit.id) ?? { id: hit.id, title: hit.title, score: 0 };
    entry.score += 1 / (k + index + 1);
    fused.set(hit.id, entry);
  });
}

for hits in lists:
    for rank, hit in enumerate(hits, start=1):
        entry = fused.setdefault(hit["id"], {"id": hit["id"], "title": hit["title"], "score": 0.0})
        entry["score"] += 1 / (k + rank)

Each hit adds 1 / (k + rank) to its total. Nothing here reads hit.score. That is the point of RRF.

29.4 The reranker and its fallback

Spring asks for typed output and lets Spring AI parse it. Node and Python send a strict JSON schema, the subject of note 13. All three return "no result" on any failure, and the pipeline keeps the fused order.

try {
    RerankScores reply = chatClient.prompt()
            .system(INSTRUCTIONS)
            .user(formatCandidates(question, chunks))
            .call()
            .entity(RerankScores.class);
    return applyScores(chunks, reply == null ? null : reply.scores());
} catch (RuntimeException e) {
    log.warn("Reranking failed, keeping the fused order: {}", e.getMessage());
    return Optional.empty();
}

const response = await client.responses.create({
  model,
  instructions: RERANK_INSTRUCTIONS,
  input: formatCandidates(question, chunks),
  text: { format: RERANK_FORMAT },
});
return applyScores(chunks, JSON.parse(response.output_text).scores);

response = await self._client.responses.create(
    model=self._model,
    instructions=RERANK_INSTRUCTIONS,
    input=_format_candidates(question, chunks),
    text={"format": RERANK_FORMAT},
)
return apply_scores(chunks, json.loads(response.output_text).get("scores"))

The model's reply is not trusted. applyScores ignores ids the model made up, clamps scores to 0 to 10 and gives 0 to candidates it skipped. A stable sort keeps the fused order on ties.

29.5 The context block

def build_context_block(context: list[ContextChunk]) -> str:
    """The extracts as the model reads them: label, heading path, category, then the text."""
    return "\n\n".join(
        f"[{_label(i)}] {chunk['title']} (applies to: {chunk['category']})\n{chunk['text']}"
        for i, chunk in enumerate(context)
    )

The label is what the model cites. The category is what lets "the category rule wins" work.

29.6 Where to look in the code

29.7 What this demo leaves out on purpose

30. The Full Mental Model

Put it all in one picture.

The ingest half decides what can ever be found. The query half decides what is found this time and how the model reads it.

In one line:

right chunks, in the right order, laid out clearly = a RAG answer you can trust

Basic RAG gets the right chunks some of the time. Each upgrade in this note closes one specific gap, and the test set tells you which gap you actually have.

Interview Questions

"What is the difference between a retrieval failure and a generation failure in RAG, and why does it matter?"

A retrieval failure means the text needed for the answer never reached the model. A generation failure means it did reach the model and the model still answered wrongly. They need different fixes: chunking, search and filters for the first, context layout and instructions for the second. You tell them apart by inspecting the retrieved chunks before reading the answer.

"Why can fixed-size chunking break a policy chatbot, and what would you use instead?"

A fixed cut can separate a rule from its exception, so a question that needs both retrieves only one, and the second chunk may not even name the product it applies to. For documents with headings, cut at the headings and keep the heading path in each chunk, then split only oversized sections by size with a little overlap.

"Why add a similarity threshold if you already have top K?"

Top K always returns K results, even for a question the documents know nothing about, because something is always closest. A threshold drops matches that are too weak, so an off-topic question gets an honest "not found" without a model call. The threshold depends on the embedding model and must be tuned on real questions.

"Why does vector search miss exact product codes, and how does hybrid search fix it?"

Embeddings capture meaning, so two product codes surrounded by the same words land close together and a similar but wrong product can rank first. Keyword search with BM25 matches the exact token, such as "ax-204". Hybrid search runs both and merges the lists, so meaning and exact terms both count.

"How does reciprocal rank fusion work, and why not just add the scores?"

Each list gives a document 1 / (k + rank) points, usually with k = 60, and the totals decide the new order. Cosine and BM25 scores are on different scales, so adding them would let one search dominate. RRF uses only positions, so a document near the top of both lists wins.

"What is a cross-encoder reranker and why is it used as a second stage instead of for all retrieval?"

A cross-encoder reads the question and one document together and outputs a relevance score, which is far more accurate than comparing two separately made vectors. It has to run once per question and document pair, so it is too slow for a million documents. You retrieve 10 to 30 candidates cheaply, then rerank only those and keep the best 3 to 5.

"A metadata filter made answers worse. What is a likely cause?"

The filter was too strict. A rule that applies to every product, like a damage reporting deadline, is often tagged general, and an "electronics only" filter removes it. Widen the filter to include general chunks, and take the category from structured data like the order rather than a guess.

"The right chunks were retrieved but the model followed the wrong rule. What would you change?"

That is a generation failure, so change the context, not the search. Label each extract with its category, state that a category rule beats the general rule, tell the model to check every condition in the question, and give it a clear way to say the answer is not there.

Key Takeaways

• A model cannot follow a rule that never reached its prompt. Most "the AI got it wrong" bugs in RAG are retrieval failures.
• Debug in order: look at the retrieved chunks first, then at the answer. Log the chunk ids behind every answer.
• Advanced RAG is the same pipeline with better boxes. Add the upgrade that fixes the miss you can see, not all of them.
• Chunk by structure first and size second. Keep the heading path in every chunk so "within 7 days" knows what it belongs to.
• Overlap protects boundaries and costs storage: 100 tokens of overlap on 500 token chunks means 25% more vectors.
• Top K, threshold and metadata filter answer three different questions: how many, how close, and which ones may compete.
• A similarity score is not a probability. Tune the threshold on your own questions and your own embedding model.
• A filter that is too strict removes general rules. Include general, and take the category from the order, not from another model call.
• Embeddings blur exact codes. BM25 finds them. Hybrid search uses both.
• RRF merges ranked lists by position, 1 / (60 + rank), so scores on different scales never have to be compared.
• A reranker reads the question and each candidate together. Use it on 10 to 30 candidates, keep 3 to 5. It reorders; it cannot find what retrieval missed.
• Context engineering is the generation side: labels, order, "specific beats general", "check every condition", and a clear way out.
• Every number here is a starting point. A labelled test set of 100 real questions and recall at K tell you which change actually helped.