Advanced RAG: Chunking, Hybrid Search and Reranking

This topic builds directly on the RAG lesson you just learned. The big idea is:

Basic RAG finds relevant information. Advanced RAG improves the chances of finding all the right information, ranking it correctly, and helping the AI use it properly.

1. Why do we need Advanced RAG?

Imagine an online shop called Kartly.

It has three return rules:

A customer asks:

Can I return my damaged headphones after 10 days?

To answer correctly, the AI needs to find the electronics rule and the damaged-product rule. The general 30-day rule alone is not enough.

What goes wrong with basic RAG?

The AI may be answering reasonably based on what it received. The real problem is that the correct rules never reached the AI.

2. The two types of RAG failure

This is the most important debugging concept in the entire topic.

A. Retrieval failure

The system failed to find the required information.

Example: It retrieves the 30-day rule but misses the 7-day electronics rule.

Fix: Improve chunking, search, filters or the number of retrieved results.

B. Generation failure

The system found the correct information, but the AI used it incorrectly.

Example: It receives both the 30-day and 7-day rules, but follows the general rule.

Fix: Improve the prompt, the ordering and labeling of the retrieved text, or the model.

Remember this forever: Before changing the prompt, inspect the retrieved chunks. A better prompt cannot use information it never received.

3. Advanced RAG is not a completely new architecture

It is the same RAG pipeline, with improvements added where needed.

You don't need every upgrade from day one. First find the problem, then add the technique that fixes it.

4. Chunking: cut documents intelligently

A chunk is a small piece of text with its own embedding.

Fixed-size chunking

Imagine a policy:

Electronics can be returned within 7 days. Products must be in their original packaging. Customer-caused physical damage is not eligible for return.

If you split this text at an arbitrary character or token limit, the main rule might land in one chunk and its exception in another.

Original policy
      |
      +---- Chunk 1: Return within 7 days
      |
      +---- Chunk 2: Customer-caused damage is not eligible

A question about a damaged phone might retrieve only Chunk 1. The AI never sees the exception.

Structural chunking: usually better for policies

Cut at the headings that the document author already created.

Return Policy
  |
  +-- General Returns
  |     +-- 30-day rule and conditions
  |
  +-- Electronics
  |     +-- 7-day rule, packaging, exceptions
  |
  +-- Damaged Products
        +-- 48-hour reporting rule

Keep the heading in the chunk text:

Return Policy > Electronics

This tells the AI and the search system what the seven-day period actually refers to. If a section is still too long, split it by size at sentence boundaries and use some overlap.

What is overlap?

Overlap repeats some text between adjacent chunks so a boundary doesn't lose important context.

For example, a 500-token chunk with 100 tokens of overlap means the next chunk starts after 400 new tokens.

Chunk A: [1 ---------------- 500]
Chunk B:                     [401 ---------------- 900]
                               <--- 100 shared tokens --->

The trade-off: overlap protects meaning, but stores repeated text and can waste prompt space. A reasonable starting point from the notes is 400 to 600 tokens per chunk with 10 to 20% overlap, followed by testing.

Memory trick: Structure first, size second, overlap to protect boundaries.

5. Top K, similarity threshold and metadata filters

These three settings solve three different problems. Don't confuse them.

Top K = How many?

If topK = 4, the search returns at most four candidates.

Too few can miss a required rule. Too many can add irrelevant text and increase cost.

Threshold = How close?

Reject results whose similarity scores are below a chosen minimum.

This helps when a customer asks something unrelated to the knowledge base.

Metadata filter = Which ones are allowed?

Restrict candidates using fields such as category, language or source.

For example, prefer electronics policies for a headphone question.

Try it yourself

What these settings mean:

• The system can return at most 4 chunks.
• Only results with scores at least 0.30 are eligible.
• The search considers electronics chunks plus general rules such as damage-reporting deadlines.

This is a conceptual simulator, not a live search. Actual results depend on the documents, filters and embedding model.

One subtle but important point: a similarity score is not a probability. A score of 0.6 does not mean there's a 60% chance the answer is correct. Tune thresholds against your own questions and data.

Also, filters can cause trouble. If the damage-reporting rule is categorized as general, an electronics-only filter may remove it. That's why the notes recommend including general rules when they apply across categories.

6. Semantic search versus keyword search

This is the next major concept.

Imagine the customer asks:

Does the AX-204 wireless headphone support fast charging?

Kartly sells two products:

• AX-204: Does not support fast charging.
• BX-900: Supports fast charging.

Both descriptions talk about wireless headphones and charging. Vector search might rank BX-900 higher because its description is more semantically similar to the question.

But the customer asked about one exact model: AX-204.

Neither is enough for every question. Hybrid search combines both.

What is BM25?

BM25 is a widely used formula for ranking keyword search results. You don't need to memorize the formula; remember its three ideas:

1. Frequency: A query word appearing in a chunk helps, but repeating it endlessly shouldn't dominate.
2. Rarity: Rare words and exact product codes are more useful than common words like "the".
3. Length: A huge document shouldn't automatically win just because it contains more words.

For the AX-204 question, a chunk containing the exact code can outrank a chunk about BX-900, even if BX-900 sounds more similar overall.

7. RRF: combining two rankings

Hybrid search gives us two ranked lists:

Problem: the scores use different scales

Cosine similarity might be around 0.5, while BM25 might be 8. Adding those scores directly would be misleading.

The solution is Reciprocal Rank Fusion (RRF). It uses ranking positions instead of the original scores.

RRF score = sum over all lists of  1 / (k + rank)

In the notes, k is usually 60 and rank starts at 1. A document gets points from every list where it appears; its points are added together.

AX-204 charging:
  semantic rank 2 + keyword rank 1
  = 1/62 + 1/61

BX-900 charging:
  semantic rank 1 + keyword rank 2
  = 1/61 + 1/62

These two examples tie. RRF doesn't magically know that AX-204 is the correct product; it combines the evidence from both lists. A reranker can make a more detailed judgment next.

Memory trick: Semantic search understands the topic; BM25 catches the exact words; RRF combines their rankings.

8. Reranking: found, but ranked too low

Suppose hybrid search finds the damage rule, but ranks it fourth.

If you keep only the top three, the damage rule disappears, even though search found it.

You could increase Top K, but then more irrelevant chunks go into the prompt. A reranker gives you another option: take a small set of candidates and score them more carefully against the question.

For example, the damage rule might move from rank 4 to rank 1 after reranking.

The important limitation: a reranker can reorder candidates, but it cannot recover a document that the first-stage search never found. Fix retrieval coverage first.

Bi-encoder versus cross-encoder

These names sound difficult, but the difference is simple.

Bi-encoder = fast finder

• Embeds the question separately.
• Embeds each document separately, usually during indexing.
• Compares the resulting vectors.
• Fast enough to search a huge collection.

Cross-encoder = careful judge

• Reads the question and a candidate document together.
• Produces a relevance score for that pair.
• Usually more accurate at judging relevance, but slower.
• Best used on a short list of candidates.

Remember: The bi-encoder finds; the cross-encoder judges.

9. Context engineering: help the AI apply the right rule

Even after retrieval and reranking work, the AI can still make mistakes.

Suppose the final context contains:

• [S1] General returns: Products can be returned within 30 days.
• [S2] Electronics: Electronics can be returned within 7 days.
• [S3] Refund timeline: Approved refunds take 5 to 7 business days.

The model might still follow the first rule it reads and answer incorrectly.

Context engineering means arranging and explaining the retrieved information so the model can use it correctly.

The notes recommend:

• Label each source: Use IDs such as [S1], with its heading and category.
• Explain precedence: A specific category rule overrides a conflicting general rule.
• Check every condition: Consider product type, damage status and days since delivery.
• Allow "I don't know": If the retrieved text doesn't answer the question, say so.
• Keep data separate from instructions: Retrieved text is evidence, not a command to the AI.
• Remove duplicates: Repeated chunks waste tokens and can make a rule seem more important than it is.

These changes address a generation failure: the relevant text was already present, but the model didn't use it properly.

10. How do you know whether an improvement actually works?

Don't change five settings at once and judge the result by a few examples.

Build a test set of real questions with known correct answers and the chunk IDs needed to answer them. The notes suggest starting with 50 to 200 questions.

Three useful metrics:

Example: if a question needs three chunks and search finds two, recall for that question is 2/3, or about 66.7%.

For a question needing only one chunk, hit rate may be enough. For a question needing three separate rules, recall is more revealing: finding just one of three may still produce a wrong answer.

The debugging checklist

When a user reports a wrong answer:

1. Inspect which chunks were retrieved.
2. Check whether the necessary rules were present.
3. If not, fix chunking, filters, Top K, or retrieval.
4. If yes, inspect the order, labels and prompt instructions.
5. Run the same test set again and compare the metrics.

Log chunk IDs and scores so you can trace which evidence the AI saw. Avoid logging full sensitive document contents unnecessarily.

11. The full Advanced RAG architecture

Now connect everything into one picture.

Think of it like a library:

• Chunking puts each rule on the right page.
• Metadata points you to the right shelf.
• Semantic search finds books about the topic.
• BM25 finds exact names and codes.
• RRF combines the two lists.
• Reranking chooses the most relevant pages.
• Context engineering helps the reader apply the rules correctly.

12. Revision flashcards

Test your memory

1. Why do we need Advanced RAG? Because basic RAG can miss the rules the answer needs. The AI may answer reasonably from what it received, but the correct rules never reached it.
2. Retrieval failure vs generation failure? Retrieval failure: the system failed to find the required information. Generation failure: it found the correct information, but the AI used it incorrectly.
3. What is structural chunking? Cutting at the headings the document author already created, and keeping the heading in the chunk text, such as Return Policy > Electronics.
4. Top K vs threshold vs metadata filter? Top K is how many, the threshold is how close, and the metadata filter is which ones are allowed.
5. Why use hybrid search? Semantic search finds similar meaning, keyword search finds exact values such as AX-204. Neither is enough for every question, so hybrid search combines both.
6. What is RRF? Reciprocal Rank Fusion combines two ranked lists using ranking positions instead of the original scores: the sum over all lists of 1 / (k + rank).
7. Bi-encoder vs cross-encoder? The bi-encoder finds: it embeds the question and documents separately and is fast enough for a huge collection. The cross-encoder judges: it reads the question and a candidate together, slower but more accurate, so use it on a short list.
8. What can a reranker not do? It can reorder candidates, but it cannot recover a document that the first-stage search never found.
9. What is context engineering? Arranging and explaining the retrieved information so the model can use it correctly: label each source, explain precedence, check every condition, allow "I don't know", keep data separate from instructions and remove duplicates.
10. How do you debug a wrong RAG answer? Inspect which chunks were retrieved. If the necessary rules were missing, fix chunking, filters, Top K or retrieval. If they were present, inspect the order, labels and prompt instructions. Then run the same test set again and compare the metrics.

The one-minute memory summary

Memorize this sequence:

CHUNK → FILTER → SEARCH → FUSE → RERANK → CONTEXT → GENERATE

• Chunk: Keep each rule and its exceptions together.
• Filter: Search the right category without excluding general rules.
• Search: Use vectors for meaning and BM25 for exact words.
• Fuse: Use RRF to combine rankings.
• Rerank: Carefully judge a short candidate list.
• Context: Label and order the evidence, and explain which rules take priority.
• Generate: Produce an answer supported by the retrieved evidence.

The most important engineering rule is: find the first stage that fails, fix that stage, and measure the improvement. Don't add every advanced technique just because it exists.