Contents

• Building RAG Systems: A Beginner’s Guide

◦ Part 1 — The Big Picture

▪ What is a Large Language Model (LLM)?

▪ What problem are we solving?
▪ What is RAG (Retrieval-Augmented Generation)?
▪ RAG builds on things you already know
▪ The “small idea” and the “big idea” behind RAG
◦ Part 2 — Your First RAG (The Simplest Possible Version)

▪ Step 1: Load your knowledge base

▪ Step 2: Find relevant context by keyword matching
▪ Step 3: Add the context to the prompt
▪ Step 4: Ask the model
▪ Step 5: Give it a simple interface with Gradio
◦ Part 3 — Chunking: Breaking Documents Into Pieces

▪ Why not just feed whole documents to the model?

▪ What is a token?
▪ What is chunking?
▪ Chunk size and overlap
▪ How big can a knowledge base get?
▪ Attaching metadata
◦ Part 4 — Embeddings and Vectors: Searching by Meaning

▪ The problem with keyword search

▪ Two families of LLMs: auto-regressive and auto-encoding
▪ What is an embedding?
▪ A little history (so the idea feels less magical)
▪ How “closeness” is actually measured: cosine similarity
▪ A nice bonus: fuzzy matching and typo tolerance
▪ What is a vector store?
▪ Picking an embedding model

1 / 35
 ▪ Inspecting the vectors
▪ Seeing the invisible: visualizing embeddings
◦ Part 5 — A Clean RAG Pipeline with LangChain

▪ What LangChain is (and its trade-offs)

▪ LangChain comes in many packages
▪ A lighter alternative: litellm
▪ The Document object
▪ Connect to the saved vector store
▪ The two key objects: retriever and LLM
▪ What is temperature?
▪ The invoke() method
▪ Putting it together
▪ From notebook to production: organizing the code
◦ Part 6 — Evaluation: Is Your RAG Any Good?

▪ A test set
▪ Categories of questions — and where RAG struggles
▪ Evaluating retrieval
▪ Evaluating the answer with an “LLM-as-judge”
▪ How the judge returns clean scores: structured outputs
◦ Part 7 — Advanced RAG: Techniques for Better Answers

▪ First, an honest admission: RAG is a bit of a hack

▪ Technique 1: Smart chunking with an LLM (semantic
chunking)
▪ Technique 2: Document pre-processing (enriching chunks)
▪ Technique 3: Query rewriting
▪ Technique 4: Reranking
▪ The full advanced pipeline
▪ The wider zoo of RAG techniques
▪ “RAG is dead” — or is it?

2 / 35
 ◦ Putting It All Together: The RAG Journey

▪ Where to take this next: your own knowledge worker

Building RAG Systems: A

Beginner’s Guide
A learner-friendly walkthrough of Retrieval-Augmented Generation (RAG),
from the simplest possible version to advanced, production-style techniques.

These notes assume you have never built anything with a large
language model before. Every technical term is explained the first time
it appears. Each section is written to stand on its own, so you can read
one part without having read the parts before it (though reading in
order builds the ideas most naturally).

Part 1 — The Big Picture

What is a Large Language Model (LLM)?

A Large Language Model (LLM) is an AI system trained on huge
amounts of text so it can predict and generate human-like language.
When you type a question, it produces an answer word by word.
Examples include OpenAI’s GPT models. In this guide we mostly use a
small, inexpensive model called gpt-4.1-nano .

LLMs are powerful, but they have two limitations that matter a lot:

• They only know what they were trained on. An LLM has no
knowledge of your company’s private documents, your latest data, or
anything that happened after its training ended.

3 / 35
 • They can “hallucinate.” A hallucination is when the model
confidently makes up an answer that sounds correct but isn’t. This is
dangerous when accuracy matters (like insurance, law, or medicine).

What problem are we solving?

Imagine an insurance technology company, Insurellm, that wants a
chatbot to answer employee questions about its people, products, and
contracts. The chatbot must be:

• Accurate — no made-up answers.

• Low cost — cheap to run.

We can’t retrain a giant model every time a document changes (that’s

slow and expensive). So instead we give the model the right information
at the moment it answers. That technique is called RAG.

What is RAG (Retrieval-Augmented Generation)?

RAG stands for Retrieval-Augmented Generation. Break it down:

• Retrieval — find the pieces of information that are relevant to the

user’s question.
• Augmented — add (“augment”) those pieces into the prompt you
send to the LLM.
• Generation — let the LLM generate an answer using that added
information.

In plain words: look up the relevant facts first, hand them to the model,
then ask the model to answer using those facts.

This gives you accuracy (the answer is grounded in real documents) at

low cost (you don’t retrain anything — you just look things up). It is one
of the most immediately useful techniques in all of LLM engineering,
and commercial products are built on exactly this idea.

4 / 35
The core RAG loop, in one sentence: User asks a question → system
retrieves relevant text → that text is inserted into the prompt → the LLM
answers based on it.

RAG builds on things you already know

RAG isn’t a brand-new invention — it’s the natural next step of
techniques you may already use to get better answers out of a model.
These are called inference-time techniques: tricks applied when you ask
the model (at “inference”), without retraining it. Two common ones:

• Multi-shot prompting — putting several example question-and-

answer pairs into the prompt so the model sees the pattern you
want.
• Tools — giving the model the ability to call a function (like looking
up a ticket price) to fetch information it doesn’t have.

The thread connecting all of these is simple: the more relevant

information you put in the prompt, the better the model answers. RAG just
takes that idea further — it systematically finds relevant information and
adds it to the prompt every time.

The “small idea” and the “big idea” behind RAG

It helps to build RAG up in two stages.

The small idea: Suppose you keep all your useful information in a
database — a knowledge base (just a fancy word for a database of
background information). When a user asks a question, you first scan it
for something you can look up. For example, if the question contains
the word “London,” you run a database query for anything about
London, then paste whatever you find into the prompt: “Here is some
background that might be relevant… now answer the question.” The model
then naturally weaves those facts into its reply. That’s it. You probably
could have thought of this yourself.

5 / 35
 User question ──▶ [Your code] ──▶ query the knowledge base for
anything relevant
│ │
│◀───────────────────────┘ (relevant
facts)
▼
build a prompt = question + relevant facts
│
▼
[LLM] ──▶ answer sent back to the user

The big idea improves on the weak part of the small idea: matching by
exact keywords (looking for the literal string “London”) is fragile. The big
idea replaces keyword matching with matching by meaning, using
vectors — which is exactly what Parts 3 and 4 build toward. Everything
else stays the same.

Part 2 — Your First RAG (The Simplest

Possible Version)
Before using any fancy tools, it helps to build RAG by hand so you
understand what’s really happening. This “brute-force” version isn’t
clever, but it makes the concept concrete.

Step 1: Load your knowledge base

A knowledge base is just your collection of documents — the source of
truth the chatbot will draw from. Here we read each employee and
product file into a Python dictionary, where the key is a name and the
value is the file’s text.

6 / 35
 import glob
from pathlib import Path

knowledge = {}

# Grab every file in the employees folder

for filename in [Link]("knowledge-base/employees/*"):
name = Path(filename).[Link](' ')[-1]
# last word of the filename
with open(filename, "r", encoding="utf-8") as f:
knowledge[[Link]()] = [Link]()

Now knowledge["lancaster"] gives you the full text about the

employee Lancaster.

Step 2: Find relevant context by keyword matching

The simplest way to “retrieve” is to check whether any word in the user’s
question matches a key in our dictionary. If it does, that document is
probably relevant.

def get_relevant_context(message):
# keep only letters and spaces, then lowercase and split into
words
text = ''.join(ch for ch in message if [Link]() or [Link]
ace())
words = [Link]().split()
return [knowledge[word] for word in words if word in
knowledge]

get_relevant_context("Who is Lancaster and what is carllm?")

# returns the Lancaster document and the carllm document

Step 3: Add the context to the prompt

The retrieved documents are stitched into a system prompt. A system
prompt is a special instruction that sets the model’s role and gives it
background before it sees the user’s question.

7 / 35
 SYSTEM_PREFIX = """
You represent Insurellm, the Insurance Tech company.
You are an expert in answering questions about Insurellm; its
employees and its products.
You are provided with additional context that might be relevant to
the user's question.
Give brief, accurate answers. If you don't know the answer, say
so.

Relevant context:
"""

def additional_context(message):
relevant = get_relevant_context(message)
if not relevant:
return "There is no additional context relevant to the
user's question."
return "The following context might be relevant:\n\n" +
"\n\n".join(relevant)

Telling the model “if you don’t know the answer, say so” is a simple but
important guardrail against hallucination.

Step 4: Ask the model

from openai import OpenAI

openai = OpenAI()

def chat(message, history):

system_message = SYSTEM_PREFIX + additional_context(message)
messages = [{"role": "system", "content": system_message}] + h
istory + \
[{"role": "user", "content": message}]
response = [Link](model="gpt-4.1-
nano", messages=messages)
return [Link][0].[Link]

8 / 35
Notice the messages list. LLM chats are structured as a list of messages,
each with a role: - system — instructions and context (what we built
above), - user — the person’s question, - assistant — the model’s
replies (stored in history so it remembers the conversation).

Step 5: Give it a simple interface with Gradio

Gradio is a Python library that builds a web chat window in one line —
perfect for quickly trying out your idea (a prototype).

import gradio as gr
[Link](chat, type="messages").launch()

Why this simple version isn’t enough: it only finds documents when
the user types an exact keyword. Ask “Who leads the sales team?”
without naming a person, and it retrieves nothing. Real questions rarely
contain the exact words in your files. To fix this, we need to match by
meaning, not by exact words. That’s what the rest of this guide is about.

Part 3 — Chunking: Breaking Documents

Into Pieces

Why not just feed whole documents to the model?

Two reasons:

1. Context limits. An LLM can only read so much text at once. That
budget is measured in tokens.
2. Precision. If you hand the model an entire 10-page document to
answer one small question, most of it is noise. Smaller, focused
pieces give better, cheaper answers.

9 / 35
What is a token?
A token is a chunk of text the model actually processes — often a word
or part of a word. Models are priced and limited by tokens, so counting
them matters. The tiktoken library counts them:

import tiktoken
encoding = tiktoken.encoding_for_model("gpt-4.1-nano")
tokens = [Link](entire_knowledge_base)
print(f"Total tokens: {len(tokens):,}")

What is chunking?
Chunking means splitting your documents into smaller passages
(“chunks”) — for example, roughly 1,000 characters each. Each chunk
becomes an independent piece that can be searched and retrieved on
its own.

Chunk size and overlap

Two settings control chunking:

• Chunk size — how big each piece is.

• Chunk overlap — how much text is repeated between neighboring
chunks.

Overlap matters because an important sentence might sit right at a

boundary. If chunk 1 ends mid-thought and chunk 2 begins mid-
thought, a search might miss it. Repeating some text across the
boundary (say 200 characters) means the full idea appears intact in at
least one chunk.

10 / 35
 from langchain_text_splitters import
RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, ch
unk_overlap=200)
chunks = text_splitter.split_documents(documents)
print(f"Divided into {len(chunks)} chunks")

The tools that do this splitting are called text splitters (sometimes
“chunkers”). The simplest is the plain CharacterTextSplitter , which just
cuts every N characters — quick but blunt, since it can slice through the
middle of a word or sentence. The RecursiveCharacterTextSplitter is
smarter: it tries to split at natural boundaries in order of preference
(first at blank lines between sections, then at single line breaks, then at
sentence ends, then at words) so chunks stay readable rather than
being cut mid-thought.

Chunking is experimental. There is no single “correct” chunk size

or splitter. Getting chunking right is trial and error — you try
different sizes and strategies and measure what gives the best
answers (Part 6 shows how to measure). The best approach is
informed trial and error: notice a specific problem (say, bad
answers to a certain kind of question), form a hypothesis, change
the chunking, and check whether it improved.

How big can a knowledge base get?

To put scale in perspective: a sample knowledge base of 76 files might
contain around 300,000 characters, which is roughly 64,000 tokens.
That’s small enough to technically fit inside a large model’s context
window — you could paste the whole thing into one prompt. But doing
that is like re-sending an entire book with every single question: it
works at small scale, gets expensive quickly, and becomes impossible
once your data is 10× or 100× bigger. RAG exists precisely so you can
support massive knowledge bases (hundreds of thousands of

11 / 35
documents) while only ever sending the model the handful of chunks
that matter. That’s the whole point: scale without drowning the model —
or your budget — in irrelevant text.

Attaching metadata
Metadata is extra descriptive information stored alongside each chunk
— for example, whether it came from the employees , products ,
contracts , or company folder. This lets you track where an answer
came from later.

import os, glob

from langchain_community.document_loaders import DirectoryLoader,
TextLoader

documents = []
for folder in [Link]("knowledge-base/*"):
doc_type = [Link](folder) # e.g. "employees"
loader = DirectoryLoader(folder, glob="**/*.md",
loader_cls=TextLoader,
loader_kwargs={'encoding': 'utf-8'})
for doc in [Link]():
[Link]["doc_type"] = doc_type # remember the
source type
[Link](doc)

Part 4 — Embeddings and Vectors: Searching

by Meaning
This is the idea that makes real RAG work. Slow down here — it’s the
heart of everything.

12 / 35
The problem with keyword search
Our Part 2 chatbot only matched exact words. But “car insurance” and
“auto coverage” mean nearly the same thing while sharing no words. We
want to search by meaning, not spelling.

Two families of LLMs: auto-regressive and auto-

encoding
Where do these “meaning numbers” come from? It helps to know that
there are actually two kinds of language model:

• Auto-regressive models — the familiar kind (GPT, Claude, Gemini,

Llama, and so on). “Auto-regressive” means the model looks back
over the input and predicts the next token, one token at a time,
feeding each new token back in to predict the one after. This is the
model that generates text.
• Auto-encoding models — a different kind, trained to read an entire
input and produce a single output that reflects the whole thing. That
output might be a classification (for example, “is this review positive
or negative?”), or — the case we care about — a list of numbers that
captures the meaning of the whole input. That list of numbers is an
embedding.

So when we “turn text into a vector,” we’re using an auto-encoding

model (an encoder) rather than a text-generating one. (Side note for
the curious: vectors also flow inside generating models between their
layers, but you don’t need that detail to use RAG.)

What is an embedding?
An embedding is a list of numbers (called a vector) that represents the
meaning of a piece of text. A model reads a chunk and outputs, say, 384
or 3,072 numbers that capture its meaning. The key property:

13 / 35
 Texts with similar meanings get similar numbers. Texts with
different meanings get very different numbers.

So “car insurance” and “auto coverage” end up close together in this

“number space,” even though they share no words. Think of it as giving
every chunk a set of coordinates on a giant map of meaning, where
related ideas sit near each other.

• The number of values in each vector is its dimensions (e.g. 384

dimensions). You can think of the numbers as coordinates locating
that piece of text as a point in a high-dimensional space of meaning.
• To find relevant chunks, we turn the question into a vector too, then
look for chunks whose vectors are closest to it. Closeness in this
space means similarity in meaning.

A little history (so the idea feels less magical)

The idea of mapping words to meaningful vectors is older than today’s
chatbots:

• word2vec (from the earlier era of NLP, Natural Language Processing)

mapped each individual word to a vector. Similar words landed near
each other — the vector for “orange” sat close to “tangerine” and
“mandarin.” It even captured relationships, so that arithmetic like
king − man + woman ≈ queen roughly worked.
• BERT (created by Google in 2018, before the term “LLM” was
common) stands for Bidirectional Encoder Representations from
Transformers. It could embed whole sentences and is still used today.
• OpenAI embeddings (like text-embedding-3-large ) are a modern,
high-quality family of encoders.

The takeaway: an encoder takes text in and maps it to a vector such

that numbers close together mean similar meanings. That’s all you really
need to hold onto.

14 / 35
How “closeness” is actually measured: cosine similarity
We keep saying vectors that are “close” mean similar things. Strictly
speaking, closeness here isn’t measured by plain straight-line distance
but by cosine similarity — a measure of whether two vectors point in
the same direction, regardless of their length. You don’t need the math
to use RAG; just know that “similar direction = similar meaning,” and
that vector databases handle this comparison for you.

A nice bonus: fuzzy matching and typo tolerance

Because search is by meaning and not by exact spelling, embeddings
give you fuzzy matching for free. If a user asks about “Avery Lancaster”
but misspells it, the misspelled version still produces a vector very close
to the correct one, so the right chunk is still retrieved. Ask about a
product but get its name slightly wrong, and meaning-based search still
finds it. This is a huge upgrade over the exact-keyword approach from
Part 2, which would simply fail on any typo.

What is a vector store?

A vector store (or vector database) is a specialized database that stores
these vectors and can quickly find the ones most similar to a query
vector. Chroma is the vector store used throughout these notes.

Picking an embedding model

You need a model that turns text into vectors. Two common choices:

• HuggingFace all-MiniLM-L6-v2 — free, runs on your own machine,

produces 384-dimensional vectors. Great for keeping costs low.
• OpenAI text-embedding-3-large — a paid, higher-quality option
with many more dimensions.

15 / 35
 from langchain_huggingface import HuggingFaceEmbeddings
from langchain_chroma import Chroma

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")

# Build the vector store from our chunks and save it to disk
vectorstore = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
persist_directory="vector_db"
)
print(f"Vectorstore created with
{vectorstore._collection.count()} documents")

Persisting to disk ( persist_directory ) means you compute the

embeddings once and reuse them, instead of recomputing every time.

Inspecting the vectors

collection = vectorstore._collection
sample = [Link](limit=1, include=["embeddings"])["embeddin
gs"][0]
print(f"{[Link]():,} vectors, each with {len(sample):,}
dimensions")

Seeing the invisible: visualizing embeddings

Vectors have hundreds of dimensions, which humans can’t picture. To
get intuition, we squash them down to 2 or 3 dimensions using t-SNE (t-
distributed Stochastic Neighbor Embedding) — a technique that reduces
many dimensions to a few while trying to keep similar points near each
other. Then we plot them.

16 / 35
 import numpy as np
from [Link] import TSNE
import plotly.graph_objects as go

result = [Link](include=['embeddings', 'documents', 'metad

atas'])
vectors = [Link](result['embeddings'])

tsne = TSNE(n_components=2, random_state=42) # reduce to 2D

reduced = tsne.fit_transform(vectors)

fig = [Link](data=[[Link](
x=reduced[:, 0], y=reduced[:, 1], mode='markers'
)])
[Link]()

When you color each point by its document type, you’ll typically see
chunks of the same type clustering together. That visible clustering is
proof the embeddings really did capture meaning.

Part 5 — A Clean RAG Pipeline with

LangChain
LangChain is a popular framework that packages the common RAG
steps into reusable building blocks, so you write less plumbing code.
Once your vector store exists, a full RAG system needs only two main
objects.

What LangChain is (and its trade-offs)

LangChain is an open-source framework launched in October 2022 by
Harrison Chase, who later built a company around it. It’s an example of
an abstraction layer — a piece of software that sits on top of many
different tools (different LLM providers, different databases) and gives
you one consistent way to work with them and to chain them together

17 / 35
into pipelines. A major new version, LangChain 1.0, arrived in October
2025 and reorganized much of the framework, so you’ll see references
to a “pre-1.0” and “post-1.0” world.

The good:

• It gets you to a working assistant, RAG pipeline, or summarizer very

quickly, with a lot of functionality (like tool use) available out of the
box.
• It’s widely adopted in companies, so it’s useful, recognizable
experience to have.

The trade-offs:

• When it first appeared, calling OpenAI, Anthropic, and Google each

required different code, so a unifying layer was genuinely needed.
Today most providers offer OpenAI-compatible endpoints, so you can
often switch models just by changing a base URL — reducing the
need for a heavy abstraction layer.
• LangChain has grown large over the years. It has a fair amount of its
own terminology and concepts to learn (including its own mini-
language, LCEL, the LangChain Expression Language), which some
find heavyweight. Lighter alternatives now exist.

LangChain is also part of a broader ecosystem: LangGraph (for

connecting agents together in a dependency graph), LangSmith (an
observability platform for monitoring LLM apps), and others.

LangChain comes in many packages

Unlike a single-package tool, LangChain is split across several packages
you install separately, each with its own dependencies — another sign
of how much it does. You’ll import from packages like langchain_openai
(OpenAI classes), langchain_chroma (the Chroma database),

18 / 35
langchain_huggingface (HuggingFace models), langchain_community
(community-contributed document loaders), and
langchain_text_splitters (the chunking tools from Part 3).

A lighter alternative: litellm

Some of the later code uses litellm (written as “light LM”) instead. It’s a
very lightweight abstraction layer with almost no learning curve: you
import one package and call a single completion() function to talk to
any model, switching providers freely. Where LangChain gives you lots
of structure at the cost of complexity, litellm gives you a thin, simple
wrapper. Both are valid — the right choice depends on how much
structure your project needs.

from litellm import completion

response = completion(model="gpt-4.1-nano", messages=messages)
answer = [Link][0].[Link]

The Document object

When LangChain loads a file, it produces a Document object with two
key fields: page_content (the actual text) and metadata (extra info like
its source path and the doc_type you attached in Part 3). Keeping this
in mind makes the retriever’s output easier to read — the chunks that
come back are just these Document objects.

Connect to the saved vector store

from langchain_chroma import Chroma

from langchain_huggingface import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = Chroma(persist_directory="vector_db", embedding_func
tion=embeddings)

19 / 35
Use the same embedding model you built the store with — questions
and chunks must live in the same “meaning space” to be comparable.

The two key objects: retriever and LLM

• A retriever searches the vector store and returns the most relevant
chunks for a question.
• An LLM generates the final answer.

from langchain_openai import ChatOpenAI

retriever = vectorstore.as_retriever()
llm = ChatOpenAI(temperature=0, model_name="gpt-4.1-nano")

What is temperature?
Temperature controls how varied the model’s output is by influencing
which word it picks next:

• temperature=0 → always choose the most likely next word.

Predictable, consistent answers. Best for factual Q&A.
• Higher temperature → more variety and surprise. Better for creative
writing.

A common misconception is that temperature is “creativity.” More

precisely, it controls the randomness of word selection. (Note: even at
temperature=0 , answers aren’t guaranteed identical every time; full
reproducibility also needs a fixed random seed.) If you want a creative
style, the better lever is usually the system prompt, not temperature.

The invoke() method

LangChain objects share a simple invoke() method — you call it to run
them:

20 / 35
 [Link]("Who is Avery?") # returns relevant chunks
[Link]("Who is Avery?") # returns a raw model answer
(no retrieval)

Putting it together
Now combine the pieces into the full RAG loop: retrieve chunks → build
a prompt with them → ask the LLM.

from langchain_core.messages import SystemMessage, HumanMessage

SYSTEM_PROMPT_TEMPLATE = """
You are a knowledgeable, friendly assistant representing the
company Insurellm.
You are chatting with a user about Insurellm.
If relevant, use the given context to answer any question.
If you don't know the answer, say so.
Context:
{context}
"""

def answer_question(question, history):

docs = [Link](question) #
1. retrieve
context = "\n\n".join(doc.page_content for doc in docs) #
2. gather text
system_prompt =
SYSTEM_PROMPT_TEMPLATE.format(context=context) # 3. augment
response = [Link]([
SystemMessage(content=system_prompt),
HumanMessage(content=question),
]) #
4. generate
return [Link]

Wrap it in Gradio and you have a working, meaning-aware assistant:

import gradio as gr
[Link](answer_question).launch()

21 / 35
The surprising lesson: a genuinely useful RAG system is just retrieve →
stuff into prompt → generate. It’s far simpler than most people expect.

From notebook to production: organizing the code

Experimenting in a notebook is great for learning, but to actually ship a
system you move the code into proper Python modules (reusable .py
files). A clean way to organize a RAG system is two modules inside a
package (a folder), often called implementation :

• [Link] — the data pipeline: read the knowledge base, split into
chunks, turn chunks into vectors, and store them in Chroma. You
could run this automatically whenever new files arrive.
• [Link] — the query side: it exposes two key functions,
fetch_context(question) (get relevant chunks) and
answer_question(question, history) (fetch context, then answer).

A separate [Link] can then launch a Gradio interface on top of these.

The benefit of this structure is that it’s swappable: as long as any new
implementation provides the same fetch_context and
answer_question functions, you can drop in a better version (with
smarter chunking or reranking) without changing anything else. This is
exactly how you’d upgrade a basic RAG into an advanced one.

Part 6 — Evaluation: Is Your RAG Any Good?

Building a RAG system is only half the job. The other half — often the
more important half — is measuring how well it works. Without
evaluation you’re just guessing.

Evaluation puts a scientific framework around what is otherwise an art

form. Without numbers you’re just guessing whether a change helped.
The real skill — the part that connects engineering to business value —
is deciding what to measure so that improving the number actually

22 / 35
improves the outcome your business cares about. With good evaluations
in place, improving a RAG system becomes a loop: measure → change
something → measure again → keep what helped.

RAG has two parts to evaluate separately, because they can fail
independently:

1. Retrieval — did we fetch the right chunks?

2. Answer — given those chunks, did the model write a good answer?

A test set
You start with a set of test questions, each paired with a known-correct
reference answer and some keywords that a good answer or retrieval
should contain, plus a category label.

A convenient file format for storing these tests is JSONL ( JSON Lines). In
a JSONL file, each line is its own complete JSON object (a dictionary), but
the file as a whole is not one big JSON array — there are no
surrounding square brackets and no commas between lines. This makes
it easy to append new tests one line at a time. Think of it as the JSON
equivalent of a CSV file, where each line is a “row.” It’s a common format
for test data and for batch API calls.

{"question": "Who won the prestigious IIOTY award in 2023?",

"keywords": ["Maxine", "Thompson", "IIOTY"], "reference_answer":
"Maxine Thompson won...", "category": "direct_fact"}
{"question": "How long has Avery worked at Insurellm?",
"keywords": ["Avery", "founding"], "reference_answer": "...",
"category": "temporal"}

You don’t have to write hundreds of tests by hand. LLMs are

excellent at generating synthetic data — you write the first example or
two, then ask a model to produce many more in the same format. This
is a fast way to build a “golden” test set of, say, 150 questions for your
own knowledge base.

23 / 35
Loading the tests can be made clean with Pydantic (a Python library
where you define a data shape as a class that subclasses BaseModel ).
That lets you access fields by name ( [Link] , [Link] )
instead of dictionary lookups.

from collections import Counter

# See how many tests fall into each category
Counter([Link] for t in tests)
# e.g. {'direct_fact': 70, 'temporal': 20, 'spanning': 20, ...}

example = tests[0]
print([Link]) # "Who won the IIOTY award in
2023?"
print([Link]) # "direct_fact"
print(example.reference_answer) # the ideal answer
print([Link]) # ['Maxine', 'Thompson', 'IIOTY']

Categories of questions — and where RAG struggles

Tagging each test with a category lets you see which kinds of questions
your system handles well or poorly. Common categories:

• Direct fact — a single fact found in one place (“Who won the 2023
award?”).
• Temporal — time-based (“How long was someone in a role?”).
• Comparative — comparing two things (“Who has been here
longer?”).
• Numerical — questions about numbers, like salaries above a
threshold.
• Relationship — who relates to whom (“Who reports to whom?”).
• Spanning — questions needing information from many documents
at once (“How many employees earn more than \$60,000?”).
• Holistic — questions about the knowledge base as a whole,
answerable only if you’d read most of it.

24 / 35
Two of these — spanning and holistic — are RAG’s Achilles’ heel. Basic
RAG only pulls a few chunks at a time, so a question whose answer is
scattered across dozens of documents (or requires an overview of
everything) simply can’t be answered from a handful of chunks. Knowing
this weakness is what motivates advanced techniques like hierarchical
RAG (Part 7).

Evaluating retrieval
Retrieval metrics ask: were the correct chunks found, and were they near
the top of the results? (Higher-ranked results matter more, since the
model pays most attention to what you show first.) Common measures:

• MRR (Mean Reciprocal Rank) — rewards putting the first correct

chunk as high as possible. It’s simply 1 divided by the position of the
first correct result: a correct chunk in position 1 scores 1.0, position 2
scores 0.5, position 3 scores 0.33, and so on. Higher is better.
• NDCG (Normalized Discounted Cumulative Gain) — rewards having
relevant chunks ranked highly overall. “Normalized” means it’s scored
against the best possible ordering, so you aren’t penalized for having
fewer relevant chunks than positions. Higher is better. (MRR and
keyword coverage are the more commonly used everyday metrics;
NDCG is a useful extra.)
• Keyword coverage — what fraction of the expected keywords
actually showed up in the retrieved chunks. It behaves like a recall-
style measure (did we surface the important terms at all?).

evaluate_retrieval(example)
# RetrievalEval(mrr=0.17, ndcg=0.29, keywords_found=2,
# total_keywords=3, keyword_coverage=66.7)

25 / 35
Evaluating the answer with an “LLM-as-judge”
How do you score free-form text answers automatically? A popular
approach is LLM-as-judge: you use a second LLM to read the generated
answer, compare it to the reference answer, and score it. Typical scores:

• Accuracy — is the answer factually correct?

• Completeness — does it include everything it should?
• Relevance — does it actually address the question asked?

eval, answer, chunks = evaluate_answer(example)

print([Link]) # e.g. 5.0
print([Link]) # e.g. 4.0 (missed the surname → less
complete)
print([Link]) # e.g. 5.0
print([Link]) # a written explanation of the scores

The written feedback is especially useful: it tells you why an answer lost
points, which points you toward what to fix — better chunking, better
retrieval, or a better prompt.

How the judge returns clean scores: structured outputs

To get reliable scores back (rather than a paragraph you’d have to
parse), the judge uses a technique called structured outputs. Instead
of letting the model reply with free-form text, you tell it: “reply as JSON
that follows this exact shape.” You define that shape with a Pydantic class
and pass it to the model call via a response_format parameter. The
model is then forced to return data matching your schema — here, four
fields: feedback, accuracy, completeness, and relevance.

26 / 35
 from pydantic import BaseModel
from litellm import completion

class AnswerEval(BaseModel):
feedback: str
accuracy: float
completeness: float
relevance: float

response = completion(model=MODEL, messages=messages, response_for

mat=AnswerEval)
result = AnswerEval.model_validate_json([Link][0].messag
[Link])

Structured outputs are worth remembering — they’re a broadly useful

way to make an LLM return predictable, machine-readable data instead
of prose.

Part 7 — Advanced RAG: Techniques for

Better Answers

First, an honest admission: RAG is a bit of a hack

It’s worth naming this openly. Transformers (the neural networks behind
LLMs) are already very good at paying attention to what matters in their
input — that’s literally what they’re trained to do. RAG exists because we
have too much information to hand them all at once, so we do a rough,
non-scientific job of guessing which chunks are relevant and feed only
those in. Seen this way, RAG is a workaround, and it comes with a whole
“zoo” of further hacks piled on top to guess relevance better. That’s not
a criticism — it’s just useful context for why there are so many
techniques, each patching a weakness of the last.

27 / 35
The basic pipeline works, but you can make retrieval noticeably smarter.
Here the code drops LangChain and works “natively” (calling the tools
directly) for maximum flexibility and control.

A quick tool note: these examples use Pydantic and structured outputs.
Pydantic is a Python library for defining data shapes as classes.
Combined with an LLM feature called structured output (via
response_format ), it forces the model to reply in an exact JSON
structure you define — no messy free text to parse.

from pydantic import BaseModel, Field

class Chunk(BaseModel):
headline: str = Field(description="A brief heading, likely to
match a query")
summary: str = Field(description="A few sentences summarizing
the chunk")
original_text: str = Field(description="The exact original
text, unchanged")

Technique 1: Smart chunking with an LLM (semantic

chunking)
Instead of splitting purely by character count, ask an LLM to divide a
document into sensible, self-contained chunks. It can respect meaning
and natural boundaries far better than a mechanical splitter. This idea —
splitting a document by meaning rather than by fixed length, so each
chunk covers one coherent topic (like one stage of a career, or one
feature of a product) — is called semantic chunking. LangChain even
ships a semantic chunker in its experimental package, but you can also
achieve it simply by asking an LLM to break a document into meaningful
pieces.

28 / 35
Technique 2: Document pre-processing (enriching
chunks)
This is a powerful idea. Rather than storing raw text, have the LLM
rewrite each chunk into a more searchable form: add a headline and a
short summary, then keep the original text too. You embed all three
together.

Why this helps: a user’s question often matches a clear summary or

headline more closely than it matches the raw source text. Enriching the
chunk makes it easier to find.

def as_result(self, document):

metadata = {"source": document["source"], "type": document["ty
pe"]}
return Result(
page_content=[Link] + "\n\n" + [Link] +
"\n\n" + self.original_text,
metadata=metadata,
)

Then you embed these enriched chunks and store them in Chroma, this
time using OpenAI’s higher-quality text-embedding-3-large model:

emb = [Link](model="text-embedding-3-large",
input=texts).data
vectors = [[Link] for e in emb]
[Link](ids=ids, embeddings=vectors, documents=texts, metad
atas=metas)

Technique 3: Query rewriting

Users ask messy, vague, or conversational questions. Query rewriting
uses an LLM to turn the user’s question into a short, focused search
query before hitting the vector store — one that’s more likely to surface
the right chunks.

29 / 35
 def rewrite_query(question, history=[]):
message = f"""
You are answering questions about the company Insurellm.
You are about to search a Knowledge Base to answer the user's
question.
Conversation so far: {history}
The user's current question: {question}
Respond ONLY with a single, very short specific question to search
the Knowledge Base.
"""
response = completion(model=MODEL, messages=[{"role":
"system", "content": message}])
return [Link][0].[Link]

Rewriting also uses conversation history to resolve context — turning a

follow-up like “What about her?” into a standalone, searchable question.

Technique 4: Reranking
Vector search is fast but imperfect: the truly best chunk isn’t always
ranked first. Reranking fixes this in two stages:

1. Retrieve broadly — pull back more chunks than you need (say the
top 10 or 20).
2. Rerank precisely — ask an LLM to carefully reorder those chunks by
true relevance, then keep the best ones.

30 / 35
 def rerank(question, chunks):
system_prompt = """
You are a document re-ranker.
You are given a question and a list of retrieved chunks.
Rank order the chunks by relevance, most relevant first.
Reply only with the list of ranked chunk ids.
"""
# ... build a user prompt listing every chunk with an id ...
response = completion(model=MODEL, messages=messages, response
_format=RankOrder)
order = RankOrder.model_validate_json([Link][0].mess
[Link]).order
return [chunks[i - 1] for i in order]

The payoff is real: a chunk that vector search buried at position 5 might,
after reranking, jump to position 0 — meaning the model now sees the
best evidence first.

The full advanced pipeline

Putting the advanced techniques together, one question flows like this:

def answer_question(question, history=[]):

query = rewrite_query(question, history) # 1. rewrite the
question
chunks = fetch_context_unranked(query) # 2. retrieve
broadly
chunks = rerank(query, chunks) # 3. rerank for
precision
messages = make_rag_messages(question, history, chunks) # 4.
augment prompt
response = completion(model=MODEL, messages=messages)
# 5. generate answer
return [Link][0].[Link], chunks

Including each chunk’s source in the prompt is a nice final touch — it

lets the assistant ground its answer in traceable documents.

31 / 35
The wider zoo of RAG techniques
The four techniques above are among the most useful, but they’re part
of a larger toolkit. Here are the other common members of the “zoo,”
each aimed at surfacing better context. You won’t always need them,
but knowing they exist tells you what to reach for when a particular
kind of question is failing.

• Chunking R&D. Systematically experiment with chunk sizes and

different splitters, guided by your evaluation metrics. Simple, but
often the highest-impact change.
• Encoder selection. Try different embedding models and pick the
best for your data and budget. A few practical notes: for images in a
knowledge base, you can use a multimodal encoder, but a more
reliable trick is to have a model write a text caption for the image
and embed that caption. For PDFs and Word files, don’t embed the
raw binary and don’t waste an LLM converting formats — use
ordinary Python libraries to convert them to markdown or text first,
then embed that.
• Prompt engineering / static context. The most “obvious” technique,
and the most often forgotten. Beyond the retrieved chunks, add
helpful fixed context to every prompt — basic company facts, the
current date, conversation history — anything that reliably helps. It
costs a few tokens and can make a big difference.
• Document pre-processing (covered as Technique 2) — rewriting
documents into a form better suited to retrieval before embedding
them (for example, turning a raw table of numbers into descriptive
sentences a question is more likely to match).
• Query rewriting / query pre-processing (covered as Technique 3) —
cleaning up the user’s question before searching, often using
conversation history to make a follow-up question self-contained.

32 / 35
• Query expansion. Instead of generating one search query, have the
LLM generate several related queries and retrieve chunks for each,
casting a wider net so you don’t miss relevant content phrased
differently.
• Reranking (covered as Technique 4) — reordering the retrieved
chunks (especially the larger pile you get after query expansion) so
the most relevant are first, then trimming the rest to avoid polluting
the prompt.
• Hierarchical RAG. A fix for RAG’s weakness with spanning and holistic
questions. You summarize your knowledge base at higher levels (a
summary of all products, a summary of all employees, a condensed
table of every salary) and store those summaries too. At query time
you search the coarse summaries first, then drill down into fine-
grained chunks. This gives the model a shot at “how many
employees earn over X?” questions that no single raw chunk could
answer. It’s admittedly hand-wavy — you can only pre-summarize
along dimensions you anticipated.
• Graph RAG. Documents often relate to each other (an employee has
a manager; a chunk belongs to a document). If you record those
relationships in each chunk’s metadata — or store everything in a
dedicated graph database (like Neo4j) that represents data as nodes
and connecting edges — then after finding a relevant chunk you can
also pull in its neighbors “one or two hops away.” It’s powerful when
your data is genuinely relationship-rich, but for most cases plain
metadata is enough; a full graph database is rarely essential.
• Agentic RAG. Rather than always doing a fixed “look up vectors →
stuff into prompt → answer” sequence, you give an LLM tools (a
vector-search tool, a SQL tool, a file-search tool) and let it decide
how to dig for the answer — including looping and trying again if
the first attempt falls short. In effect the model performs query
rewriting, expansion, and reranking on its own initiative. It’s flexible

33 / 35
 and can crack hard questions, but it’s less predictable and repeatable
than a fixed pipeline. (You could even close the loop by having it call
the evaluator on its own answer and retry until the scores are high.)

“RAG is dead” — or is it?

You’ll hear people claim RAG is obsolete, usually for one of two reasons:

1. “Context windows are now huge, so just paste everything in.”

True for small knowledge bases, but at real scale (hundreds of
thousands of documents) that’s slow and expensive. Some way of
throwing out the irrelevant 90% before calling the model will always
be valuable.
2. “Agents make the old pipeline old-fashioned.” Fair — but an agent
equipped with vector or graph search is still retrieving context to
augment its generation. That’s just RAG wearing new clothes.

The practical view: whatever you call it, if you’re using retrieval
techniques to feed relevant context into an LLM, that’s still Retrieval-
Augmented Generation. “Long live RAG.”

Putting It All Together: The RAG Journey

Here is the whole arc of what you’ve learned, from simplest to most
capable:

• Ingest your documents into a knowledge base.

• Chunk them into small, overlapping pieces (mechanically, or smartly
with an LLM).
• Optionally enrich each chunk with a headline and summary
(document pre-processing).
• Embed each chunk into a vector that captures its meaning, and
store it in a vector store like Chroma.

34 / 35
 • At question time, optionally rewrite the query, then retrieve the
most similar chunks, and optionally rerank them for precision.
• Augment the prompt with those chunks and let the LLM generate a
grounded answer.
• Evaluate both retrieval and answer quality so you know it’s actually
working — and improve from there.

Where to take this next: your own knowledge worker

RAG is one of the most prototypical business uses of generative AI, and
the same recipe works on your data. A great way to cement the skill is
to build a personal knowledge worker:

• Point it at your own files — documents from your local drive or

Google Drive — and ingest them into a vector store.
• Ask it questions across everything you’ve ever written or collected,
giving you a conversational assistant with expert knowledge of your
own material (often more than you remember yourself).
• Run it entirely on open-source models so it costs nothing and keeps
everything private — nothing leaves your computer.
• Go further with read-only access to your email and documents
(Google Workspace or Microsoft Office both offer APIs) so it can
answer using your full history of correspondence.

The best way to improve any of this is the same loop from Part 6:
measure, iterate, measure. There’s no magic trick — the trick is the
work of trying different chunking, prompts, encoders, and techniques
and keeping what moves the numbers.

The most encouraging takeaway: the core idea is genuinely simple —

look up relevant text, hand it to the model, ask it to answer. Everything
beyond that (better chunking, embeddings, reranking, query rewriting,
evaluation) is a refinement layered onto that one clear foundation.

35 / 35