01 / 15

SYSTEM DESIGN

Designing a
Production-Ready
RAG System
Stop thinking of RAG as "Embedding + Vector DB".
Start thinking in Layers.

A clean mental model to design RAG systems

that actually work in the real world.

Swipe to learn more →

RAG = 4 Core Layers 02 / 15

1 Data Layer
Ingestion, chunking & embeddings

2 Retrieval Layer
Vector DB, search & re-ranking

3 Generation Layer
Context building & LLM response

4 Evaluation + Governance
Metrics, guardrails & safety

If you understand these 4 layers clearly,

you can design any RAG system.
 L AY E R 1 — D ATA L AY E R

STEP 1 03 / 15
Data Ingestion

Raw → Clean → Structured

How it works:
1 Read from sources (PDF, DB, APIs, Web, Docs)

2 Extract raw text from each source

3 Remove noise — HTML tags, headers, footers

4 Normalize format (UTF-8, spacing, special chars)

5 Convert to clean, structured text

OUTPUT Clean text ready for chunking

This step decides your retrieval quality.

 L AY E R 1 — D ATA L AY E R

STEP 2 04 / 15
Chunking

Big Document → Small Meaningful Units

WHY?
LLMs cannot process entire books. We break text into meaningful
pieces that can be individually indexed and retrieved.

Chunking Types

Fixed-Size Recursive

Semantic Overlapping

Goal: Each chunk should represent ONE idea.

Bad chunks = Bad retrieval

 L AY E R 1 — D ATA L AY E R

STEP 3 05 / 15
Embeddings

Text → Vector in Semantic Space

Pass chunk to embedding model

Each text chunk is sent to the model

Convert to dense vector

Model outputs high-dimensional vector

VECTOR SIMILARITY
Similar meaning → closer vectors
Different meaning → far apart
Text becomes searchable using math.

Embedding model quality directly affects retrieval accuracy.

 L AY E R 2 — R E T R I E VA L L AY E R

STEP 4 06 / 15
Vector Database

Smart Memory for Vectors

1 2 3
Store Index Retrieve
• Chunk text • ANN algorithms • Query → vector
• Embedding vector • HNSW, IVF • Cosine similarity
• Metadata • Fast search • Top-K results

R E T R I E VA L F L O W

→ → →
User Convert Similarity Top-K
Query to Vector Search Chunks

Vector DB is not just storage.

It is optimized semantic search infrastructure.
 L AY E R 2 — R E T R I E VA L L AY E R

STEP 5 07 / 15
Retrieval Flow

Query → Vector → Similarity Search

User sends query

1 Natural language question enters the system

Query → Embedding
2 Same model converts query to vector

Search Vector DB
3 Find closest matching chunk vectors

Return Top-K chunks

4 Most relevant chunks are retrieved

But raw retrieval is not enough.

That's where re-ranking enters.
 L AY E R 2 — R E T R I E VA L L AY E R

STEP 6 08 / 15
Re-Ranking

First Filter → Deep Comparison

STAGE 1 STAGE 2

Fast Accurate
VS
Bi-Encoder Cross-Encoder
Separate encoding of query Joint reading of query +
and chunks for speed chunk for precision

KEY DIFFERENCE

EMBEDDINGS CROSS-ENCODER
Encode query & chunk Reads both jointly → better
separately relevance

OUTPUT
More precise context ranking
 L AY E R 2 — R E T R I E VA L L AY E R

STEP 7 09 / 15
Query Expansion

THE PROBLEM

Users don't always ask perfectly.

Expansion Techniques

Rewrite Query Sub-Queries

Rephrase for clarity Split into multiple angles

Add Synonyms Multi-Vector

Broaden vocabulary match Search across variants

R E S U LT
This increases recall.
Better recall = fewer missed answers.
 L AY E R 3 — G E N E R AT I O N L AY E R

STEP 8 10 / 15
Context Building

Assemble Evidence Before Answering

Assembly Steps
Take Top-K retrieved chunks

Apply token limit constraints

Merge with system prompt

Add original user query

FINAL PROMPT STRUCTURE

1 System Instructions — rules & behavior

2 Retrieved Context — top chunks

3 User Query — original question

↓ Sent to LLM ↓
 L AY E R 3 — G E N E R AT I O N L AY E R

STEP 9 11 / 15
Generation Layer

WITHOUT RAG WITH RAG

Raw LLM VS RAG LLM

Answers from parametric Answers using external
memory only context

RAW LLM RAG LLM

Hallucination risk Reduced hallucination

Static knowledge Dynamic knowledge

No citation Source grounded

Generation quality depends on:

Prompt design Context quality LLM capability
 L AY E R 4 — E VA L U AT I O N + G O V E R N A N C E

STEP 10 12 / 15
Evaluation Metrics

We evaluate two things separately:

Retrieval Level Metrics

Recall@K Precision@K

MRR nDCG

Generation Level Metrics

Faithfulness Relevance

Answer correctness Hallucination rate

If retrieval fails, generation fails.

 L AY E R 4 — E VA L U AT I O N + G O V E R N A N C E

STEP 11 13 / 15
Guardrails & Governance

WHY NEEDED?

Prevent unsafe output Remove toxic responses

Block data leakage Enforce policies

Guardrail Types

1 Input Filtering
Sanitize & validate incoming queries

2 Output Moderation
Check generated responses for safety

3 Prompt Injection Detection

Block adversarial manipulation attempts

4 Access Control
Role-based data access permissions

Production RAG always needs guardrails.

Full RAG Architecture 14 / 15

The Complete Mental Model

1. Data Layer
Ingestion → Chunking → Embeddings

2. Retrieval Layer
Vector DB → Search → Re-Rank → Query Expansion

3. Generation Layer
Context Building → LLM Response

4. Evaluation + Guardrails
Metrics → Safety → Governance

PRO TIP

If something breaks, debug layer by layer.

Don't debug randomly.
 15 / 15

If You Understand This,

You Can Design
Any RAG System.
Next: We'll break down each layer practically.

Follow for Deeper Builds

→