Module 1

Introduction
Introduction to RAG
Module Overview

Introduction to RAG

RAG Architecture Deep Dive

Real-World Applications

Zain Hasan
Hands-on Projects

Module 1 Module 2 Module 3 Module 4 Module 5

Basic RAG Retriever Vector Database LLM Monitoring & Evaluation

Zain Hasan
Introduction to
RAG
Introduction to RAG
Retrieval Augmented Generation (RAG)

LLMS are already powerful RAG further improves them

Summarize Text

+
Generate Code

LLM New
Rewrite Content information

Zain Hasan
 Why are hotels expensive on the weekend?
"More people travel on weekends, so there’s more competition for rooms.”

Zain Hasan
Why are hotels in Vancouver super expensive
this coming weekend?

Zain Hasan
Why are hotels in Vancouver super expensive this
coming weekend?

Taylor Swift is in town this weekend

Zain Hasan
Why doesn’t Vancouver have more hotel capacity
close to downtown?

Zain Hasan
Two Steps for Answering Questions

“Retrieval” “Generation”
Collect Reason
Information & Respond

Why are hotels

expensive on the
No need to collect Respond based on
information your knowledge
weekend?

Why doesn’t Vancouver

have more hotel Synthesize
Extensive Research
capacity close to research
downtown?

Zain Hasan
 Traditional Language Model Usage

Very recent
Internet information Specialized
Data knowledge

US History

OR
Italian Dishes
Prompt LLM Great answer Not so great

Movies

Zain Hasan
 User Query
LLM
Why doesn’t Vancouver have more
hotel capacity close to downtown?

Urban Development in Vancouver

Mathematical Zoning laws and city planning since the early 1900s limited
Models hotel growth downtown.
Forum Comment
Probably because land costs are super high near
downtown.

Training Data

Massive dataset from the open internet

Web Pages Books Forums Code

Zain Hasan
What LLMs don’t know
Private Databases
Unfortunately, many types of information are missing
LLMS can’t access confidential information

Hard to access information

Some information isn’t widely available online, making it inaccessible to LLMs.

Real time data

LLMs are trained on past data and don’t update automatically

How do you make sure the LLM knows this useful information?

Zain Hasan
 Just put it in the prompt!

User Prompt News reports

Why are hotels in Vancouver
super expensive this coming +
weekend? Forum posts LLM

Zain Hasan
User Prompt

Why are hotels in Vancouver super

expensive this weekend?
LLM Response

Augmented Prompt
RAG System

Taylor Swift is performing her Eras Tour in

Vancouver. This weekend at BC Place Stadium on
December 6–8, 2024

Zain Hasan
Retriever

● Manages knowledge base of trusted

information
Retriever

● Finds the most relevant Information

and shares information with the LLM

● Improves generation
Knowledge
Base

Zain Hasan
Retrieval Augmented Generation

Zain Hasan
Applications of
RAG
Applications of RAG
Applications of RAG - Code Generation
LLM needs your project’s context
Classes, functions, definitions, and coding style

Use your codebase as a knowledge base

RAG retrieves project-specific content for the LLM

Improves code generation and Q&A

Answers are tailored to your actual repository

Zain Hasan
Applications of RAG - Company Chatbots

Tailored to your company

Every business has its own products, policies,
and communication style

Uses your internal documents

Manuals, support guides, FAQs

Grounds answers in real context

Reduces generic or incorrect answers

Zain Hasan
Applications of RAG - Specialized Knowledge
High-impact domains
Legal and medical use cases

Uses specialized documents

Case files, journals, private data

Enables accurate, secure use

Supports precision and privacy needs

Zain Hasan
Applications of RAG

Search engines as retrievers

Return websites for a given query

AI summarizes search results

Presents key info in a skimmable format

RAG with the internet as a knowledge base

Summaries powered by real-time retrieval

Zain Hasan
Personalized RAG

More software includes personal assistants Calendar Events

Messaging app, email client, etc. Emails

These tools need context Text

Contacts
More context leads to better results Messages

Your data is the knowledge base

Texts, contacts, etc.
Your Data

Zain Hasan
 + your data
LLM

Zain Hasan
 RAG Architecture
Overview

Introduction to RAG
Normal
LLM Use
Prompt LLM Response

Retriever Relevant
Documents
RAG
System

Knowledge
Base

Zain Hasan
Normal
LLM Use
Prompt

Retriever Relevant Augmented Prompt

Documents
RAG
System “Why are hotels in Vancouver so expensive this coming
weekend? Here are the five relevant articles that may
help you respond. <retrieved articles>”

Knowledge
Base

Zain Hasan
Normal
LLM Use
Prompt LLM Response
Added latency

Retriever Relevant Augmented Prompt

Documents
RAG
System TaylorareSwift
“Why hotels
is performing
in Vancouver
hersoEras
expensive
Tour inthis coming
weekend? Here
Vancouver. Thisareweekend
the five
at BC
relevant
Place Stadium
articlesonthat may
help you 6–8,
December respond.
2024<retrieved articles>”

Knowledge
Base
Better responses

Zain Hasan
Advantages of RAG
Injects missing knowledge
Retriever
Adds info not in the training data (e.g. policies, updates)

Reduces hallucinations
Grounds answers with relevant context

Keeps models up to date

Reflects new info by updating the knowledge base

Enables source citation Knowledge

Includes sources for verifiable answers Base

“Why are hotels in Vancouver so expensive this coming

Focuses model on generation weekend? Here are the five relevant articles that may
Retriever finds facts, LLM writes responses help you respond. <retrieved articles>”

Zain Hasan
Introduction to
LLMs
Introduction to RAG
LLMs are just fancy autocomplete

Zain Hasan
What a beautiful day, the sun is shining
rising
out Prompt

What a beautiful day the sun is shining

What a beautiful day the sun is rising Completions

What a beautiful day the sun is out

Zain Hasan
What a beautiful day, the sun is exploding

Neural Network sun

● a complex mathematical model of language
● stores which words frequently appear together,
in which order, and contextual meaning
shining
● LLMs use this model to generate text

Zain Hasan
 What a beautiful day, the sun is shining in the sky

Token
● a piece of a word London door
● some words get single tokens

● compound words use multiple tokens

unhappy programmatically
● punctuation marks
Completely, I agree!
● ~10,000 - 100,000 tokens in LLM’s vocabulary, allowing
models to represent any possible word with fewer tokens

Zain Hasan
What a beautiful day, the sun is

Process
Current State

80%
Vocabulary
~10,000 - 100,000 tokens
5%
Calculate
4%
Probabilities 1% .00002% .00001%

g
ht
g

ng

in

g
in

ou

in
ig
si

od
in

or
br
ri

pl
sh

sn
ex
Select
Next Token
shining

Zain Hasan
What a beautiful day, the sun is shining

Process
Current State

new tokens make sense

35%
in context of old ones
25%
Calculate 20%
Probabilities 10%

h
on

in
ug
ro
th
Select
Next Token
in

Zain Hasan
 shining in the sky

What a beautiful day, the sun is

warming our faces

Autoregressive
● “self-influencing”

● new tokens make sense in context of old ones

● running the same prompt leads to different completion

Zain Hasan
How LLMs Learn
Trains model
usually billions of parameters!

Large text collections LLM

Before training, LLMs generate gibberish

Billions of
parameters "Forward to Saturn’s dance floor!" she yowled,
tail transmitting disco beats.

Zain Hasan
How LLMs Learn
Update parameters

Accurate?
What a beautiful day is,
the sun _____ Predictions

Roses are red, violets are

LLM Billions of
parameters ___

Training

Zain Hasan
Why LLMs Hallucinate
• LLMs generate probable word sequences
LLMs just reproduce statistical patterns from their training data

• Knowledge gaps cause inaccurate responses

Responses can “sound right” but aren't true.

• Truthful ≠ probable
LLMs are designed to generate "probable" text, not truthful text

Zain Hasan
How RAG solves the problem

relevant context
“grounds” the LLM’s
responses

LLM Retriever Knowledge

Base

Zain Hasan
Why not add everything?
Higher Computational Cost Context Window Limit

● Longer prompts take more ● Eventually you hit the limit of

computation to run LLM's context window

● Model performs ● Smaller models: only a few

computationally complex thousand tokens
scan of every token
● Largest models: millions
● Scan happens before
generating each new token

Zain Hasan
Zain Hasan
 Introduction to
information retrieval
Introduction to RAG
 Your Question

How can I make New York style

pizza at home?

Collection Organization Search

Librarian
Books on Different helps you
many shelves and find best
Library topics sections sections or
books

Retriever
Documents in “index” for
searches
a database search
the index
Retriever

Zain Hasan
Search with a librarian

● Understands the meaning

of your question

● Identifies the right shelves

to search

● Eventually finds relevant

books
Librarian

Zain Hasan
 Your Question

How can I make New York style

pizza at home?

Pizza Basics

A History of NYC

“What does the

Retriever prompt mean?”
Sauce Secrets

“What documents in
the knowledge base Cooking at Home
are similar?”

Knowledge
Base

Zain Hasan
 Your Question

How can I make New York style

pizza at home?

Pizza Basics 0.95

A History of NYC 0.6

“What does the
Retriever prompt mean?”
Sauce Secrets 0.7

“What documents in
the knowledge base Cooking at Home 0.8
are similar?”

Knowledge
Base

Zain Hasan
Retriever Tradeoffs
● Relevance vs irrelevance
Need to return relevant documents and withhold irrelevant ones

● Return every document?

Mountains of irrelevant docs. Wastes context window

● Return the single highest ranked document?

Miss valuable information

● No perfect solution
Retriever usually doesn’t perfectly rank documents

● Monitor and experiment

Change settings to find what works
Zain Hasan
Search Engine
Retrieves relevant webpages

Database
Retrieves relevant tables and rows

Historical Context
Information Retrieval was already mature when LLMs were first developed

Zain Hasan
Practical Implementation

Relational Vector
Database Database
Specialized for retrieval
Already widely adopted
in a RAG system

Zain Hasan
 Module 1
Conclusion
Introduction to RAG
Key Concepts
● RAG pairs an LLM with a knowledge base

● Data is private, recent, or highly specific and so missing

from the LLM’s training data

● Retriever finds relevant documents and adds them to an

augmented prompt

● LLMs ground their responses in the retrieved information

Zain Hasan