Building RAG Agents with

LLMs
Introduction

1
Building RAG Agents with
LLMs
Introduction

2
 Large Language Models
Backbones for Language Understanding

Backbones for language tasks, including classification and generation.

[Link]
3
 Dialog/Retrieval Agents
LLMs with Context and Control

User Asks
Something

Agent
Responds

[Link]
4
 Dialog/Retrieval Agents
LLMs with Context and Control

LLM Orchestration: Software + LLM

helps to route to software and LLMs.

User Asks
Something

Agent
Responds

[Link]
5
 Dialog/Retrieval Agents
LLMs with Context and Control

Retrieval: Tool runs algorithms

(database, code execution,
semantic search, return a constant
LLM Orchestration: Software + LLM value, etc) to provide context
helps to route to software and LLMs.

User Asks
Something

Agent
Responds

[Link]
6
 Dialog/Retrieval Agents
LLMs with Context and Control

Retrieval: Tool runs algorithms

(database, code execution,
semantic search, return a constant
LLM Orchestration: Software + LLM value, etc) to provide context
helps to route to software and LLMs.

Augmented: Based on tool

responses, the software pipeline
synthesizes some “context”
User Asks to feed to LLM w/ question.
Something

Agent
Responds

[Link]
7
 Dialog/Retrieval Agents
LLMs with Context and Control

Retrieval: Tool runs algorithms

(database, code execution,
semantic search, return a constant
LLM Orchestration: Software + LLM value, etc) to provide context
helps to route to software and LLMs.

Augmented: Based on tool

responses, the software pipeline
synthesizes some “context”
User Asks to feed to LLM w/ question.
Something
Generation: Based on question,
instructions, and enhanced context,
the LLM returns a response.
Agent
Responds

[Link]
8
 Chat Applications
Full Applications Build with LLMs

[Link]
9
 Chat Applications
Full Applications Build with LLMs

10
Production-Ready APIs That Run Anywhere | NVIDIA
 Chat Applications
Full Applications Build with LLMs

11
Production-Ready APIs That Run Anywhere | NVIDIA
Prerequisites
RAG Agents in Production

• Prior LLM/LangChain Exposure

• Intermediate Python Experience
• Exposure to Web Engineering

12 12
Course Objectives
RAG Agents in Production

• Environment
• LLM Services
• Intro to LangChain (LCEL)
• Running State Chains
• Document Loading
• Embeddings
• Document Retrieval
• RAG Evaluation

13 13
Building RAG Agents with
LLMs
Part 1: Your Course Environment

15
 Typical Jupyter Labs Interface

jupyter lab . &; open -a Safari [Link] 16

 Typical Jupyter Labs Interface

Web Browser

Your Device
Your Device
Files
Python
OS
C++
Hardware

jupyter lab . &; open -a Safari [Link] 17

 Typical Jupyter Labs Interface

Web Browser

:8888
Your Device
Your Device
Files
Python
OS Jupyter
C++
Hardware Labs

jupyter lab . &; open -a Safari [Link] 18

 Typical Jupyter Labs Interface

Web Browser

:8888
Your Device
Your Device
Files
Python
OS Jupyter
C++
Hardware Labs

jupyter lab . &; open -a Safari [Link] [Link] 19

 Typical Jupyter Labs Interface

Web Browser

:8888
Your Device Web Browser
Your Device
Files
Python
OS Jupyter Your Device
C++
Hardware Labs Files
Python
OS
C++
Hardware

jupyter lab . &; open -a Safari [Link] [Link] 20

 Typical Jupyter Labs Interface

Web Browser

:8888
Your Device Web Browser
Your Device
Files
Python
OS Jupyter Your Device
C++
Hardware Labs Files
Python
OS
C++
Hardware

jupyter lab . &; open -a Safari [Link] [Link] 21

DLI Jupyter Labs Interface

22
 DLI Jupyter Labs Interface

Remote Host

Your Device

Files
Jupyter Python
OS
Labs
C++
Hardware

Web Browser
Your Device
Your Device

23
 DLI Jupyter Labs Interface

Remote Host

Good to go!
… sort of …
Your Device

Files
Jupyter Python
OS
Labs
C++
Hardware

Web Browser
Your Device
Your Device

24
 DLI Jupyter Labs Interface

Remote Host More

Python
Data Scheduler Proxy
Processes?
Node
Loader Service
Env
Shell
Your Device

Files
Python
Jupyter
OS
Labs C++
Hardware

Web Browser
Your Device
Your Device

25
 DLI Jupyter Labs Interface

Remote Host More

Python
Data Scheduler Proxy
Processes?
Node
Loader Service
Env
Shell
Your Device

Files
Python
Jupyter
OS
Labs C++
Hardware

Dividing
Web Browser
Resources?
Your Device
Your Device

26
 Containerization with Docker
Compartmentalizing Functionality Into Microservices

Host
Python
Data Scheduler Proxy
Node
Loader Service
Env
Shell
Your Device

Files
Python Host
Jupyter
OS
Labs C++
Hardware
Python Sh Volume
:8070
Your Device
:8888
Files
:88 Volume Python
OS
Jupyter GPU/2 C++
Hardware

27
 Microservices Workflow

Web Browser
Your Device
Your Device

28
 Microservices Workflow

Remote Host
[Link] Resources
[Link] Services
[Link] Containers
[Link] Processes Your Device

Files
Python
OS
C++
Hardware

Web Browser
Your Device
Your Device

29
 Microservices Workflow

Remote Host
[Link] Resources
[Link] Services
[Link] Containers Python Sh Volume
:8070
[Link] Processes :8888
Your Device

Files
:88 Volume Python
OS
Jupyter GPU/2 C++
Hardware

Web Browser
Your Device
Your Device

30
 Microservices Workflow

Remote Host
[Link] Resources
[Link] Services Database
Data Loader
Environment
[Link] Containers :8070
[Link] Processes :8888
Your Device

Files
:88 Python
Jupyter Notebook OS
Environment C++
Hardware

Web Browser
Your Device
Your Device

31
 Microservices Workflow

Remote Host
[Link] Resources
[Link] Services Database
Data Loader
Environment
[Link] Containers :8070
[Link] Processes :8888
Your Device

Files
:88 Python
Jupyter Notebook OS
Environment C++
Hardware

Web Browser
Your Device
Your Device

32
 Scaling Containerized Applications

Arbitrary Host
Company GenAI
Database Database
Data Loader
Service
Environment
:8070
Your Device
:8888
Files
Python
Jupyter Notebook OS
Environment C++
Hardware

Web Browser
Your Device Other
Web
Your Device Other
Web Your Device Other
Web
Your Device Other
Web Browser
Devices Your Device
Browser Your Device
Devices Devices Browser
Devices
Browser

33
 Scaling Containerized Applications

Arbitrary Host
Company GenAI
Database Database
Data Loader
Service
Environment
:8070
Your Device
:8888
Files
Python
Jupyter Notebook OS
Environment C++
Hardware

Your Device
Your Device Your Device
Web Browser
Your Device Other
Web
Your Device Other
Web Your Device Other
Web
Your Device Other
Web Browser
Devices Your Device
Browser Your Device
Devices Devices Browser
Devices
Browser

34
Our Environment
Jupyter Notebooks

Remote Host

Your Device
Docker
Router

Jupyter Notebook Server

:88

35
35
Our Environment
Jupyter Notebooks + Frontend

:8090 Remote Host

Frontend
Your Device
Docker
Router

Jupyter Notebook Server

:88

36
36
Gradio

37
Simple Gradio ChatInterface

[Link] [Link] 38
Gradio in HuggingFace Spaces

[Link] 39
Custom Gradio Block Interface

[Link]
40
Our Environment
Jupyter Notebooks + Frontend

:8090 Remote Host

Frontend
Your Device
Docker
Router

Jupyter Notebook Server

:88

41
41
Building RAG Agents with
LLMs
Part 2: LLM Services

42
Our Environment
Jupyter Notebooks + Frontend

:8090 Remote Host

Frontend
Your Device
Docker
Router

Jupyter Notebook Server

:88

43
43
 Dialog/Retrieval Agents
LLMs with Context and Control

Execute with dialog management and information retrieval in production

[Link]
44
 Dialog/Retrieval Agents
LLMs with Context and Control

Execute with dialog management and information retrieval in production

[Link]
45
Standalone Environment LLM

Remote Host

Frontend
Your Device
Docker
Router

Jupyter Notebook Server

46
Standalone Environment LLM

Remote Host

Frontend
Your Device
Docker
Router

Jupyter Notebook Server LLM

47
 Standalone Environment LLM

H200
Jupyter Notebook
A100

A10
Jupyter Notebook
4070
<s>[INST]<<SYS>>
{{system_message}}
<</SYS>> <s>[INST]<<SYS>>
VRAM-bound You are a code generator.
{{instruction}} [/INST] Please provide Python code
Jupyter Notebook {{primer}} per the instruction.
CPU-only <</SYS>>

Write a Fibonacci method [/INST]

```python
## Implementation of Fibonacci w/

48
 Standalone Environment LLM

H200
Jupyter Notebook
A100

A10
Jupyter Notebook
4070

VRAM-bound
Jupyter Notebook
CPU-only

49
 Standalone Environment LLM

H200
Jupyter Notebook
A100

A10
Jupyter Notebook
4070

VRAM-bound
Jupyter Notebook
CPU-only

50
 Remote LLM Access

<s>[INST]<<SYS>>
{{system_message}}
<</SYS>>

{{instruction}} [/INST]
Remote Host {{primer}}
Llama

Frontend
<s>[INST]<<SYS>>
Your Device You are a code generator.
Please provide Python code
per the instruction.
<</SYS>>
VRAM-bound
Write a Fibonacci method [/INST]
Jupyter Notebook ```python
CPU-only ## Implementation of Fibonacci w/

51
 Large Model Hosting Platforms

Remote Host NVIDIA

GPU CLOUD
Frontend
Your Device

VRAM-bound
Jupyter Notebook
CPU-only

52
 Large Model Hosting Platforms

NVIDIA
GPU CLOUD

1.GPT4/3.5 [Link] Available Models

[Link] Tools/Multimodal Support [Link]/Query Router Accessible
[Link]/Query Router Internal [Link] to Local Deployment/Self-Hosting
[Link] to Local Deployment Less Clear [Link] to Compose Complex Systems

53
Query Router Access

GPT4

Dalle-3

Embed

<s>[INST]<<SYS>>
{{system_message}}
<</SYS>>

{{instruction}}
[/INST]
{{primer}}????

54
 Query Router Access

{
"messages": [{ GPT4
"content": "...",
"role": "system"
Dalle-3
},{ /chat/completions
Embed
"content": "...",
"role": "user" Facilitate
}],
"model": ”gpt-4”,
Monitor <s>[INST]<<SYS>>
{{system_message}}
"temperature": 0.2, Optimize <</SYS>>
"top_p": 0.7,
"max_tokens": 1024, Load Balance {{instruction}}
"stream": True [/INST]
} {{primer}}????

[Link] 55
 Query Router Access

{
"messages": [{ GPT4
"content": "...",
"role": "system"
Dalle-3
},{ /chat/completions
Embed
"content": "...",
"role": "user" Facilitate
}],
"model": ”gpt-4”,
Monitor <s>[INST]<<SYS>>
{{system_message}}
"temperature": 0.2, Optimize <</SYS>>
"top_p": 0.7,
"max_tokens": 1024, Load Balance {{instruction}}
"stream": True [/INST]
} {{primer}}????

[Link] 56
Large Model Hosting Platforms - OpenAI

o n ate
n eti g s er
s ti o pl i n en
el p l e o m dd s/ g
od m t/c be g e
m co ha em m a
c i

OpenAI Gateway

GPT embed-<> Dalle-3

(3.5/4)

57
 Large Model Hosting Platforms - OpenAI

o n ate
n eti g s er
s ti o pl i n en
el p l e o m dd s/ g
od m t/c be g e
m co ha em m a
c i
Messages
Model Name
Settings OpenAI Gateway Responses
API Key

GPT embed-<> Dalle-3

(3.5/4)

58
 Large Model Hosting Platforms - OpenAI

End-User Application

Retriever Microservice

o n ate
n eti g s er
s ti o pl i n en
el p l e o m dd s/ g
od m t/c be g e
m co ha em m a
c i
Messages
Model Name
Settings OpenAI Gateway Responses
API Key

GPT embed-<> Dalle-3

(3.5/4)

Server Management

[Link] 59
simplifying-ai-inference-in-production-with-
triton/
 Large Model Hosting Platforms

NVIDIA
GPU CLOUD

1.GPT4/3.5 [Link] Available Models

[Link] Tools/Multimodal Support [Link]/Query Router Accessible
[Link]/Query Router Internal [Link] to Local Deployment/Self-Hosting
[Link] to Local Deployment Less Clear [Link] to Compose Complex Systems

60
 Large Model Hosting Platforms

NVIDIA
GPU CLOUD

1.GPT4/3.5 [Link] Available Models

[Link] Tools/Multimodal Support [Link]/Query Router Accessible
[Link]/Query Router Internal [Link] to Local Deployment/Self-Hosting
[Link] to Local Deployment Less Clear [Link] to Compose Complex Systems

61
 Query Router Access GPT4

Dalle-3

Embed

<s>[INST]<<SYS>>
{
{{system_message}}
"messages": [{ <</SYS>>
"content": "...",
"role": "system" {{instruction}} [/INST]
},{ /chat/completions {{primer}} Mixtral SDXL
"content": "...",
"role": "user" Facilitate
}],
Monitor
"model": ”mixtral”,
{ E5
"temperature": 0.2, Optimize “context”:{{context}}
"top_p": 0.7, “model”:“query”/”doc”
"max_tokens": 1024, Load Balance }
"stream": True
}

[Link] 62
Full Deployment Stack
General
Chatbot
Image Document
Generator Copilot

Custom
Server
NeMo
NIMs
Retriever

Video
Mixtral E5 SDXL RIVA Gen
TensorRT-LLM / vLLM

K8s
Azure Org
/AWS Cluster
[Link] 63
simplifying-ai-inference-in-production-with-
triton/
 Our Environment
Jupyter Notebooks + Frontend
+ LLM Client

:8090 Remote Host

Frontend LLMs

Your Device
Docker
Router
:9000

Jupyter Notebook Server

:88

64
64
NVIDIA Foundation Model Endpoints

NVIDIA
GPU CLOUD

65
NVIDIA Foundation Model Endpoints

NVIDIA
GPU CLOUD

66
NVIDIA Foundation Model Endpoints

NVIDIA
GPU CLOUD

67
From Raw Requests to LangChain Model

69
 From Raw Requests to LangChain Model

<s>[INST]<<SYS>>
{{system_message}}
<</SYS>>
> llm(“Hello world”) {{instruction}} [/INST]
ChatMessage(content=”hello”) Query Router {{primer}} Llama Mistral

> embedder(“Hello”)
[0.4535437, 0.0800435, ...] Facilitate
Monitor { E5
“context”:{{context}}
Optimize “model”:“query”/”doc”
}

70
LLM Interfaces
The Whole Stack

[Link].
[Link]
/v2/nvcf

Scaled
Function
Deployment
Solution

71
71
 LLM Interfaces
The Whole Stack

[Link]/
[Link]/

[Link].
[Link]
[Link] /v2/nvcf

v1/models
v1/completions
v1/chat/completions
v1/embeddings

[Link]
Scaled
v1/models Function
v1/completions
Deployment
v1/chat/completions
Solution
v1/embeddings

72
72
 LLM Interfaces
The Whole Stack

[Link]/
[Link]/

NVIDIABase [Link].
[Link]
[Link] /v2/nvcf

v1/models
v1/completions
v1/chat/completions
v1/embeddings

[Link]
Scaled
v1/models Function
OpenAI v1/completions
Deployment
v1/chat/completions
Solution
v1/embeddings

73
73
 LLM Interfaces
The Whole Stack

[Link]/
[Link]/

NVIDIABase [Link].
[Link]
[Link] /v2/nvcf

v1/models
v1/completions
v1/chat/completions
v1/embeddings

[Link]
Scaled
v1/models Function
OpenAI v1/completions
Deployment
v1/chat/completions
Solution
v1/embeddings

74
74
 LLM Interfaces
The Whole Stack

[Link]/
[Link]/

ChatNVIDIA
NVIDIABase [Link].
[Link]
[Link] /v2/nvcf

v1/models
v1/completions
v1/chat/completions
v1/embeddings

[Link]
ChatOpenAI Scaled
v1/models Function
OpenAI v1/completions
Deployment
v1/chat/completions
Solution
v1/embeddings

75
75
Building RAG Agents with
LLMs
Part 3: Intro to LangChain

76
LangChain Structure

77
 Chain Building
Just The LLM

Input LLM Output

Hey AIMessage(“Hello”)
There
LLM

79
79
 Chain Building
Simple Prompt+LLM Chain

Input Prompt LLM Output

{‘input’ : “Hey Hey AIMessage(“Hello”)

There”} There
Prompt
Context: {context} LLM
Input: {input}

80
80
 Chain Building
Simple Prompt+LLM Chain

Input Prompt LLM Output

chain = prompt | chat

[Link](inputs)

{‘input’ : “Hey Hey AIMessage(“Hello”)

There”} There
Prompt
Context: {context} LLM
Input: {input}

81
81
 Chain Building
Simple Prompt+LLM Chain

## StrOutputParser()
Input Prompt LLM Output def get_content(value):
return getattr(value, “content”, value)

chain = prompt | chat | get_content

[Link](inputs)

{‘input’ : “Hey Hey AIMessage(“Hello”)

There”} There
Prompt
Context: {context} LLM
Input: {input}

82
82
 Chain Building
Invoking Runnables

Input Prompt LLM Output

chain = prompt | chat | StrOutputParser()

83
83
 Chain Building
Invoking Runnables

Input Prompt LLM Output

chain = prompt | chat | StrOutputParser() msg = [Link](...)

for token in [Link](...)

84
84
 Chain Building
Building Information Pipelines

Input Prompt LLM Output

External

85
85
 Chain Building
Building Information Pipelines

Input Prompt LLM Output

External

Input Prompt LLM

Prompt LLM Output

If/Else Prompt

Database
Internal

Output
86
86
 LangChain Extended Ecosystem

87
87
[Link]
 Our Environment
Jupyter Notebooks + Frontend
+ LLM Client

:8090 Remote Host

Frontend LLMs

Your Device
Docker
:9012 Router
:9000

Jupyter Notebook Server

:88

88
88
Building RAG Agents with
LLMs
Part 4: Running State Chain

89
 Chain Building
Recall Our Assumptions

External

Input Prompt LLM

Prompt LLM Output
What to do? No Lookup
If/Else Prompt
Needs Lookup

Off-Topic
Database
Internal

Output

90
90
 Chain Building
Towards Running State

Prompt
Input Classify LLM Topic
Sentence

inputs = {‘input’ : ‘sentence’}

cls_chain = cls_prompt | llm
topic = cls_chain.invoke(inputs)

91
91
 Chain Building
Towards Running State

Prompt Prompt
Input Classify LLM Topic Generate LLM Output
Sentence Sentence

gen_chain = out_prompt | llm

inputs = {‘input’ : ‘sentence’}
out_chain = cls_chain | gen_chain
cls_chain = cls_prompt | llm
for token in out_chain.stream(inputs):
topic = cls_chain.invoke(inputs)
print(token, end=””)

92
92
 Chain Building
Towards Running State

Prompt Prompt
Input Classify LLM Generate LLM Output
Sentence Sentence

Prompt
Combine LLM Output
Sentences

Input

inputs = {‘input’ : ‘sentence’}

new_sentence = out_chain.invoke(inputs)

93
93
 Chain Building
Towards Running State

Prompt Prompt
Input Classify LLM Generate LLM Output
Sentence Sentence

Prompt
Combine LLM Output
Sentences

Input

inputs = {‘input’ : ‘sentence’} [Link]({‘new’ : ‘new_sentence’})

new_sentence = out_chain.invoke(inputs) for token in merge_chain.stream(inputs):
print(token, end=””)

94
94
 Chain Building
Towards Running State

Prompt Prompt
Input Classify LLM Generate LLM Output
Sentence Sentence

Branch Chain Prompt

Combine LLM Output
Sentences

Input
RunnableAssign Branch Chain

out_chain = RunnableAssign({
new_sentence : cls_chain | gen_chain
}) | merge_chain

95
95
Typical Running State
Regular Fibonacci w/ While Loop

96
96
 Running State Chain Components
Towards LCEL While Loop

State
n=8
fib = [0,1]

97
97
 Running State Chain Components
Towards LCEL While Loop

State State
n=8 n=7
fib = [0,1] fib = [0,1,1]
msg=“Hello”

98
98
 Running State Chain Components
Towards LCEL While Loop

State
n=8 [0,1,1,2]
fib = [0,1,1]

99
99
 Running State Chain Components
Towards LCEL While Loop

State State
n=8 [0,1,1,2] n=8
fib = [0,1,1] fib = [0,1,1,2]

100
100
 Running State Chain Components
Towards LCEL While Loop

State
State State
n=8
n=8 n=8
fib =
fib = [0,1,1] fib = [0,1,1,2]
[0,1,1,2, 3]

101
101
Final Running State Loop

102
102
Typical Running State Loop
Comparing Typical with Running State

103
103
 Final Running State Loop
Big Picture

RunnableAssign

RunnableLambda

RunnableBranch

104
104
 Airline Chatbot

+Knowledge +Response
Base

Prompt DB Prompt Prompt

Input Update LLM Update LLM
Knowledge Lookup Format
Knowledge

+Customer
Info

History User

105
105
 Modern Chain Paradigms
Towards Powerful Running State

Unstructured
Prompt LLM
Generation

Input

106
106
 Modern Chain Paradigms
Towards Powerful Running State

Unstructured
Prompt LLM
Generation

Code Structured
Prompt Environment
LLM Retrieval
Input

[Link]
107
107
 Modern Chain Paradigms
Towards Powerful Running State

Unstructured
Prompt LLM
Generation

Code Structured
Prompt Environment
LLM Retrieval
Input

Grammar Guided
Prompt LLM
/Schema Generation

{“first_name” : “unknown”,
“last_name” : “unknown”, {“first_name” : “Jane”,
“confirmation” : -1} “last_name” : “Doe”,
Update given new info: “confirmation” : -1}
“Sure, my name is Jane Doe!” LLM 108
108
 Modern Chain Paradigms
Towards Powerful Running State

Unstructured
Prompt LLM
Generation

Code Structured
Prompt Environment
LLM Retrieval
Input

Grammar Guided
Prompt LLM
/Schema Generation

Tool
Prompt LLM Choice Tool Tooling
Tool
Schema 109
109
 Final Objective
Knowledge Base + Running State Chain

{ {
“first_name” : “unknown”, “first_name” : “Jane”,
“last_name” : “unknown”, “last_name” : “Doe”,
“confirmation” : -1, “confirmation” : -1,
... ...
} }
Update given new info:
LLM
“Sure, my name is Jane Doe!”

110
110
 Airline Chatbot

+Knowledge +Response
Base

Prompt DB Prompt Prompt

Input Update LLM Update LLM
Knowledge Lookup Format
Knowledge

+Customer
Info

111
111
 Airline Chatbot

+Knowledge +Response
Base

Prompt DB Prompt Prompt

Input Update LLM Update LLM
Knowledge Lookup Format
Knowledge

+Customer
Info

112
112
 Airline Chatbot

+Knowledge +Response
Base

Prompt DB Prompt Prompt

Input Update LLM Update LLM
Knowledge Lookup Format
Knowledge

+Customer
Info

History User

113
113
Building RAG Agents with
LLMs
Part 5: Working with Documents

114
 Modern Chain Paradigms
Towards Powerful Running State

Unstructured
Prompt LLM
Generation

Code Structured
Prompt Environment
LLM Retrieval
Input

Grammar Guided
Prompt LLM
/Schema Generation

Tool
Prompt LLM Choice Tool Tooling
Tool
Schema 115
115
 Document Reasoning

Sure, I can
Prompt answer by
Context LLM
Question referring this
blog post!

[Link]
116
116
 Document Reasoning
Company
Database

Sure, I can
Prompt answer by
Context LLM
Question referring this
blog post!

117
117
 Document Reasoning
Company
Database

Sure, I can
Prompt answer by
Context LLM
Question referring this
blog post!
Local
files
Your Device

118
118
 Document Reasoning
Company
Database

Sure, I can
Prompt answer by
Context LLM
Question referring this
blog post!
Local
files
Your Device

[Link]
119
119
 Document Reasoning
Company
Prompt
Database Context……
………………
………………
………………
………………
………………
………………
………………
………………
……………… I forgot the
………………
……………… instructions, but
……………… LLM
……………… I can still say
………………
……………… things
………………
………………
Local ………………
files ………………
………………
Your Device ………………
………………
………………
………
Question
[Link]
120
120
[Link]
 Chunking

[Link] 121
121
 Document Stuffing

Prompt
Oh yes, the intro
(DOCS 1-4)
LLM tells me
Use Them
Please
everything! You
Question
see…

[Link] 122
122
 Map Reduce Chain

Smaller
Chunk

Prompt Smaller
LLM
Useful Info Chunk

Smaller
Chunk

⠇
[Link] 123
123
 Refinement Chain

Top 10
Chunks

for doc in docs: Prompt LLM Summary

yield doc Useful Info

Main
Ideas

[Link] 124
124
 Knowledge Graph Construction
Abstraction
Main Ideas
Prompt
Useful LLM Abstractions
Constructs

for doc in docs: Prompt LLM Chapters

yield doc Chapter Logic

Per-Chapter
Main Ideas

Prompt
Character LLM Names
Information

Identity Key
Points
[Link] 125
125
 Knowledge Graph Traversal
Abstraction
Main Ideas
How does Flying Abstractions
work according to Prompt LLM
Find Info
your book?

Chapters

Per-Chapter
Main Ideas

Prompt LLM Sure! From the Names

Use Info
chapter on Birds,
I can tell you… Identity Key
Points
[Link] 126
126
 Refinement Chain
Your Assignment

Top 10
Chunks

for doc in docs: Prompt LLM Summary

yield doc Useful Info

Main
Ideas

[Link] 127
127
 Refinement Chain
Your Assignment

RunnableLambda

for doc in docs: Prompt LLM Summary

yield doc Useful Info

[Link] 128
128
Optional Tangent: LangGraph

[Link]
[Link]
129
129
 Refinement Chain
Your Assignment

RunnableLambda

for doc in docs: Prompt LLM Summary

yield doc Useful Info

[Link] 130
130
Building RAG Agents with
LLMs
Part 6: Embedding Model for Retrieval

131
 Knowledge Graph Traversal
Abstraction
Main Ideas
How does Flying Abstractions
work according to Prompt LLM
Find Info
your book?

Chapters

Per-Chapter
Main Ideas

Prompt LLM Sure! From the Names

Use Info
chapter on Birds,
I can tell you… Identity Key
Points
[Link] 132
132
 Modern Chain Paradigms
Towards Powerful Running State

Unstructured
Prompt LLM
Generation

Code Structured
Prompt Environment
LLM Retrieval
Input

Unstructured
Prompt LLM
Retrieval
???
Unstructured
Retrieval

133
133
 Modern Chain Paradigms
Towards Powerful Running State

Unstructured
Prompt LLM
Generation

Code Structured
Prompt Environment
LLM Retrieval
Input

Unstructured
Prompt LLM
Retrieval
Vector Database
Unstructured
Retrieval

134
134
 Transformer Architecture
Primary Backbone of LLMs

Element-Wise
Feed-Forward

Element-Wise
Feed-Forward
Sequence
Attention
Interface
Sequence
Attention
Interface

Element-Wise Element-Wise
Feed-Forward Feed-Forward
[Link]
136
136
Transformer Architecture
Autoregressing vs Embedding Flavors

137
137
Transformer Architecture
Autoregressing vs Embedding Flavors

138
138
Retrieval QA Embedding
Asymmetric Query/Document Model

139
[Link]
 Embedding and Comparing
Querying for Semantically Similar Entries

High-performance computing.
Happy Holidays!
DLSS Gaming Statistics
Any cool video games lately?
Biological vision structure
What’s with GPUs these days?
Mitochondria, powerhouse

[Link] 140
 Embedding and Comparing
Querying for Semantically Similar Entries

High-performance computing.
Happy Holidays!
DLSS Gaming Statistics
Any cool video games lately?
Biological vision structure
What’s with GPUs these days?
Mitochondria, powerhouse

[Link] 141
 Embedding and Comparing
Querying for Semantically Similar Entries

High-performance computing.
Happy Holidays!
DLSS Gaming Statistics
Any cool video games lately?
Biological vision structure
What’s with GPUs these days?
Mitochondria, powerhouse

[Link] 142
 Language Embedding Schemes
Bi-Encoder versus Cross-Encoder

Bi-Encoder Embedding Cross-Encoder Reranker

Cosine-Similarity

u v Classifier

Encoder Encoder Encoder

Passage 1 Passage 2 Passage 1 Passage 2

[Link] 144
 Language Embedding Schemes
Symmetric versus Asymmetric

Symmetric Asymmetric/
Generalized
Cosine-Similarity

u v

Encoder 1 Encoder 1

Passage 1 Passage 2

[Link] [Link] 145

Language Embedding Schemes
Generalized Definition

[Link] 146
 Embedding and Comparing
Querying for Semantically Similar Entries

High-performance computing.
Happy Holidays!
DLSS Gaming Statistics
Any cool video games lately?
Biological vision structure
What’s with GPUs these days?
Mitochondria, powerhouse

[Link] 147
Building RAG Agents with
LLMs
Part 6.4: Guardrails

148
 Embedding and Comparing
Querying for Semantically Similar Entries

High-performance computing.
Happy Holidays!
DLSS Gaming Statistics
Any cool video games lately?
Biological vision structure
What’s with GPUs these days?
Mitochondria, powerhouse

[Link] 149
 Semantic Guardrails

Illegal
Irrelevant Topics
Questions Prompt
Answer
Please
Question

How’s the Classifier No. I shouldn’t

weather? Branch LLM
answer that

Prompt
Don’t Answer
question

Tell me about
GPUs
What’s an
LLM Service What’s a good
game with RTX? 150
150
 Embedding Classification
Classifying with embeddings

Illegal
Irrelevant Topics
Questions

Tell me about
GPUs
What’s an
LLM Service What’s a good
game with RTX?
[Link]
gpu-powered-indexes-with-rapids-raft/ 151
 Embedding Classification
Classifying with embeddings

Illegal
Irrelevant Topics
Questions

Tell me about
GPUs
What’s an
LLM Service What’s a good
game with RTX?
[Link]
gpu-powered-indexes-with-rapids-raft/ 152
 Embedding Classification
Classifying with embeddings

Illegal Classification Head

Irrelevant Topics
Questions

0 or 1?

Tell me about
GPUs
What’s an
LLM Service What’s a good
game with RTX?
[Link]
gpu-powered-indexes-with-rapids-raft/ 153
 Semantic Guardrails

Illegal
Irrelevant Topics
Questions Prompt
Answer
Please
Question

How’s the Classifier No. I shouldn’t

weather? Branch LLM
answer that

Prompt
Don’t Answer
question

Tell me about
GPUs
What’s an
LLM Service What’s a good
game with RTX? 154
154
Building RAG Agents with
LLMs
Part 7: Document Retrieval with Vector Databases

155
 Embedding and Comparing
Querying for Semantically Similar Entries

High-performance computing.
Happy Holidays!
DLSS Gaming Statistics
Any cool video games lately?
Biological vision structure
What’s with GPUs these days?
Mitochondria, powerhouse

[Link] 156
 Retrieval-Augmented Generation
Pulling in Information from a Database

[Link] [Link] 157

Retrieval QA Embedding
Asymmetric Query/Document Model

158
158
[Link]
Integrating a Vector Store

159
159
Integrating a Vector Store

[Link]
160
160
Integrating a Vector Store

VDB

[Link]
161
161
Integrating a Vector Store

VDB
Retriever

[Link]
162
162
Integrating a Vector Store

VDB <metadata>
Retriever your name is NVBot…
<conversation>
Hey, my name is Jane
<wikipedia>
…a name is a term…
[Link]
163
163
 Retrieval Reordering/Selection

<metadata>
VDB your name is NVBot…
What’s <conversation>
Retriever Reranker
my name? Hey, my name is Jane
<wikipedia>
…a name is a term…

LongContextReorder

164
164
 Query Augmentation

Prompt
Rephrase as LLM
Question

VDB
What’s
Retriever
my name?

Prompt
Rephrase as LLM
Hypothesis

165
165
 RAG Fusion

Prompt
Rephrase as LLM
Question

VDB
What’s
Retriever Reranker
my name?

Prompt
Rephrase as LLM
Hypothesis

166
166
 Integrating a Vector Store

[Link] 167
167
 Integrating a Local Vector Store

Local Host
Milvus Standalone
Frontend
Your Device

Jupyter Notebook FAISS

[Link] 168
168
 Integrating a Local Vector Store

Local Host <s>[INST]<<SYS>>

{{system_message}}
Milvus <</SYS>>
Standalone
Frontend {{instruction}} [/INST]
{{primer}} Llama Mistral
Your Device

Jupyter Notebook FAISS { E5

Query Router “context”:{{context}}
“model”:“query”/”doc”
}

doc1
doc2
doc3 169
169
 GPU-Accelerating Vector Stores

[Link] [Link]
170
exes-with-rapids-raft/ e/faiss-a-library-for-efficient-similarity-search/
 Compute Scale Progression

Milvus Cluster

Milvus Standalone

Jupyter Notebook
Server FAISS
.

171
171
[Link]
 Simple Conversation RAG Setup
Hello! My
name is Jane
Hello Jane!
How are you? VDB
Retriever

Prompt
What’s My Name? History LLM It’s Jane!
Context
Question

Retriever
VDB

172
172
 Simple RAG Agents

Prompt
Question
How does RAG Classifier Sure! According
LLM
work? Branch to the paper…
Prompt
Context
Question

Retriever
VDB

173
173
 Proper RAG Agent
Tool-Selection Agents

TOOLSET
How does RAG
work?

Prompt
Context LLM
Question

LLM Data
Prompt Retriever
What Should I LLM
do?

Final Answer Prompt

Question LLM
+ History Answer

174
174
Building RAG Agents with
LLMs
Part 8: RAG Evaluation

175
 LLM-As-A-Judge
Pipeline Evaluation

RAG Pipeline

How does Prompt According to

Context LLM
RAG work? Question my resources…

Retriever

176
176
 LLM-As-A-Judge
Pipeline Evaluation

RAG Pipeline

How does Prompt According to

Context LLM
RAG work? Question my resources…

Retriever

Evaluation Pipeline
Prompt Prompt
Facilitate LLM LLM
Is This Good?
Testing

Functions Functions
177
177
 Our Evaluation Chain Components
Synthetic Generation

VDB
Q: How Does
Doc1 RAG Work?
Prompt LLM
Ask+Answer A: From these
Doc2 documents…

178
178
 Our Evaluation Chain Components
RAG Pipeline Sample

VDB
Q: How Does
Doc1 RAG Work?
Prompt LLM
Ask+Answer A: From these
Doc2 documents…

How does Prompt According to

Context LLM
RAG work? Question my resources…

Retriever

179
179
 Our Evaluation Chain Components
Ground Truth vs Our Pipeline

VDB
Q: How Does
Doc1 RAG Work?
Prompt LLM
Ask+Answer A: From these
Doc2 documents…
Prompt
Which Is LLM
Better?

How does Prompt According to

Context LLM
RAG work? Question my resources… [1] Bot 2 Better

Retriever

180
180
 Our Evaluation Chain Components
Ground Truth vs Our Pipeline

VDB
Q: How Does
Doc1 RAG Work?
Prompt LLM
Ask+Answer A: From these
Doc2 documents…
Prompt
Which Is LLM
Better?

How does Prompt According to

Context LLM
RAG work? Question my resources… [1] Bot 2 Better

Retriever
[0] [1] [0] [1] [1] 4/6
181
181
 General Evaluation Chain Components
Multiple Metrics at Once

VDB
+ Embedder
RagasEvaluatorChain

LLM

Prompt
{question} Context LLM
Question

Retriever

182
[Link]
182
 RAG Evaluation with
Quantifying System Goodness with LLM-as-a-Judge

Control
Sources
Input RagasEvaluatorChain

[Link]
183
 RAG Evaluation with
Quantifying System Goodness with LLM-as-a-Judge

[Link] [Link] 184

 Evaluator Agent
Pipeline Evaluation

Prompt
Ask some LLM
questions

Chain to
Evaluate

Prompt TOOLSET
What Should I LLM
do?

Final Answer Prompt LLM

+ History Is This Good?

185
185
 Our Evaluation Chain Components
Ground Truth vs Our Pipeline

VDB
Q: How Does
Doc1 RAG Work?
Prompt LLM
Ask+Answer A: From these
Doc2 documents…
Prompt
Which Is LLM
Better?

How does Prompt According to

Context LLM
RAG work? Question my resources… [1] Bot 2 Better

Retriever
[0] [1] [0] [1] [1] 4/6
186
186
 Evaluate RAG In The Frontend
Final Assessment

Remote Host

Frontend :8090

Your Device
RAG

Jupyter Notebook

NVIDIA
GPU CLOUD

187
187
Congratulations!
AND THANK YOU SO MUCH!

188