Intro of Retrieval Augmented
Generation (RAG) and application
demos
Henry Heng LUO Follow 9 min read · Jan 27, 2024
316
RAG Background
Open in app Sign up Sign in
T oday I will provide an introduction to Retrieval Augmented Generation
(RAG) and demonstrate
Search some applications. You can access theWrite
materials for this talk on my GitHub repository at below.
GitHub - HenryHengLUO/Retrieval-Augmented-Generation-
Intro-Project: This project aims to introduce…
This project aims to introduce and demonstrate the practical
applications of RAG using Python code in a Jupyter…
[Link]
In the repository, you will find a PDF with the slides for my presentation.
Later, we will do some hands-on experiments to apply RAG. All the code and
data is available in the GitHub repo for you to follow along.
Let’s start with an overview of RAG. RAG is a powerful technique for
enhancing large language models. In my view, we should focus on how to
best apply large language models, and RAG is one of the most effective
approaches, especially for developers.
Large language models have some intrinsic limitations. They can provide
misleading or hallucinated information since they lack external knowledge.
They rely on potentially outdated information since their training data has a
cutoff date. For example, GPT-3 was trained before 2021. They lack depth and
specificity on niche topics outside their training data. Training and fine-
tuning LLMs is computationally expensive and infeasible for many
organizations. The models cannot show where their knowledge comes from
or ensure privacy compliance when sensitive data is provided.
RAG can significantly improve the accuracy and relevance of generated
content. It first retrieves relevant information from an external database or
documents before generating text. [1]
Here’s an example: imagine a user asks “How do you evaluate the fact that
OpenAI’s CEO Sam Altman went through a sudden dismissal by the board in
just three days, and was rehired by the company, resembling a real life
version of Game of Thrones in terms of power dynamics?”
ChatGPT would not be able to answer this properly since the event happened
after 2021. With RAG, we would first retrieve relevant documents, extract key
snippets like “Sam Altman returns to OpenAI as CEO, silicon valley drama
resembles a comedy”, “the drama concludes? Sam Altman to return as CEO
of OpenAI, board to undergo restructuring”, “the personnel turmoil at
OpenAI comes to an end. who won and who lost?” These three snippets
would be combined into the prompt to provide context for the question. The
large language model can then generate a coherent answer based on the
retrieved information.
RAG Timeline and Techs
L ooking at the history, RAG originated in academia with three main
approaches: pre-train, fine-tune, and inference retrieval. Recently,
more practical techniques focus on inference-time retrieval. Moreover, as
before 2022, there were few RAG techniques proposed. However, after 2023,
we can see the booming of various RAG techniques.
RAG improves the precision and relevance of LLM outputs by first retrieving
relevant information from an external knowledge source before generating a
response. The classic basic RAG process, also known as Naive RAG, mainly
includes three basic steps.
1. Indexing: Documents are split into shorter texts (“chunks”) and indexed
in a vector database using an encoder model.
2. Retrieval: Relevant chunks are found based on similarity between the
question and chunks.
3. Generation: The LLM generates an answer conditioned on the retrieved
context.
The Advanced RAG paradigm involves additional processing in Pre-Retrieval
and Post-Retrieval.
1. Before retrieval, methods such as query rewriting, routing, and
expansion can be used to align the semantic differences between
questions and document chunks.
2. After retrieval, rerank the retrieved document corpus can avoid the “Lost
in the Middle” phenomenon, or the context can be filtered and
compressed to shorten the window length.
The Modular RAG is also introduced. Structurally, it is more free and
flexible, introducing more specific functional modules, such as query search
engines and the fusion of multiple answers. Technologically, it integrates
retrieval with fine-tuning, reinforcement learning, and other techniques. In
terms of process, the RAG modules are designed and orchestrated, resulting
in various RAG patterns.
To build a good RAG system, three critical questions need to be considered:
What to retrieve? When to retrieve? How to use the retrieved content?
Augmentation Sources. Unstructured data such as text paragraphs, phrases,
or individual words. Structured data can also be used, such as indexed
documents, triple data, or subgraphs, or retrieving from content generated
by LLMs themselves.
Augmentation Stages. performing during the pre-training, fine-tuning, and
inference stages.
Augmentation process. The initial retrieval was a once process, but iterative
retrieval, recursive retrieval, and adaptive retrieval methods, where LLMs
decide the timing of retrieval on their own, gradually emerged in the
development of RAG.
Below figure shows more detailed information about RAG triage, including
augmentation stage (pre-training, fine-tuning, inference), augmentation
source (unstructured data, structured data, LLM generated content),
augmentation process (once retrieval, iterative retrieval, adaptive retrieval,
recursive retrieval).
Below figure shows RAG relevant terminology and their reference papers.
RAG features
To better know the RAG is through comparison. RAG is like giving the
model a textbook for customized information retrieval, which is
very suitable for specific queries. Let me give an analog to explain. RAG
gives models an external knowledge source, like giving a student a textbook
for an open-book exam. Nevertheless, fine-tuning is like a student gradually
acquiring knowledge suited for specific tasks, who is internalizing
knowledge over time, better suited for mimicking specific structures, styles,
or formats.
Based on the need for external knowledge and model customization, RAG
and fine-tuning each have appropriate applications. Using them together can
achieve the best performance. RAG requires little model adaptation but
extensive external knowledge, while fine-tuning adapts models significantly
but requires less external data. For most cases, RAG, Fine-tuning, Prompt
Engineering combine together can yield the best results.
RAG evaluation
A fter implementing RAG, thorough evaluation is critical, using three
quality scores, like context relevance, answer fidelity and answer
relevance. The evaluation involves four key capabilities, robustness to noise,
refusal ablity, integration of information, and counterfactual analysis.
Standardized benchmarks like RGB and RECALL, as well as automated
evaluation tools like RAGAS, ARES, and TruLens, are available to evaluate
RAG systems.
RAG future
W hile powerful, RAG faces some challenges. With large context
windows, performance may not improve. Making retrieval robust
and filtering low-quality content is difficult. Incorrect content retrieved may
poison the final answer. Balancing RAG and fine-tuning can be tricky. It’s
unclear if larger models always improve RAG. The role of LLM need to
further explored. Productionizing RAG at scale and protecting sensitive data
are other concerns. Expanding RAG to handle images, audio and video
remains an open problem.
Get Henry Heng LUO’s stories in your inbox
Join Medium for free to get updates from this writer.
Enter your email Subscribe
Remember me for faster sign in
But RAG shows promise for question answering, recommendation systems,
information extraction and report generation. Mature RAG technology
stacks are booming, such as Langchain and LlamaIndex, the market is also
seeing an emergence of more targeted RAG tools, such as customized tools
and simplified tools. Hence, the ecosystem will continue expanding with
new tools tailored for RAG.
RAG practice
So far I’ve provided a high-level overview of RAG. Next, I will
demonstrate some hands-on RAG experiments so you can apply
these techniques in your own projects. I have three Python scripts to
showcase different RAG pipelines by utilising LlamaIndex.
1. Basic RAG pipeline
2. Sentence-window RAG pipeline
3. Auto-emerging RAG pipeline
The basic RAG pipeline augments a large language model with an existing
database. Queries first retrieve relevant context from the database before
generating an answer.
We can chunk documents into smaller passages of 64 tokens with 2 token
overlap. The passages are encoded into vectors and indexed in a vector
database. Given a query, we search for the most similar passages, wrap them
into a prompt with the question, and send this to the language model to
generate an answer.
The Sentence-window retrieval pipeline is useful when we need more
context. Instead of token chunks, we segment documents into sentences. We
retrieve the most similar sentence as well as the previous and next sentences
to form a context window. The context windows are re-ranked and provided
to the language model.
The auto-emerging retrieval pipeline creates a hierarchy for retrieval. Small
16 token passages are linked to form 64 token passages, which are in turn
linked to 256 token passages. If enough small passages link to a parent, they
are merged into the parent chunk. The final chunks are reranked and
retrieved. This allows dynamically sized context.
GitHub - HenryHengLUO/Retrieval-Augmented-Generation-
Intro-Project: This project aims to introduce…
This project aims to introduce and demonstrate the practical
applications of RAG using Python code in a Jupyter…
[Link]
To use the code, first create a Python virtual environment from the yaml file
provided in [Link] file under python_env folder. Add your OpenAI API key
in [Link] file under common folder. The sample data is in [Link]
under data folder, but you can provide your own documents.
The basic pipeline chunks the document, indexes it in a vector database,
takes a query, retrieves similar passages, wraps them into a prompt and
sends it to the language model. We can see the original source of the
retrieved passages.
The sentence-window pipeline does retrieval at the sentence level,
expanding sentences with previous two sentences and next one sentence.
Re-ranking is demonstrated to select the most relevant window.
The auto-emerging pipeline builds a hierarchy of passages from 16 to 256
tokens, merging passages into larger chunks as needed. It provides longer
context while maintaining precision.
The code is designed to be plug-and-play. You can take it and apply RAG to
your own documents and use cases after configuring the API key and virtual
environment. Please try it out and let me know if you have any other
questions!
References
[1] Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J. and Wang,
H., 2023. Retrieval-augmented generation for large language models: A
survey. arXiv preprint arXiv:2312.10997.
[2] [Link]
Intro-Project
[3] [Link]
[4] [Link]
Appendix
This article was modified and rewritten according to Henry’s presentation at
the GDG Hong Kong AI/ML Study Group on January 27, 2024. In conclusion,
all attendees memorized a funny joke from the demo: "Henry is the most
pretty boy in Hong Kong."
I appreciate Thomas and Kin for organizing the wonderful event.
NLP Rags LLM Llamaindex Jupyter Notebook
Written by Henry Heng LUO Follow
439 followers · 2 following
A highly self-motivated and enthusiastic data scientist.
No responses yet
Write a response
What are your thoughts?