RAG Code Explanation
Step 1: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.
!pip install -q langchain pypdf2 faiss-cpu langchain-community streamlit pyngrok
!pip install -q streamlit
!npm install -g localtunnel
Step 2: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.
from PyPDF2 import PdfReader
from typing_extensions import Concatenate
from langchain.text_splitter import CharacterTextSplitter
from langchain_community.vectorstores import FAISS
from [Link] import HuggingFaceEmbeddings
from [Link].question_answering import load_qa_chain
from langchain_community.llms import HuggingFaceHub
import os
Step 3: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.
[Link]["HUGGINGFACEHUB_API_TOKEN"] ='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'
pdfreader = PdfReader(r"/content/IoT based Smart [Link]")
Step 4: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.
raw_text = ''
RAG Code Explanation
for i, page in enumerate([Link]):
content = page.extract_text()
if content:
raw_text += content
Step 5: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.
text_splitter = CharacterTextSplitter(
separator = "\n",
chunk_size = 800,
chunk_overlap = 200,
length_function = len,
)
texts = text_splitter.split_text(raw_text)
Step 6: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
document_search = FAISS.from_texts(texts, embeddings)
Step 7: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.
# Generate embeddings
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
# Generate vectorstore
document_search = FAISS.from_texts(texts, embeddings)
#print embeddings for the first few chunks
for i, text in enumerate(texts[:3]):
vector = embeddings.embed_query(text)
# print(f"\nText Chunk {i+1}:\n{text[:200]}...")
RAG Code Explanation
print(f"Embedding Vector ({len(vector)} dims):\n{vector}")
Step 8: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.
chain = load_qa_chain(HuggingFaceHub(repo_id='mistralai/Mixtral-8x7B-Instruct-v0.1'),
chain_type="stuff")
def get_Chat_response(chat):
query = chat
docs = document_search.similarity_search(query)
a=[Link](input_documents=docs, question=query)
output= (a[[Link]('Helpful Answer:')+16:])
return output
Step 9: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.
get_Chat_response("what is the document about?")
Step 10: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.
%%writefile [Link]
from PyPDF2 import PdfReader
from langchain.text_splitter import CharacterTextSplitter
from langchain_community.vectorstores import FAISS
from [Link] import HuggingFaceEmbeddings
from [Link].question_answering import load_qa_chain # Correct import
from langchain_community.llms import HuggingFaceHub
import os
import streamlit as st
[Link]["HUGGINGFACEHUB_API_TOKEN"] = "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"
RAG Code Explanation
# Load and preprocess PDF once at startup
pdf = PdfReader("/content/IoT based Smart [Link]")
raw_text = "".join([p.extract_text() or "" for p in [Link]])
splitter = CharacterTextSplitter(separator="\n", chunk_size=800, chunk_overlap=200)
texts = splitter.split_text(raw_text)
emb = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
doc_search = FAISS.from_texts(texts, emb)
llm = HuggingFaceHub(
repo_id="mistralai/Mixtral-8x7B-Instruct-v0.1",
model_kwargs={"temperature":0.5, "max_new_tokens":512}
)
chain = load_qa_chain(llm, chain_type="stuff")
def get_Chat_response(q):
docs = doc_search.similarity_search(q)
ans = [Link](input_documents=docs, question=q)
return [Link]("Helpful Answer:")[-1].strip() if "Helpful Answer:" in ans else [Link]()
[Link](" Chat with your PDF using RAG")
q = st.text_input("Ask a question about the document:")
if q:
[Link]("### Answer:")
[Link](get_Chat_response(q))
Step 11: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.
!streamlit run [Link] \
--[Link] [Link] \
--[Link] 8501 \
--[Link] false \
&>/content/[Link] & npx localtunnel --port 8501 & curl [Link]
Step 12: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.
%%writefile [Link]
import streamlit as st
from PyPDF2 import PdfReader
RAG Code Explanation
from langchain.text_splitter import CharacterTextSplitter
from langchain_community.vectorstores import FAISS
from [Link] import HuggingFaceEmbeddings
from [Link].question_answering import load_qa_chain
from langchain_community.llms import HuggingFaceHub
import os
[Link]["HUGGINGFACEHUB_API_TOKEN"] = "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"
[Link](" Chat with your PDF using RAG")
# 1) Upload PDF
uploaded_file = st.file_uploader("Upload a PDF", type="pdf")
if not uploaded_file:
[Link]("Please upload a PDF file to continue.")
[Link]()
# 2) Extract & split
pdf = PdfReader(uploaded_file)
raw_text = "".join([p.extract_text() or "" for p in [Link]])
splitter = CharacterTextSplitter(separator="\n", chunk_size=800, chunk_overlap=200)
texts = splitter.split_text(raw_text)
# 3) Build vector store
emb = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
doc_search = FAISS.from_texts(texts, emb)
# 4) Load QA chain
llm = HuggingFaceHub(
repo_id="mistralai/Mixtral-8x7B-Instruct-v0.1",
model_kwargs={"temperature":0.5, "max_new_tokens":512}
)
chain = load_qa_chain(llm, chain_type="stuff")
# 5) Ask questions
query = st.text_input("Ask a question about the document:")
if query:
docs = doc_search.similarity_search(query)
ans = [Link](input_documents=docs, question=query)
answer = [Link]("Helpful Answer:")[-1].strip() if "Helpful Answer:" in ans else [Link]()
[Link]("### Answer:")
[Link](answer)
Step 13: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.
RAG Code Explanation
!streamlit run [Link] \
--[Link] [Link] \
--[Link] 8501 \
--[Link] false \
&>/content/[Link] & npx localtunnel --port 8501 & curl [Link]
Step 14: Code Cell
This code cell performs a specific step in the RAG workflow.
It contributes to either loading data, processing it, or generating output.