Scribd
Upload
9 views3 pages

Streaming Intermediate Steps in RAG

Uploaded by

fahim.nsudrive3
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
9 views3 pages

Streaming Intermediate Steps in RAG

Uploaded by

fahim.nsudrive3
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd

Streaming intermediate steps

Suppose we want to stream not only the final outputs of the chain, but also some
intermediate steps. As an example let's take our Conversational RAG chain. Here we
reformulate the user question before passing it to the retriever. This reformulated
question is not returned as part of the final output. We could modify our chain to
return the new question, but for demonstration purposes we'll leave it as is.

from [Link] import create_history_aware_retriever


from langchain_core.prompts import MessagesPlaceholder

### Contextualize question ###


contextualize_q_system_prompt = (
"Given a chat history and the latest user question "
"which might reference context in the chat history, "
"formulate a standalone question which can be understood "
"without the chat history. Do NOT answer the question, "
"just reformulate it if needed and otherwise return it as is."
)
contextualize_q_prompt = ChatPromptTemplate.from_messages(
[
("system", contextualize_q_system_prompt),
MessagesPlaceholder("chat_history"),
("human", "{input}"),
]
)
contextualize_q_llm = llm.with_config(tags=["contextualize_q_llm"])
history_aware_retriever = create_history_aware_retriever(
contextualize_q_llm, retriever, contextualize_q_prompt
)

### Answer question ###


system_prompt = (
"You are an assistant for question-answering tasks. "
"Use the following pieces of retrieved context to answer "
"the question. If you don't know the answer, say that you "
"don't know. Use three sentences maximum and keep the "
"answer concise."
"\n\n"
"{context}"
)
qa_prompt = ChatPromptTemplate.from_messages(
[
("system", system_prompt),
MessagesPlaceholder("chat_history"),
("human", "{input}"),
]
)
question_answer_chain = create_stuff_documents_chain(llm, qa_prompt)

rag_chain = create_retrieval_chain(history_aware_retriever, question_answer_chain)

API Reference:create_history_aware_retriever | MessagesPlaceholder


Note that above we use .with_config to assign a tag to the LLM that is used for the
question re-phrasing step. This is not necessary but will make it more convenient
to stream output from that specific step.

To demonstrate, we will pass in an artificial message history:


Human: What is task decomposition?

AI: Task decomposition involves breaking up a complex task into smaller and simpler
steps.

We then ask a follow up question: "What are some common ways of doing it?" Leading
into the retrieval step, our history_aware_retriever will rephrase this question
using the conversation's context to ensure that the retrieval is meaningful.

To stream intermediate output, we recommend use of the async .astream_events


method. This method will stream output from all "events" in the chain, and can be
quite verbose. We can filter using tags, event types, and other criteria, as we do
here.

Below we show a typical .astream_events loop, where we pass in the chain input and
emit desired results. See the API reference and streaming guide for more detail.

first_question = "What is task decomposition?"


first_answer = (
"Task decomposition involves breaking up "
"a complex task into smaller and simpler "
"steps."
)
follow_up_question = "What are some common ways of doing it?"

chat_history = [
("human", first_question),
("ai", first_answer),
]

async for event in rag_chain.astream_events(


{
"input": follow_up_question,
"chat_history": chat_history,
},
version="v1",
):
if (
event["event"] == "on_chat_model_stream"
and "contextualize_q_llm" in event["tags"]
):
ai_message_chunk = event["data"]["chunk"]
print(f"{ai_message_chunk.content}|", end="")

|What| are| some| typical| methods| used| for| task| decomposition|?||

Here we recover, token-by-token, the query that is passed into the retriever given
our question "What are some common ways of doing it?"

If we wanted to get our retrieved docs, we could filter on name "Retriever":

async for event in rag_chain.astream_events(


{
"input": follow_up_question,
"chat_history": chat_history,
},
version="v1",
):
if event["name"] == "Retriever":
print(event)
print()

Common questions

Powered by AI

The chain process ensures accuracy by using a history-aware retriever to reformulate queries based on previous contexts, thus refining the retrieval process. The retrieval is context-driven, which increases the relevance of the collected information, resulting in concise and accurate answers .

Without streaming intermediate outputs, debugging and fine-tuning the retrieval system becomes cumbersome. It becomes challenging to track how questions are reformulated or identify where errors might be occurring, leading to inefficiencies and reduced accuracy in the final outcomes of the system .

Task decomposition, which involves breaking down complex queries into simpler, actionable parts, parallels the retrieval chain's reformulation process. By simplifying and clarifying the inputs at each stage, both processes enhance comprehension and retrieval accuracy, ultimately streamlining complex information into manageable, useful outputs .

In the retrieval chain process, context is fundamental, as it standards the question reformulation process, ensuring the query is meaningful and accurate for retrieval. The context aids in formulating questions that retrieve the most relevant information, leading to a higher success rate in answering queries satisfactorily .

Tagging LLM configurations, such as with 'contextualize_q_llm', facilitates focused streaming of output from specific reformulation steps. This targeted approach aids in differentiating outputs from different stages, making analysis and debugging more manageable and streamlined .

The async .astream_events method is recommended because it streams output from all events in the chain, enabling real-time analysis of both the intermediate and final outputs. This comprehensive streaming is crucial for monitoring and filtering specific outputs, making the process more efficient and informative for troubleshooting and optimization .

Using a chat history placeholder informs the reformulation process by providing necessary previous context, which helps in transforming questions into self-sufficient forms. This ensures that the questions, when input into the retrieval system, are optimized for obtaining the most relevant document matches .

Streaming intermediate steps allows for the observation of all parts of the retrieval chain process, such as question reformulation, which aids in debugging and understanding the chain's function. This visibility ensures that the query passed to the retriever is appropriately contextualized using prior chat history, leading to more meaningful and accurate retrieval results .

The contextualize_q_system_prompt serves to transform user questions into standalone queries understandable without prior chat context. This reformulation ensures that when the question is passed to the retriever, it can be processed effectively, improving the accuracy of the retrieval results .

The 'contextualize_q_llm' tag is crucial for isolating and analyzing the reformulation step, ensuring the retrieval system recognizes this specific processing stage. By tagging, developers can filter and assess the strengths or weaknesses of tight integration between reformulation outputs and retrieval inputs, directly correlating to improved retrieval effectiveness and system refinement .

You might also like