Plan∗ RAG: Efficient Test-Time Planning for Retrieval Augmented Generation

Prakhar Verma † 1 Sukruta Prakash Midigeshi 2 Gaurav Sinha 2 Arno Solin 1

Nagarajan Natarajan 2 Amit Sharma 2

Abstract Test-time planning improves RAG

We introduce Plan∗ RAG, a novel framework 45 Vanilla With Plan∗
arXiv:2410.20753v2 [[Link]] 4 Feb 2025

that enables structured multi-hop reasoning in 40.44

Accuracy (%)
40 37.31
retrieval-augmented generation (RAG) through 34.09
test-time reasoning plan generation. While exist- 35 33.15
31.12
ing approaches such as ReAct maintain reason- 30
25.51
ing chains within the language model’s context 25
window, we observe that this often leads to plan
fragmentation and execution failures. Our key 20
insight is that by isolating the reasoning plan as
a directed acyclic graph (DAG) outside the LM’s RAG Self-RAG ReAct
working memory, we can enable (1) systematic
Figure 1. Plan∗ RAG improves performance on the HotpotQA
exploration of reasoning paths, (2) atomic sub-
benchmark substantially compared to various existing RAG meth-
queries enabling precise retrievals and ground- ods, demonstrating the value of externalizing planning as a directed
ing, and (3) efficiency through parallel execution acyclic graph (DAG) outside of the LLM’s context.
and bounded context window utilization. More-
over, Plan∗ RAG’s modular design allows it to mental barrier to reliable AI systems, particularly given the
be integrated with existing RAG methods, thus widespread deployment of large language models (Brown
providing a practical solution to improve current et al., 2020) across critical domains. Consider the query:
RAG systems. On standard multi-hop reasoning “Rumble Fish was a novel by the author of the coming-of-age
benchmarks, Plan∗ RAG consistently achieves im- novel published in what year by Viking Press?” Answering
provements over recently proposed methods such this requires an iterative retrieval process: identifying the
as RQ-RAG and Self-RAG, while maintaining Rumble Fish’s author, connecting to their coming-of-age
comparable computational costs. novel, and determining its publication year. Single-step
retrieval in RAG systems often fails in such cases, as it
may retrieve documents about Rumble Fish’s author and
1. Introduction Viking Press without recognizing the intermediate fact—
the author’s coming-of-age novel—must first be established.
Retrieval-Augmented Generation (RAG, Lewis et al., 2020;
Furthermore, Leng et al. (2024); Shuster et al. (2021) demon-
Petroni et al., 2020; Guu et al., 2020) has emerged as a
strate that even when relevant documents are retrieved, LMs
promising approach for grounding language model (LM)
struggle to reason across them due to fixed context win-
responses in external knowledge. However, RAG systems
dows, leading to information loss and broken reasoning
struggle with multi-hop queries that require reasoning across
chain. These limitations pose risks in critical domains such
multiple retrieved documents (Tang & Yang, 2024; Wei
as healthcare and finance (Pal et al., 2023; Zhao et al., 2024),
et al., 2022). A key challenge lies in the initial retrieval step,
where accurate multi-step reasoning is essential.
which often fails to retrieve sufficient relevant documents
due to the query’s lack of full contextual information (Ma Recent research has attempted to address these limita-
et al., 2023). This limitation has been highlighted in recent tions through structured reasoning frameworks. Chain-of-
surveys (Torfi et al., 2020; Zhao et al., 2023) as a funda- Thought (CoT) prompting (Wei et al., 2022) and systematic
†
query decomposition (Patel et al., 2022) have introduced
Work done during an internship with Microsoft Research. explicit reasoning steps, enabling more granular thought
1
Aalto University, Finland 2 Microsoft Research. Correspondence
to: <[Link]@[Link]>, <amshar@[Link]>. processes and targeted retrievals. Building upon these foun-
dations, Yao et al. (2023) proposed ReAct—a framework
that creates a reasoning chain inside the LM’s context and

1
 Efficient Test-Time Planning for Retrieval Augmented Generation

Q: Rumble Fish was a novel by the author Answer: 1967

Vanilla RAG
of the coming-of-age novel published in
fLLM : Q × D → G (✗)
what year by Viking Press?
Chain-of-Thought L
fLLM : Q × D × n i=1 ti → G
(✗)
Q1.1 Who is the author
Existing RAG Methods: Reason & React
of Rumble Fish?
✗ Implicit Reasoning S (ReAct) (✗)
fLLM : Q × ni=1 (ti ×ai ×oi ×Di ) → G
✗ Context Overflow Query Decomposition
✗ Sequential Dependency n
Q2.1 What is the coming- fLLM : Q → {qi }S
i=1 (✗)
of-age novel by ⟨A1.1⟩? fLLM : Q × D × n i=1 (qi , Gi ) → G
Plan∗ RAG:
✓ Test-time Planning Self-RAG
✓ Conditional Independence fLLM : Q × D → {(Gi , Si )}K i=0 (✗)
Q3.1 In what year ✓ Dynamic Adaptation, ⟨AI.J⟩ G = arg maxGi Si
was ⟨A2.1⟩ published Plan∗ RAG
by Viking Press? fLLM : Q → G(V, E) (✓)
fLLM : G(Pa(q))×q×D → G(q) ∀q ∈ V
Figure 2. Comparison of RAG approaches: Comparison of different RAG approaches on a HotpotQA multi-hop query. Traditional
RAG methods (right) struggle with context management and implicit reasoning, where sequential operators create implicit dependencies.
In contrast, Plan∗ RAG (left) generates an explicit reasoning plan as a DAG (G) at test-time, with special tags ⟨AI,J⟩ enabling dynamic
information flow through parent subqueries (Pa(q)). On this example query, while previous approaches fail to identify the correct
publication year, Plan∗ RAG successfully decomposes the reasoning process and arrives at the correct answer (1967).

dynamically coordinates between thoughts, retrieval opera- intended plan). This prevents context overflow and ensures
tions (actions), and information processing (observations). that only relevant information is processed at each stage.
However, a fundamental limitation of ReAct is that the rea- Second, nodes at the same depth in the DAG can be pro-
soning and retrieval plan exists entirely within the LLM’s cessed in parallel, significantly reducing latency compared
context window, leading to context overflow as reasoning to sequential approaches. Third, nodes are dynamic, con-
chains grow, plan fragmentation, and high latency from se- taining tags that allow them to adapt based on their parent’s
quential execution. While recently proposed frameworks generation, resulting in more targeted sub-queries. Fur-
like Self-RAG (Asai et al., 2023), RA-ISF (Liu et al., 2024), thermore, the DAG structure enables explicit verification at
and RQ-RAG (Chan et al., 2024) have advanced the field each node, facilitating error correction and backtracking. As
through innovations in dynamic reasoning and adaptive re- generation involves traversing through the reasoning DAG,
trieval mechanisms, the core problem still remains. These Plan∗ RAG can be incorporated with both traditional RAG
methods often struggle to balance reasoning depth and com- frameworks and recent approaches like Self-RAG (Asai
putational efficiency (see Sec. 3 for an example). This et al., 2023), enhancing their multi-hop reasoning capabili-
trade-off results in either shallow analysis that misses criti- ties (Fig. 1) while maintaining comparable computational
cal reasoning steps, or impractical latency that limits their efficiency in terms of input-output tokens (cf. Fig. 4).
deployment in real-time applications.
Contributions We summarize our contributions as follows.
To address these challenges, we propose Plan∗ RAG, a novel (i) We introduce a test-time reasoning approach that external-
approach that externalizes the reasoning process as a di- izes the reasoning process as a DAG, fundamentally altering
rected acyclic graph (DAG). Unlike existing methods that how multi-hop queries are processed in RAG systems.
maintain reasoning chains within the LM’s context or gener- (ii) We propose a novel DAG-based reasoning structure
ate isolated sequential sub-queries, Plan∗ RAG decomposes that decomposes complex queries into atomic and dynamic
the main query into interrelated atomic and dynamic sub- sub-queries, enabling parallel processing while maintaining
queries, where nodes represent atomic reasoning steps and conditional dependencies. (iii) We demonstrate that our
edges capture the conditional dependencies between them. approach can enhance both traditional RAG frameworks and
Atomic sub-queries can be answered by a single retrieval, modern approaches like Self-RAG, improving multi-hop
and further decomposition does not provide additional use- reasoning capabilities with comparable computation.
ful granularity. Additionally, sub-queries are tagged with
special tags ⟨AI.J⟩, allowing them to adapt based on updated 2. Related Work
knowledge during generation. This external representation
addresses core limitations of existing approaches in several We review two key areas relevant to our work: (1) reasoning
ways. First, by following the DAG structure for generation, in LLMs, which focuses on methods that enhance multi-step
only the parent nodes need to be included in the context reasoning, and (2) retrieval-augmented generation (RAG),
window for any given step (and that the system follows the which integrates external information to improve LLM per-

2
 Efficient Test-Time Planning for Retrieval Augmented Generation

formance in knowledge-intensive tasks. complex queries. Hsu et al. (2024) also focus on multi-hop
query performance of RAG but take an orthogonal approach.
Reasoning in LLMs Recent advancements in LLMs have
significantly enhanced their reasoning capabilities through Recent work by Ranaldi et al. (2024), explores reasoning in
various approaches. Chain-of-Thought (CoT) prompting RAGs through C-RAG which generates explanations explic-
(Wei et al., 2022) improves model performance by guid- itly contrasting the relevance of retrieved passages to support
ing LLMs through intermediate reasoning steps. Build- the final answer. Plan∗ RAG takes a different approach by
ing on this, the Tree of Thoughts (ToT) framework (Yao proposing a novel framework that performs reasoning at
et al., 2024) enables LLMs to explore and evaluate multiple test-time outside the LM’s working memory, and performs
reasoning paths simultaneously. Wang et al. (2023) pro- generation in a computationally efficient manner, enabling
posed self-consistency, a technique that samples multiple LLMs to tackle complex multi-hop queries with greater
reasoning chains and selects the most likely answer through accuracy while producing a verifiable reasoning trace.
majority voting. Similarly, least-to-most prompting (Zhou
et al., 2023) decomposes complex questions into simpler 3. Motivation
subquestions, addressing them sequentially. Recently, Hao
et al. (2023) propose Reasoning via Planning (RAP), where Multi-hop reasoning queries, requiring information from
LLMs act as both world models and reasoning agents. Addi- multiple documents, pose significant challenges for cur-
tionally, Sun et al. (2024) introduced Think-on-Graph (ToG), rent Retrieval-Augmented Generation (RAG) systems. Con-
incorporating knowledge graphs into multi-hop reasoning sider an example query from HotPotQA (Yang et al., 2018):
for deeper and more interpretable reasoning processes. Re- “Rumble Fish was a novel by the author of the coming-of-age
cent works from Welleck et al. (2024); Chen et al. (2024) novel published in what year by Viking Press?”. The query
also analyzed tree search for reasoning. While these meth- demands multiple reasoning steps: identifying the Rumble
ods have demonstrated impressive results, they still rely on Fish’s author, connecting to their coming-of-age novel, and
implicit LM reasoning capability or on sequential reason- determining its publication year. In this section, we will
ing steps, which can lead to increased context usage and use this query as a running example a) to articulate the fail-
computational costs. In contrast, Plan∗ RAG externalizes the ure modes of standard as well as recently-proposed RAG
reasoning process as a DAG, mitigating these inefficiencies. approaches on such multi-hop reasoning queries (Fig. 2);
and b) motivate the core idea of our proposed solution. For
Retrieval Augmented Generation RAG enhances large
this analysis, we fix the model to be Llama-3.1-instruct8B
language models (LLMs) by integrating relevant external
and the retriever to be Contriever (Izacard et al., 2022). In
documents, leading to notable performance improvements,
the remainder of the paper, Q denotes queries, D denotes
particularly in knowledge-intensive tasks (Lewis et al., 2020;
documents, and G denotes generations (responses).
Guu et al., 2020). Retrieval strategies in RAG models
can be categorized into three paradigms based on the fre- Standard Baselines A straight-forward RAG approach
quency of retrievals: (1) one-time retrieval, (2) retrieval is to issue the given query Q to a retriever, retrieve docu-
every k tokens, and (3) adaptive retrieval. Models employ- ments D, and prompt a language model (LM) with Q and
ing one-time retrieval include DrQA (Chen et al., 2017), D to generate an answer. For the running example, this ap-
REALM (Guu et al., 2020), and ATLAS (Izacard et al., proach produces an incorrect answer (1975), because a) the
2023). Retrieval at fixed intervals (every k tokens) is used by retrievals are incomplete; it fails to get the ground-truth cor-
RALM (Ram et al., 2023), RETRO (Borgeaud et al., 2022), rect document on the coming-of-age novel The Outsiders,
and InstructRetro (Wang et al., 2024a). In contrast, adap- which is not surprising as the query makes only an indi-
tive retrieval approaches—such as Self-RAG (Asai et al., rect reference to the novel, and b) in turn, LM generates an
2023), SPALM (Yogatama et al., 2021), Adaptive kNN incorrect answer without any reasoning.
(Drozdov et al., 2022), and Active-Retriever (Jiang et al.,
This motivates using better reasoning at inference time. A
2023)—dynamically adjust the frequency and nature of doc-
natural idea is to prompt the same LM with Q and D, but
ument retrieval based on task requirements and input context.
also elicit a reasoning for the answer in a chain-of-thought
FLARE (Jiang et al., 2023) uses token probability distribu-
(CoT) manner (Wei et al., 2022). However, for our example,
tions to trigger retrievals, while RETRO (Borgeaud et al.,
RAG with CoT generates incorrect reasoning: [S.E. Hinton
2022) employs a specialized architecture for fixed-interval
wrote Rumble Fish → Rumble Fish was published in 1975
document retrieval. Wang et al. (2024b) proposed RAFT,
→ The author of The Outsiders also published in 1975],
combining chain-of-thought reasoning with RAG through
leading to the incorrect answer 1975. Here, we see the
iterative thought refinement. Other approaches include RA-
reason for failure is that the model hallucinated (third step of
ISF (Liu et al., 2024), which combines dataset-specific spe-
the chain) as it was trying to break down the complex query.
cialized models, and RQ-RAG (Chan et al., 2024), which
Thus, the multi-hop query requires careful reasoning. Next,
enables query rewriting and decomposition for handling

3
 Efficient Test-Time Planning for Retrieval Augmented Generation
Query:
we turn to approaches that use more test-time compute. What is the distance between the locations that
hosted the last two Men’s Cricket World Cup finals?
Reason & Act (ReAct) ReAct (Yao et al., 2023) extends Q1.2: Where was
Q1.1: Where was the
traditional RAG frameworks by implementing a structured last Men’s Cricket Parallelizable subqueries
the second last
Men’s Cricket
World Cup final held?
interaction loop between reasoning and retrieval. The frame- World Cup final held?

Debuggability & Backtracking

Relevant flow of information
work decomposes each iteration into three components: a
Thought phase for planning, an Action phase for doing re- Q2.1: What are Q2.2: What are
trievals, and an Observation phase for incorporating re- the coordinates of
⟨A1.1⟩?
* Targetted Subquery Retrievals the coordinates of
⟨A1.2⟩?
trieved information. For our example, ReAct produces the
following plan (and retrievals): (T1) Need to find the author
Q3.1: What is the
of the coming-of-age novel, then find its Viking Press publi- distance between
cation year → ⟨ Retrieve ⟩ → (T2) Found S.D. Smith, need ⟨A2.1⟩ and ⟨A2.2⟩?

to verify Viking Press publication → ⟨ Retrieve ⟩ → (T3)

Located publication info for Rumble Fish, ultimately an- Figure 3. Reasoning plan example: A Reasoning DAG generated
swering 1975. This trace reveals how ReAct, despite using by the reasoning plan expert, highlighting key advantages: only
more test-time compute, fails to keep track of the original relevant information flows to each subquery, subqueries on the
same depth can be executed in parallel, and the DAG structure
query intent. ReAct approach relies entirely on the LM’s
allows for debugging and backtracking.
in-context memory, that keeps growing with iterations and
retrievals. It becomes challenging for the LM to track and
update the plan periodically in light of the new retrievals. As
tested on our example query, it exhibits the same limitations,
shown in Fig. 2, while ReAct enables dynamic reasoning, it
producing the incorrect answer 1975 despite its sophisti-
is vulnerable to error propagation, and struggles to maintain
cated verification mechanism.
entity relationships across steps.
The aforementioned methods incorporate different strategies
Query Decomposition (QD) Query decomposition meth-
and varying degrees of test-time compute to handle reason-
ods (Patel et al., 2022) address the limitations of previous
ing in RAG systems. But they all share fundamental limi-
approaches by breaking queries into independent, sequen-
tations, listed below, that lead to failures even on 2–3 hop
tial subqueries. LM first decomposes the main query into
queries such as our running example. We identify two obser-
subqueries, then solves them sequentially while maintain-
vations that motivate our proposed framework Plan∗ RAG:
ing previous subquery-answer pairs in context. For our
example, it generates two subqueries: “What is the author • First, by accumulating intermediate reasoning steps in
of the coming-of-age novel published by Viking Press?” the context window, they create information overload as
yielding “S.E. Hinton”, followed by “In what year was the context grows. It becomes challenging for the LM to
the coming-of-age novel published by Viking Press?” pro- identify and resolve the dependencies between subqueries
ducing “1975”. Firstly, we note that the decomposition is and faithfully execute a reasoning plan.
not very helpful. Secondly, the LM answers the subqueries
based on the first document retrieved for the original query, • Second, state-of-the-art RAG approaches fail to leverage
which is on Rumble Fish (and gets the author name right the inherent independence between different reasoning
for the first sub-query). Despite explicitly decomposing the paths in multi-hop queries. For instance, finding an au-
query, the method still fails to maintain coherent reason- thor’s identity and locating a publication date can be an-
ing across subqueries, and loses critical entity relationships swered independently, without overburdening the LM’s
between subsequent sub-queries. Recent approaches such context with all reasoning paths.
as RQ-RAG (Chan et al., 2024) and RA-ISF (Liu et al.,
2024) extend query decomposition, and incorporate query 4. Plan∗ RAG: Test-Time Planning
rewriting tricks, but inherit similar limitations.
We present Plan∗ RAG, a novel framework for multi-hop
Self-RAG Asai et al. (2023) recently proposed Self-RAG reasoning through dynamic, test-time planning via Directed
that introduces a novel self-evaluation mechanism. The Acyclic Graphs (DAGs). The key ideas of our formulation,
LM generates multiple candidate responses and scores them motivated by the issues faced by the current RAG systems
based on their faithfulness to retrieved evidence as well discussed in Sec. 3, are:
as relevance to the original query. This approach aims to
improve reasoning quality through verification. However, 1. We want to isolate the reasoning plan, outside of the
Self-RAG still fundamentally relies on the language model’s LM’s in-context memory, as a global data structure that
implicit reasoning capabilities within a single context win- can help with tracking, execution, and verification. We
dow, similar to the previously discussed approaches. When can generate the plan once statically, refine, and execute
the plan using data structure operations. In this work, we

4
 Efficient Test-Time Planning for Retrieval Augmented Generation

Table 1. Reasoning DAG depth (percentage/count) for multi-hop (HotpotQA, StrategyQA, MuSiQue) and single-hop (PopQA) data sets.
For single-hop queries, the DAG primarily has depth 0, which is desirable, while multi-hop queries require deeper reasoning paths.
Dataset Depth 0 Depth 1 Depth 2 Depth 3 Depth≥4
HotpotQA (Multi-hop) 0.5% (35) 12.8% (945) 79.5% (5884) 6.8% (507) 0.4% (34)
StrategyQA (Multi-hop) 0.9% (22) 42.9% (978) 51.2% (1175) 4.5% (103) 0.3% (6)
MuSiQue (Multi-hop) 0.0% (0) 2.11% (51) 66.4% (1604) 25.5% (617) 5.9% (145)
PopQA (Single-hop) 77.9% (1090) 0.8% (11) 18.9% (264) 2.4% (34) 0.0% (0)

use Directed Acyclic Graphs (Fig. 3) that are appropriate its parent nodes’ answers, and retrieved documents. When
to represent complex plans. applied recursively, this formulation enables systematic
2. DAG structure allows executing any node conditioned reasoning from root nodes to leaves, with document
only on the necessary context along its path, unlike state- retrieval potentially interleaved at each step.
of-the-art approaches that accumulate the entire trace To facilitate dynamic reasoning, we introduce a simple node
(plan and retrievals) in the LM’s in-context memory. This indexing scheme. Each node is uniquely identified as ⟨i.j⟩,
also helps make the LM calls more economical in terms where i denotes the node’s depth from the root and j indi-
of the number of tokens. cates its position among nodes at depth i. Furthermore, we
3. DAG structure allows processing nodes independently, introduce a special tag ⟨AI.J⟩ that enables dynamic depen-
conditioned on the execution of parent nodes, helping dency tracking between subqueries. In this notation, I and J
with efficiency and end-to-end latency of RAG system. are integer values representing the Question IDs required to
complete a subquery. For instance, as illustrated in Fig. 3,
At a high level, Plan∗ RAG operates in two phases: (1) de- when subquery Q2.1 depends on the answer to Q1.1, the tag
composing complex queries into a DAG structure, and ⟨A1.1⟩ enables dynamic answer propagation at inference
(2) executing the DAG while preserving dependencies. Be- time, allowing the system to adapt to updated knowledge.
sides the merits of serializing the query plan as a data
structure—such as control and efficiency listed above, 4.2. Generating a Reasoning Plan as a DAG
Plan∗ RAG is complementary to state-of-the-art approaches
like Self-RAG in the following sense. We can seamlessly The first step during inference involves generating a reason-
integrate Plan∗ RAG with methods like Self-RAG, to solve ing plan by prompting the LLM with a specialized prompt,
the sub-queries in its internal nodes (see Sec. 5.1). as detailed in App. D.1. While this initial plan is static, it can
be dynamically instantiated, and refined, given the indexing
We formalize the reasoning plan DAG in Sec. 4.1, followed scheme ⟨AI.J⟩ as discussed in Sec. 4.1. The plan therefore
by detailed analysis on properties and benefits of Plan∗ RAG is a template initially which during generation dynamically
in Sec. 4.3. The full algorithm is showcased in Alg. 1. materializes as the LM traverses through the DAG.

4.1. Reasoning Plan: Directed Acyclic Graph To ensure reliable plan generation, we explore two ap-
proaches: (1) fine-tuning a language model specifically
At test-time, Plan∗ RAG generates a reasoning plan as a for this structured output, and (2) leveraging more capa-
Directed Acyclic Graph (DAG) G(V, E), where V repre- ble language models that can follow complex prompting
sents the set of generated subqueries and E denotes the instructions. Our experiments show both approaches are
directed edges between them. The root nodes of G cor- effective, with a fine-tuned Llama3.1-instruct8B achieving
respond to independently answerable atomic subqueries, comparable performance to GPT-4o (Sec. 5.4).
while subsequent nodes represent dependent queries that
build upon their parent nodes’ answers. This hierarchical 4.3. Reasoning Plan: Properties & Benefits
structure follows the Markov assumption, ensuring that each
node’s answer depends only on its direct predecessors, thus The effectiveness of Plan∗ RAG stems from its structured
enabling efficient parallel computation. reasoning process, which offers several key advantages
over traditional RAG-based methods. By leveraging a Di-
Formally, for any subquery q∈V, its answer is computed as: rected Acyclic Graph (DAG) to represent reasoning plans,
G(q) = fLLM (G(Pa(q)), q, Dq ) ; G = fLLM (Q) , (1) Plan∗ RAG ensures efficient information flow, reduces com-
putational overhead, and improves query formulation. The
where Pa(q) denotes the set of parent nodes of q in the DAG structure naturally enables systematic verification and
DAG, Dq represents the retrieved documents relevant to granular control at the subquery level—errors can be iso-
q, and Q is the main query. The language model fLLM lated to specific nodes and automated interventions (like
generates responses by jointly considering the subquery, additional retrievals or alternative decompositions) can be

5
 Efficient Test-Time Planning for Retrieval Augmented Generation

Algorithm 1 Plan∗ RAG framework

Table 2. Evaluation with standard prompting methods: Per-
formance comparison of Plan∗ RAG against traditional RAG ap- Input: Q: Query
proaches (Vanilla-RAG, CoT-RAG, QD-RAG) and vanilla LLMs Output: Generation G
Get a reasoning plan: G ← fLLM (Q)
across different model scales.
Identify root nodes of G: qroot
Method HotpotQA StrategyQA MuSiQue PopQA Calculate depth of each node from root:
lq ← maxdist(q, qroot )
Vanilla-LLM 30.45 55.37 6.53 28.45
for i: 0 to maxq (lq ) do
GPT-3.5turbo

Vanilla-RAG 29.83 54.24 4.27 32.13

for parallel: q in {q : lq = i} do
CoT-RAG 29.96 60.66 6.50 30.02
Get parent questions and answers:
QD-RAG 29.31 59.91 4.85 36.03
Mq ← Pa(q) & Ma ← GMq
Plan∗ RAG 31.72 62.07 6.71 38.62
q̃ ← fLLM (q, Mq , Ma ) {Generate the subquery}
Vanilla-LLM 19.33 44.47 2.81 22.30 Gq ← fLLM (q̃) {Obtain generated answer for query q}
Llama2-chat13B

Vanilla-RAG 22.67 39.11 2.65 31.25 end for

CoT-RAG 25.93 29.26 5.02 34.17 end for
QD-RAG 26.97 42.79 4.73 36.88 G ← Ω(q, G1 , G2 , . . . , Gq ) {Dataset Dependent Operator}
Plan∗ RAG 27.51 54.57 5.32 41.45 return G
Vanilla-LLM 21.34 48.43 4.18 23.52
Llama3.1-instruct8B

Vanilla-RAG 25.51 47.36 4.39 39.1

CoT-RAG 30.10 59.51 6.00 37.49 where tgen (qi ) is the generation time for subquery qi and
QD-RAG 28.68 62.38 2.65 41.13 tret (Di ) is the retrieval time for documents Di . In contrast,
Plan∗ RAG 31.12 68.16 6.28 41.51 Plan∗ RAG enables parallel execution of independent nodes
at the same depth, reducing the total computation time to:
targeted at specific reasoning steps. While we leave the full ∗ PD
T Plan RAG = d=0 maxq∈Vd (tgen (q) + tret (Dq )) , (3)
exploration of these verification and intervention capabilities
to future work, this structural advantage distinguishes our where Vd represents the set of nodes at depth d and D is the
∗
approach from sequential reasoning methods. Below, we maximum depth. Since, T Plan RAG ≤ T seq , the efficiency
analyze the fundamental properties that enhance multi-hop gain increases with the number of parallel branches. This
reasoning while maintaining computational efficiency. parallelization significantly reduces latency compared to
Atomic Subqueries Plan∗ RAG decomposes complex sequential approaches like query decomposition or ReAct.
queries into atomic subqueries—unlike existing query Context Window Utilization Plan∗ RAG achieves optimal
decomposition methods that perform sequential, local context window utilization through selective information
decompositions. An atomic refers to subqueries that request propagation. For sequential reasoning methods like Query
a single piece of information and thus can be answered by Decomposition or ReAct, the context size grows linearly,
a single retrieval—further decomposition does not yield Pt
additional useful granularity. This atomic nature enables Ctseq = |Q| + i=1 (|qi | + |G(qi )| + |D|) , (4)
precise retrievals (Table 4) and reduces hallucination risk where |Q| is the main query length, qi are intermediate
by constraining the scope of each reasoning step. For questions, G(qi ) their generations, and Di the retrieved
instance, consider the query in Fig. 2, instead of broad documents. In contrast, Plan∗ RAG maintains a constant
queries like “In what year was the coming-of-age novel context size at each node q,
published by Viking Press?” (like query decomposition
method), Plan∗ RAG generates specific atomic queries like Cqours = |q| + |Dq | + p∈Pa(q) (|p| + |G(p)|) . (5)
P
“In what year was The Outsiders published by Viking Press?”
for Q3.1, where the intermediate generation for ⟨A2.1⟩ is This selective propagation of only parents information,
dynamically replaced with the corresponding generation. rather than entire reasoning history, enables efficient scaling
to multi-hop queries while maintaining reasoning quality.
Parallelization & Efficiency The DAG structure explic-
itly captures conditional independence among subqueries
through its edge relationships. This independence enables
5. Experiments
parallel execution of reasoning paths, significantly reduc- We conduct extensive experiments to evaluate Plan∗ RAG
ing latency compared to sequential approaches like query across diverse reasoning scenarios, demonstrating signifi-
decomposition or ReAct. By controlling information flow cant improvements across multiple datasets.
through explicit parent-child relationships, Plan∗ RAG min-
imizes token overhead and improves computational effi- Datasets We evaluate Plan∗ RAG on four datasets that
ciency (Fig. 4). Formally, for sequential methods, test different aspects of reasoning capabilities: HotpotQA
Pn (Yang et al., 2018) and StrategyQA (Geva et al., 2021) for 2-
T seq = i=1 (tgen (qi ) + tret (Di )) , (2) hop reasoning, MuSiQue (Trivedi et al., 2022) for complex

6
 Efficient Test-Time Planning for Retrieval Augmented Generation

80

Percentage of examples (%)

2000 ReAct ReAct
Log Avg. tokens / query

1000 QD-RAG QD-RAG

60
CoT-RAG Plan∗ RAG
500 Plan∗ RAG
250 40

100 20
50

Input 0
Output 0 1 2 3 ≥4
Reasoning depth
Figure 4. Efficiency analysis: (a) Token utilization comparison showing that Plan∗ RAG maintains comparable computational efficiency
with baseline methods. (b) Analysis of reasoning depth on HotpotQA demonstrates that Plan∗ RAG naturally adapts to the dataset’s 2-hop
nature, achieving optimal depth for 80% of queries while other methods show inconsistent reasoning depths.

multi-hop reasoning (>2 hops), and PopQA (Mallen et al., 5 documents retrieved via Contriever-MS MARCO from
2022) for single-hop queries. This diverse selection enables the original query. As shown in Table 2, we compare
comprehensive evaluation of Plan∗ RAG’s adaptability to against vanilla LLMs and three RAG variants (standard
varying reasoning depths. We consider an answer correct if retrieval, chain-of-thought prompting, and query decompo-
the predicted answer contains the ground truth, providing a sition) using GPT-3.5turbo , Llama2-chat13B , and Llama3.1-
relaxed version of exact-match to account for variations in instruct8B . Plan∗ RAG demonstrates consistent performance
answer phrasing. Dataset details are provided in App. A. gains across all datasets and baseline models.
Retriever For all experiments, we use the Contriever-MS For state-of-the-art methods that employ iterative retrieval,
MARCO (Izacard et al., 2022) retriever with embeddings we evaluate Plan∗ RAG SubQ . As shown in Table 3,
based on the 2018 English Wikipedia. The Wikipedia arti- Plan∗ RAG SubQ achieves significant improvements over
cles are segmented into non-overlapping 100-word chunks, recent frameworks like Self-RAG, ReAct, and RQ-RAG
and we retrieve the top-5 documents for each query. across respective base models. Additionally, integrating
Self-RAG to Plan∗ RAG’s reasoning DAG (Plan∗ -Self-RAG)
Baselines and Methods We evaluate Plan∗ RAG against
yields substantial performance gains, validating the effec-
several baselines as discussed in Sec. 3: LM with no re-
tiveness of the test-time, external planning approach.
trieval (Vanilla-LLM), standard retrieval with direct prompt-
ing (Vanilla-RAG), chain-of-thought prompting with re- These results establish that Plan∗ RAG’s test-time planning
trievals (CoT-RAG), query decomposition with retrievals provides clear advantages over existing reasoning frame-
(QD-RAG), and state-of-the-art RAG methods, like Self- works, including those designed for multi-hop reasoning.
RAG (Asai et al., 2023), ReAct (Yao et al., 2023), and RQ-
RAG (Chan et al., 2024). These methods have outperformed 5.2. Reasoning DAG Depth
methods like SAIL (Luo et al., 2023), Toolformer (Schick
et al., 2024), and commercial systems like [Link] and Table 1 shows the reasoning DAG depths for all the datasets.
ChatGPT. We conduct experiments using three base models: As expected, the multi-hop query datasets exhibit a DAG
GPT-3.5turbo , Llama2-chat13B , and Llama3.1-instruct8B . depth greater than 1, indicating multiple atomic queries must
be answered at different depths to answer the main query. In
We implement two variants of Plan∗ RAG: (1) a base ver- contrast, the single-hop dataset typically shows a reasoning
sion that uses documents retrieved from the original query; depth of 0, which is both expected and desired, as simpler
(2) Plan∗ RAG SubQ that performs iterative retrievals for each queries do not require further decomposition into subqueries.
subquery in the DAG, enabling precise retrievals. Given This showcases that the reasoning DAG effectively adapts
Plan∗ RAG’s agnostic design discussed in Sec. 4, we also its complexity based on the query complexity.
evaluate its integration with Self-RAG (Plan∗ -Self-RAG).
On HotpotQA (a 2-hop dataset), we compare depth distri-
butions for sequential methods like Query-Decomposition
5.1. Accuracy Performance
RAG (QD-RAG) and ReAct in Fig. 4. Both the methods
We evaluate Plan∗ RAG against both standard prompting exceed the ideal reasoning depth, with a significant pro-
techniques and state-of-the-art RAG frameworks in Ta- portion of samples having depths greater than 3, indicating
ble 2 and Table 3. For fair comparison with standard suboptimal reasoning paths.
prompting methods, all approaches use the same set of top-
Information Gain (IG) We introduce cumulative informa-

7
 Efficient Test-Time Planning for Retrieval Augmented Generation

Table 3. Comparison with state-of-the-art methods: Performance of Table 4. Retrieval effectiveness: Evaluation of re-
Plan∗ RAG against recent advanced RAG methods (RQ-RAG, Self-RAG) and trieval quality on HotpotQA distractor setting.
their Plan∗ -augmented variants. Plan∗ RAG SubQ achieves consistent improve- Plan∗ RAG achieves higher precision and recall com-
ments across both single-hop and multi-hop reasoning tasks. pared to ReAct and query decomposition.
Base Method HotpotQA StrategyQA MuSiQue PopQA Method Precision Recall Accuracy
ReAct 0.04 34.61 25.68
RQ-RAG 33.78 47.46 13.43 32.66
Llama27B

QD-RAGSubQ 0.02 32.16 23.54

Self-RAG 31.48 43.71 7.07 43.67 Plan∗ RAGSubQ 27.43 36.57 31.27
Plan∗ -Self-RAG 36.65 60.67 14.76 44.06
Table 5. Reasoning planners: Accuracy comparison
Llama3.1 Llama213B

Self-RAG 34.09 42.75 8.31 45.18

showing that a fine-tuned Llama3.1-instruct8B planner
Plan∗ -Self-RAG 37.31 62.87 16.72 45.23 achieves comparable performance to GPT-4o.
instruct8B

ReAct 33.15 54.67 13.36 35.81 Vanilla-RAG CoT-RAG QD-RAG Plan4o Plan8B
Plan∗ RAG SubQ 40.44 65.45 14.88 41.98 24.83 30.11 28.25 31.97 31.28

tion gain (IG) to measure information aggregation across questions and their corresponding reasoning DAGs in the
reasoning depths. Formally, at depth d, required JSON format as training data. Experiments on a
HotpotQA-dev subset (1300 samples) demonstrate compa-
IG(Q, G, d) = IG(Q, {qi }di=0 ) , (6)
rable performance to GPT-4o, achieving [31.28] accuracy
where Q represents the main query and {qi }di=0 denotes versus [31.97] with GPT-4o (Table 5). These results indicate
the set of subqueries up to depth d. A higher IG indicates that the success of our test-time planning approach does not
substantial new information contribution toward the final hinge on specific LLMs; and that modest training data is
answer, while a similar value as the previous depth suggest sufficient to fine-tune LMs to generate structured output.
redundant or unnecessary reasoning steps. We experiment
with the HotpotQA dataset and employ GPT-3.5turbo model 6. Discussion and Conclusion
to score the information gain between 1-10 across the rea-
soning plan. Table 6 showcases the average IG score over In this paper, we presented Plan∗ RAG, a framework that
reasoning depth, demonstrating consistent incremental in- transforms multi-hop reasoning in RAG through test-time
formation gain as the DAG is traversed. planning. Our key insight—externalizing the reasoning
structure as a DAG outside the LM’s context—addresses
5.3. Targeted Subqueries and Retrievals fundamental limitations of existing approaches. Empirical
results validate three core benefits: systematic exploration
A key feature of Plan∗ RAG’s reasoning plan is the atomic, of reasoning paths, efficient parallel execution, and im-
dynamic subqueries. As discussed in Sec. 4.3, the sub- proved performance on multi-hop datasets. Significant im-
queries in the reasoning plan are more targetted and help provements in retrieval precision demonstrates that atomic
improve retrievals. We evaluate retrieval effectiveness with subqueries enable more focused document retrieval, while
the HopotQA Distractor setting, which provides ground bounded context windows overcome the overflow chal-
truth labels for supporting documents. As shown in Table 4, lenges faced by sequential approaches. A key strength of
Plan∗ RAG significantly outperforms both ReAct and Query- Plan∗ RAG is its modular design, demonstrated by success-
Decomposition (QD-RAGSubQ ) in both precision and recall, ful integration with existing RAG methods like Self-RAG.
contributing to improved accuracy. This flexibility, combined with comparable computational
High precision is particularly crucial for multi-hop reason- costs, establishes Plan∗ RAG as a practical solution for real-
ing, as it minimizes the introduction of irrelevant informa- world multi-hop reasoning tasks
tion that could propagate errors through the reasoning chain. A key limitation in Plan∗ RAG is reliance on a static plan
generated a priori. While the plan may need updates as
5.4. Reasoning Plan Generation more information is collected, the DAG structure facilitates
subquery updates and backtracking since it is a formal ob-
So far in the experiments we have utilized the more capable
ject outside the LLM’s context window. Future work can
GPT-4o model for generating reasoning plans. However,
explore: (1) verification and backtracking using the reason-
to demonstrate that Plan∗ RAG framework is not depen-
ing DAG, (2) feedback loops between reasoning steps for
dent on powerful LLMs, we finetune a Llama3.1-instruct8B
dynamic plan refinement, and (3) extending the framework
reasoning planner with LoRA adapters on a small subset
to tasks like fact verification and mathematical deduction.
of HotpotQA-train dataset. We use input-output pairs of

8
 Efficient Test-Time Planning for Retrieval Augmented Generation

References Hao, S., Gu, Y., Ma, H., Hong, J. J., Wang, Z., Wang, D. Z.,
and Hu, Z. Reasoning with language model is planning
Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. Self-
with world model. In Proceedings of the 2023 Conference
RAG: Learning to retrieve, generate, and critique through
on Empirical Methods in Natural Language Processing,
self-reflection. In The Twelfth International Conference
2023.
on Learning Representations, 2023.
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Ruther- Hsu, S., Khattab, O., Finn, C., and Sharma, A. Grounding
ford, E., Millican, K., Van Den Driessche, G. B., Lespiau, by trying: LLMs with reinforcement learning-enhanced
J.-B., Damoc, B., Clark, A., De Las Casas, D., Guy, A., retrieval. arXiv preprint arXiv:2410.23214, 2024.
Menick, J., Ring, R., Hennigan, T., Huang, S., Maggiore,
L., Jones, C., Cassirer, A., Brock, A., Paganini, M., Irv- Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski,
ing, G., Vinyals, O., Osindero, S., Simonyan, K., Rae, J., P., Joulin, A., and Grave, E. Unsupervised dense infor-
Elsen, E., and Sifre, L. Improving language models by mation retrieval with contrastive learning. Transactions
retrieving from trillions of tokens. In Proceedings of the on Machine Learning Research, 2022.
39th International Conference on Machine Learning, vol-
ume 162 of Proceedings of Machine Learning Research, Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni,
pp. 2206–2240. PMLR, 2022. F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., and
Grave, E. Atlas: Few-shot learning with retrieval aug-
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., mented language models. Journal of Machine Learning
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Research, 24(251):1–43, 2023.
Askell, A., et al. Language models are few-shot learners.
In Advances in Neural Information Processing Systems,
Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-
volume 33, pp. 1877–1901. Curran Associates, Inc., 2020.
Yu, J., Yang, Y., Callan, J., and Neubig, G. Active re-
Chan, C.-M., Xu, C., Yuan, R., Luo, H., Xue, W., Guo, trieval augmented generation. In Proceedings of the 2023
Y., and Fu, J. RQ-RAG: Learning to refine queries for Conference on Empirical Methods in Natural Language
retrieval augmented generation. In First Conference on Processing, pp. 7969–7992, 2023.
Language Modeling, 2024.
Leng, Q., Portes, J., Havens, S., Zaharia, M., and Carbin, M.
Chen, D., Fisch, A., Weston, J., and Bordes, A. Reading Long context rag performance of large language models.
Wikipedia to answer open-domain questions. In Proceed- arXiv preprint arXiv:2411.03538, 2024.
ings of the 55th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), pp. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V.,
1870–1879. Association for Computational Linguistics, Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel,
2017. T., Riedel, S., and Kiela, D. Retrieval-augmented genera-
Chen, Z., White, M., Mooney, R., Payani, A., Su, Y., tion for knowledge-intensive NLP tasks. In Advances in
and Sun, H. When is tree search useful for llm plan- Neural Information Processing Systems, volume 33, pp.
ning? it depends on the discriminator. arXiv preprint 9459–9474. Curran Associates, Inc., 2020.
arXiv:2402.10890, 2024.
Liu, Y., Peng, X., Zhang, X., Liu, W., Yin, J., Cao, J., and
Drozdov, A., Wang, S., Rahimi, R., Mccallum, A., Zamani, Du, T. Ra-isf: Learning to answer and understand from
H., and Iyyer, M. You can’t pick your neighbors, or can retrieval augmentation via iterative self-feedback. arXiv
you? When and how to rely on retrieval in the kNN- preprint arXiv:2403.06840, 2024.
LM. In Findings of the Association for Computational
Linguistics (EMNLP), pp. 2997–3007, 2022. Luo, H., Zhang, T., Chuang, Y.-S., Gong, Y., Kim, Y., Wu,
X., Meng, H. M., and Glass, J. R. Search augmented in-
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and
struction learning. In Proceedings of the 2023 Conference
Berant, J. Did Aristotle use a laptop? A question an-
on Empirical Methods in Natural Language Processing,
swering benchmark with implicit reasoning strategies.
2023.
Transactions of the Association for Computational Lin-
guistics, 9:346–361, 2021.
Ma, X., Gong, Y., He, P., Zhao, H., and Duan, N. Query
Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. rewriting in retrieval-augmented large language mod-
Retrieval augmented language model pre-training. In els. In Proceedings of the 2023 Conference on Empiri-
Proceedings of the International Conference on Machine cal Methods in Natural Language Processing, pp. 5303–
Learning, pp. 3929–3938. PMLR, 2020. 5315, 2023.

9
 Efficient Test-Time Planning for Retrieval Augmented Generation

Mallen, A., Asai, A., Zhong, V., Das, R., Hajishirzi, H., and Torfi, A., Shirvani, R. A., Keneshloo, Y., Tavaf, N., and Fox,
Khashabi, D. When not to trust language models: Investi- E. A. Natural language processing advancements by deep
gating effectiveness and limitations of parametric and non- learning: A survey. arXiv preprint arXiv:2003.01200,
parametric memories. arXiv preprint arXiv:2212.10511, 2020.
2022.
Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal,
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can A. Musique: Multihop questions via single-hop ques-
a suit of armor conduct electricity? A new dataset tion composition. Transactions of the Association for
for open book question answering. arXiv preprint Computational Linguistics, 10:539–554, 2022.
arXiv:1809.02789, 2018.
Wang, B., Ping, W., Mcafee, L., Xu, P., Li, B., Shoeybi, M.,
Pal, A., Umapathi, L. K., and Sankarasubbu, M. Med- and Catanzaro, B. InstructRetro: Instruction tuning post
halt: Medical domain hallucination test for large lan- retrieval-augmented pretraining. In Proceedings of the
guage models. In Proceedings of the 27th Conference on 41st International Conference on Machine Learning, vol-
Computational Natural Language Learning (CoNLL), pp. ume 235 of Proceedings of Machine Learning Research,
314–334, 2023. pp. 51255–51272. PMLR, 2024a.
Patel, P., Mishra, S., Parmar, M., and Baral, C. Is a ques-
Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi,
tion decomposition unit all we need? In Proceedings of
E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-
the 2022 Conference on Empirical Methods in Natural
consistency improves chain of thought reasoning in lan-
Language Processing, pp. 4553–4569, 2022.
guage models. In The Eleventh International Conference
Petroni, F., Lewis, P., Piktus, A., Rocktäschel, T., Wu, Y., on Learning Representations, 2023.
Miller, A. H., and Riedel, S. How context affects language
models’ factual predictions. In Automated Knowledge Wang, Z., Liu, A., Lin, H., Li, J., Ma, X., and Liang, Y.
Base Construction, 2020. RAT: Retrieval augmented thoughts elicit context-aware
reasoning in long-horizon generation. arXiv preprint
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, arXiv:2403.05313, 2024b.
A., Leyton-Brown, K., and Shoham, Y. In-context
retrieval-augmented language models. Transactions of Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi,
the Association for Computational Linguistics, 11:1316– E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting
1331, 2023. elicits reasoning in large language models. In Advances
in Neural Information Processing Systems, volume 35,
Ranaldi, L., Valentino, M., and Freitas, A. Elicit- pp. 24824–24837. Curran Associates, Inc., 2022.
ing critical reasoning in retrieval-augmented language
models via contrastive explanations. arXiv preprint Welleck, S., Bertsch, A., Finlayson, M., Schoelkopf, H., Xie,
arXiv:2410.22874, 2024. A., Neubig, G., Kulikov, I., and Harchaoui, Z. From de-
coding to meta-generation: Inference-time algorithms for
Schick, T., Dwivedi-Yu, J., Dessı̀, R., Raileanu, R., Lomeli,
large language models. arXiv preprint arXiv:2406.16838,
M., Hambro, E., Zettlemoyer, L., Cancedda, N., and
2024.
Scialom, T. Toolformer: Language models can teach
themselves to use tools. In Advances in Neural Informa- Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W.,
tion Processing Systems, volume 36. Curran Associates, Salakhutdinov, R., and Manning, C. D. HotpotQA: A
Inc., 2024. dataset for diverse, explainable multi-hop question an-
Shuster, K., Poff, S., Chen, M., Kiela, D., and Weston, J. swering. In Proceedings of the 2018 Conference on Em-
Retrieval augmentation reduces hallucination in conver- pirical Methods in Natural Language Processing, 2018.
sation. arXiv preprint arXiv:2104.07567, 2021.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan,
Sun, J., Xu, C., Tang, L., Wang, S., Lin, C., Gong, Y., Ni, K., and Cao, Y. ReAct: Synergizing reasoning and act-
L., Shum, H.-Y., and Guo, J. Think-on-graph: Deep and ing in language models. In International Conference on
responsible reasoning of large language model on knowl- Learning Representations, 2023.
edge graph. In The Twelfth International Conference on
Learning Representations, 2024. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao,
Y., and Narasimhan, K. Tree of thoughts: Deliberate
Tang, Y. and Yang, Y. Multihop-RAG: Benchmarking problem solving with large language models. In Advances
retrieval-augmented generation for multi-hop queries. In in Neural Information Processing Systems, volume 36.
First Conference on Language Modeling, 2024. Curran Associates, Inc., 2024.

10
 Efficient Test-Time Planning for Retrieval Augmented Generation

Yogatama, D., de Masson d’Autume, C., and Kong, L. Adap-

tive semiparametric language models. Transactions of the
Association for Computational Linguistics, 9:362–373,
2021.
Zhao, H., Liu, Z., Wu, Z., Li, Y., Yang, T., Shu, P., Xu,
S., Dai, H., Zhao, L., Mai, G., et al. Revolutionizing
finance with llms: An overview of applications and in-
sights. arXiv preprint arXiv:2401.11641, 2024.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y.,
Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of
large language models. arXiv preprint arXiv:2303.18223,
2023.

Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang,
X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q. V.,
and Chi, E. H. Least-to-most prompting enables complex
reasoning in large language models. In The Eleventh
International Conference on Learning Representations,
2023.

11
 Efficient Test-Time Planning for Retrieval Augmented Generation

Appendices
The supplementary document is organized as follows: App. A presents the characteristics and specifications of datasets used
in the evaluations. App. B details baseline models employing traditional prompting approaches, their implementation and
specific prompts. App. C describes the state-of-the-art RAG methods evaluated in the experiments. App. D elaborates on
the proposed Plan∗ RAG framework, encompassing reasoning plan generation, dynamic subquery generation, and answer
generation through DAG traversal. Finally, App. E provides details on the experiment setup.

A. Dataset details
In this section, we discuss the datasets used in the experiments. The datasets are particularly characterized into multi-hop
and single-hop depending on the nature of the queries they contain.

A.1. Multi-Hop QA
For multi-hop queries, we focus on three datasets: HotpotQA (Yang et al., 2018), StrategyQA (Geva et al., 2021), and
MuSiQue-Ans (Trivedi et al., 2022)
HotpotQA HotpotQA is a multi-hop dataset collected from English Wikipedia. The questions are diverse and not
constrained to any pre-existing knowledge bases or knowledge schemas. The dataset contains 7,405 queries in the dev-
fullwiki split. Although, each question in the dataset comes with two gold paragraphs, as well as a list of sentences in these
paragraphs that crowd workers identify as supporting facts necessary to answer the question, we do not use these in our
experiments and use contriever to fetch the relevant documents from the Wikipedia embeddings.
StrategyQA StrategyQA is a question-answering benchmark requiring multiple reasoning steps to answer each question.
Questions are short, topic-diverse, and require specific reasoning strategies for answers. We use the dataset that is available
on the Self-RAG repository Asai et al. (2023). The dataset consists of 2,234 question-answer pairs. The dataset covers a
broad range of topics including science, history, and common sense reasoning.
MuSiQue-Ans MuSiQue-Ans is a multi-hop question answering dataset comprising 2,417 questions spanning 2 to 4
reasoning hops. Compared to existing QA datasets, MuSiQue-Ans presents a higher level of difficulty, evidenced by a
threefold increase in the gap between human and machine performance. It is specifically designed to be resilient against
models that rely on disconnected or shallow reasoning patterns.

A.2. Single-Hop QA
To evaluate model performance on single-hop queries, we use PopQA (Mihaylov et al., 2018).
PopQA PopQA is an open-domain question-answering dataset designed to assess a model’s ability to retrieve and generate
answers based on factual knowledge. The dataset consists of factual questions, many of which require specific knowledge of
popular culture, history, and general world facts. In total the size of the dataset is 1,399 question-answer pairs.

B. Standard Prompting Methods

In this section we discuss various standard prompting methods experimented in Sec. 5.1 (cf. Table 2) along with the specific
prompts. For all the models, we set temperature=0 and as all the RAG methods discussed below use the query to retrieve
top-k documents, all the methods use the same set of retrieved documents.
Note that for the Llama2 model, we wrap the prompts with the required system tags:
<s>,[INST],<<SYS>>,<</SYS>>.
Vanilla-LLM We experiment with three LLM models without any retrievals to answer the queries using its parametric
knowledge: GPT-3.5turbo , Llama2-chat13b , and Llama3.1-instruct8b . Below is the prompt that we use:

Be precise and give answer to the query. Response should be a valid JSON, that can be
passed to [Link] directly, with a key as Response which only has 2-3 words. Do not
use complete sentences or punctuation. In JSON, put every value as a string always,
not float.

12
 Efficient Test-Time Planning for Retrieval Augmented Generation

Example:

Query: What is the capital of France?

{"Response": "Paris"}
Query: How do you make coffee?
{"Response": "Brew ground beans"}

Vanilla-RAG We experiment with the standard RAG method, i.e. retrieving top-k relevant documents for each query and
providing them as context to the LLM. For our experiments, we use Contriever (Izacard et al., 2022) as the retriever with
k=5. Similar to Vanilla-LLM, we experiment with 3 models: GPT-3.5turbo , Llama2-chat13b , and Llama3.1-instruct8b . Below
is the prompt that we use:
You are a concise answering assistant. Use the Retrievals while generating the answer
and keep the answer grounded in the retrievals. Generate a JSON with a single key "
Response" and a value that is a short phrase or a few words. In JSON, put every value
as a string always, not float.
Note: Generate only JSON and no explanation or repeating the query or retrievals.

Example:

Query: Who was the PM of India when India performed its first nuclear test?
Retrievals: [["Indira Gandhi was the PM of India in 1974"], ["Indira Gandhi was the
first woman PM of India."]]
Generation: {
"Response": "Indira Gandhi"
}

Query: What is the capital of France?

Retrievals: [["Paris is the capital of France."]]
Generation: {
"Response": "Paris"
}

CoT-RAG We combine chain-of-thought (CoT) prompting with RAG, using Contriever as the retriever with k=5. We
experiment with the three models: GPT-3.5turbo , Llama2-chat13b , and Llama3.1-instruct8b . Below is the prompt that we use:
You are a highly intelligent assistant skilled at solving problems by reasoning step-
by-step. For each query, explain your reasoning clearly and logically in multiple
steps, and then provide the final answer (Response). Use retrievals and keep your
generation grounded in the retrievals. Produce a list of reasoning steps along with
the Answer (Response) (precise and max 3-4 words).
Note: Generate only reasoning steps and response. no explanation or repeating the
query or retrievals. Strictly follow the below template and only return the JSON with
no explanation.

Example:

Query: What is the sum of 15 and 27?

Retrievals: []
Generation: {
"Reasoning_steps" : ["The first number is 15", "The second number is 27", "
Adding these gives 15 + 27 = 42"],
"Response": "42"
}

Query: Who was the PM of India when India performed its first nuclear test?
Retrievals:[["Indira Gandhi was the PM of India in 1974"], ["Indira Gandhi was the
first woman PM of India."]]
Generation:{
"Reasoning_steps" : ["India performed its first nuclear test in 1974", "Indira
Gandhi was the PM of India in 1974."],
"Response": "Indira Gandhi"
}

13
 Efficient Test-Time Planning for Retrieval Augmented Generation

QD-RAG We experiment with query decomposition (QD) based RAG, where the original query is first broken down into
simpler sub-queries. We use Contriever as the retriever with k=5 and evaluate three models: GPT-3.5turbo , Llama2-chat13b ,
and Llama3.1-instruct8b . Below is the prompt used for query decomposition:
You are a helpful assistant that breaks complex queries into simpler ones that are
easy to answer. Therefore, your job is to simplify complex queries into multiple
queries that can be answered in isolation to eachother. If the query is simple, then
keep it as it is.
NOTE: Always return a python list of subqueries that be passed to eval() directly.

Examples:

Query: Did Microsoft or Google make more money last year?

Subqueries: [‘How much profit did Microsoft make last year?’, ‘How much profit did
Google make last year?’]

Query: What is the capital of France?

Subqueries: [‘What is the capital of France?’]

Query: Who has the highest score in the last two ODI cricket world cups?
Subqueries: [‘Who has the highest score in the last ODI cricket world cup?’, ‘Who has
the highest score in the second last ODI cricket world cup?’, ‘Who scored the highest
score among these two?’]

After decomposing the query into sub-queries, we solve each sub-query sequentially. The answers to previous sub-queries
are provided as “Known answers” to help maintain consistency and build upon intermediate findings. Below is the prompt
used for answering each sub-query:
You are a concise answering assistant. Use the Retrievals while generating the answer
and keep the answer grounded in the retrievals. If Known answers are given, use them
while generating the response. Generate a JSON with a single key "Response" and a
value that is a short phrase or a few words. In JSON, put every value as a string
always, not float.
Strictly follow the format below, and provide only the "Generation" part.

Example:

Query: Who was the PM of India when India performed its first nuclear test?
Retrievals: [["Indira Gandhi was the PM of India in 1974"], ["Indira Gandhi was the
first woman PM of India."]]
Known answers: Q=When did India perform the first nuclear test? A=India conducted its
first nuclear test on May 18, 1974, at the Pokhran Test Range in Rajasthan, India.
Generation: {
"Response": "Indira Gandhi was the PM of India when India performed its first
nuclear test in 1974."
}

Query: What type of literature did John Keble write?

Retrievals: [["John Keble (1792-1866) was an English clergyman, poet, and theologian,
best known for his contributions to religious poetry"], ["Keble wrote essays and
sermons emphasizing the importance of tradition, the authority of the Church, and the
significance of the sacraments."]]
Generation: {
"Response": "Religious and devotional poetry"
}

C. State-of-the-Art RAG Methods

In this section we discuss various state-of-the-art RAG methods experimented in Sec. 5.1 (cf. Table 3).
Self-RAG Asai et al. (2023) proposed Self-RAG, wherein the base Llama2 models are trained to learn a set of special
reflection tokens. The reflection tokens are then used at the time of inference to judge the requirement for retrievals,

14
 Efficient Test-Time Planning for Retrieval Augmented Generation

relevance of the retrieved documents and the accuracy of the output. We tested Self-RAG7b and Self-RAG13b on both single
and multi-hop datasets. As a retriever, we use contriever and set k=5. The temperature for both the models was set to 0 to
maintain non-stochasticity. We use the official codebase released by the authors for all the experiments.
RQ-RAG Chan et al. (2024) proposed RQ-RAG, a framework where the base Llama2 model is trained to enable it to
dynamically refine search queries through rewriting, decomposing, and clarifying ambiguities. Further, control tokens are
used to direct the generation process. In addition, the authors use three different sampling methods which includes selection
based on perplexity (PPL), confidence, and an ensemble approach, in order to select the final answer. In our experiments,
total of k=5 documents are retrieved at each depth for any given query and the maximum depth is set to 2. The ensemble
answer is used in all the experiments as that gave the best results. Similar to Self-RAG, temperature was set to 0 to remove
any stochasticity. We use the official codebase released by the authors for all the experiments.
ReAct ReAct (Yao et al., 2023) combines reasoning and acting with language models to solve diverse language reasoning
and decision-making tasks. The model uses an interleaved sequence of Thought, Action, and Observation steps to answer
questions. For the experiment, we use Contreiver as a retriever, and the actions are limited to two options: ‘search’ and
‘finish’. Therefore, the prompt for the models were changed accordingly. We experimented using both Llama3.1-instruct8b
and GPT-3.5turbo models, with the temperature set to 0 in both cases. We used only one retrieval (k=1) per thought as setting
k=5 documents per thought gave poor results due to exploding contexts. The prompt used is as follows:
You are a question answering agent, you need to solve the given question with
interleaving Thought, Action, Observation steps. You need to do this one step at a
time, given some previous steps. Thought can reason about the current situation, and
Action can be of two types:
Search[entity], which searches and returns the top 1 relevant article.
Finish[answer], which returns the answer and finishes the task.

Action can be only of the above two forms. They need to either include Search or
Finish with the entity or answer enclosed in the bracket respectively.

Here are some examples:

Question: Musician and satirist Allie Goertz wrote a song about the "The Simpsons"
character Milhouse, who Matt Groening named after who?

Thought 1: I only need to search Milhouse and find who it is named after.
Action 1: Search[Who was Milhouse named after?]
Observation 1: [Lisa kisses Milhouse. Lisa told Milhouse he should not give up
searching for other girls and that life has unexpected things to offer. She told him
he is cute in the moonlight, which caused him to fall off a cliff nearly to his death,
but a bald eagle caught him, which left him saying "everything is coming up Milhouse
!" Milhouse was designed by Matt Groening for a "Butterfinger" commercial, and it was
decided to use the character in the series. Milhouse was named after U.S. President
Richard Nixon, whose middle name was Milhous. The name was the most "unfortunate name
"]

Thought 2: Milhouse was named after U.S. president Richard Nixon, so the answer is
Richard Nixon.
Action 2: Finish[Richard Nixon]

D. Plan∗ RAG
We present the prompts and details of Plan∗ RAG, including its reasoning plan, tag replacement, and answer generation LM.
For all the models, we set temperature=0. Note that for the Llama2 model, we wrap the prompts with the required system
tags: <s> [INST] <<SYS>> and <</SYS>>.

D.1. Reasoning Plan

The reasoning plans, as discussed in Sec. 4.2, are created by prompting the language model with the query and a set of
contextual examples. The prompt guides the LLM to generate a minimal DAG where each sub-query’s answer depends
only on its parent nodes. We employ the special tag ⟨AI.J⟩ in sub-queries to explicitly denote dependencies on parent node

15
 Efficient Test-Time Planning for Retrieval Augmented Generation

answers. The prompt used for the DAG generation is:

You are a reasoning DAG generator expert. The goal is to make a reasoning DAG with
minimum nodes. Given a query, if it is complex and requires a reasoning plan, split it
into smaller, independent, and individual subqueries. The query and subqueries are
used to construct a rooted DAG so make sure there are NO cycles and all nodes are
connected, there is only one leaf node with a single root and one sink. DAG
incorporates Markov property i.e. you only need the answer of the parent to answer the
subquery. The main query should be the parent node of the initial set of subatomic
queries such that the DAG starts with it. Return a Python list of tuples of parent
query and the subatomic query which can be directly given to eval().

Strictly follow the below template for output.

For the subquery generation, input a tag <AI.J> where the answer of the parent query
should come to make the query complete.

NOTE: Make the DAG connected and for simple queries return the original query only
without any reasoning DAG.

Example:

Query: Who is the current PM of India?

DAG: "Q: Who is the current PM of India?"

Query:What is the tallest mountain in the world and how tall is it?
DAG: [
("Q: What is the tallest mountain in the world and how tall is it?", "Q1.1:
What is the tallest mountain in the world?"),
("Q1.1: What is the tallest mountain in the world?", "Q2.1: How tall is <A1
.1>?")
]

Query: What percentage of the worlds population lives in urban areas?

DAG: [
("Q: What percentage of the worlds population lives in urban areas?", "Q1.1:
What is the total world population?"),
("Q: What percentage of the worlds population lives in urban areas?", "Q1.2:
What is the total population living in urban areas worldwide?"),
("Q1.1: What is the total world population?", "Q2.1: Calculate the percentage
living in urban areas worldwide when total population is <A1.1> and population
living in urban areas is <A1.2>?"),
("Q1.2: What is the total population living in urban areas worldwide?", "Q2.1:
Calculate the percentage living in urban areas worldwide when total
population is <A1.1> and population living in urban areas is <A1.2>?")
]

D.2. Dynamic Subquery Generation

The interdependence of child nodes on their parents is captured through special ⟨AI.J⟩ tags. During sub-query answer
generation, we prompt the LLM to replace these tags with their corresponding parent answers to create coherent sub-query
question. Below is the prompt used for tag replacement:

You are provided a question with tags where the corresponding tag answers need to be
replaced. Replace tags with answers of the previous question in such a way that the
final question is coherent and logical.
Just replace all parts of the answers in the main question. Do not reason or answer
the question. Your role is just to replace tags with all parts of the answer.

NOTE: Only output the question with no explanation or any other details.

Example:

16
 Efficient Test-Time Planning for Retrieval Augmented Generation

Query: Q2.1: Who was the president of India when the captain of the Indian cricket
team was <A1.1> and vice-captain was <A1.2> in 2018?
Q1.1: Who was the captain of India cricket team in 2018?
A1.1: The captain of Indian cricket team in 2018 was [Link].
Q1.2: Who was the vice-captain of India cricket team in 2018?
A1.2: The vice-captain of Indian cricket team in 2018 was Virat Kohli.
Output: Q2.1: Who was the president of India when the captain of the Indian cricket
team was [Link] and vice-captain was Virat Kohli?

D.3. Generator LLM

Plan∗ RAG involves traversing the reasoning plan to generate answers for each sub-query and ultimately resolve the main
query. The following prompt is used for each answer generation:
You are a concise answering assistant. If relevant and provided, use the Retrievals
while generating the answer or use your own knowledge. If Known answers are given, use
them while generating the response. Generate a JSON with a single key "Response" and
a value that is a short phrase or a few words. In JSON, put every value as a string
always, not float. Strictly follow the format below, and provide only the "Generation"
part.

Example:

Query: Who was the PM of India when India performed its first nuclear test?
Retrievals: [["Indira Gandhi was the PM of India in 1974"], ["Indira Gandhi was the
first woman PM of India."]]
Known answers: Q=When did India perform the first nuclear test? A=India conducted its
first nuclear test on May 18, 1974, at the Pokhran Test Range in Rajasthan, India.
Generation: {
"Response": "Indira Gandhi was the PM of India when India performed its first
nuclear test in 1974."
}

Query: What type of literature did John Keble write?

Retrievals: [["John Keble (1792-1866) was an English clergyman, poet, and theologian,
best known for his contributions to religious poetry"], ["Keble wrote essays and
sermons emphasizing the importance of tradition, the authority of the Church, and the
significance of the sacraments."]]
Generation: {
"Response": "Religious and devotional poetry"
}

E. Experiments
This section presents comprehensive details of our experimental evaluations. We begin with an analysis of a HotpotQA
example query (Fig. 2), demonstrating comparative method outputs in App. E.1. We then evaluate performance on multi-hop
and single-hop datasets in App. E.2, presenting results in Table 2 and Table 3. Following this, we analyze Plan∗ RAG’s
retrieval effectiveness in App. E.3 and information gain results in App. E.4. Finally, we describe the reasoning plan
generation experiments using a finetuned Llama3.1-instruct8b model in App. E.5.

E.1. HotpotQA Example Query

We analyze a query from HotpotQA to demonstrate how different methods approach multi-hop reasoning:
Query: “Rumble Fish was a novel by the author of the coming-of-age novel published in what year by Viking Press?”
Ground Truth: 1967
As discussed in Sec. 3, we compare various methods using Llama3.1-instruct8b with temperature=0 and Contriever as the
retriever (k=5). Below are the outputs for each method:
Vanilla-LLM generates S.E. Hinton 1975.

17
 Efficient Test-Time Planning for Retrieval Augmented Generation

Vanilla-RAG generates 1975.

Chain-of-Thought (CoT) - RAG produces the following reasoning chain:
S.E. Hinton wrote Rumble Fish → Rumble Fish was published in 1975 → The author of The Outsiders also published in
1975
Final generation: 1975
ReAct generates the following sequence:
Thought 1: I need to find the author of the coming-of-age novel, then find the year it was published by Viking Press.
Action 1: Search[Author of the coming-of-age novel published by Viking Press]
Thought 2: The author of the coming-of-age novel is S.D. Smith, but I am not sure if he published it with Viking Press.
Action 2: Search[Coming-of-age novel by S.D. Smith published by Viking Press]
Thought 3: I found that S.E. Hinton published Rumble Fish, and it was published by Viking Press in 1975.
Action 3: Finish[1975]
Final generation: 1975
Query Decomposition (QD) - RAG decomposes the query as follows, we show both the query and the generation: Q1:
What is the author of the coming-of-age novel published by Viking Press?
A1: S.E. Hinton
Q2: In what year was the coming-of-age novel published by Viking Press?
A2: 1975
Q3: What is the title of the novel written by the author of the coming-of-age novel?
A3: The Outsiders
Final generation: 1975
Self-RAG using the fine-tuned Llama213b model generates 1975.
Plan∗ RAG generates the following reasoning plan:
Q1.1: Who is the author of Rumble Fish? → Q2.1: What is the coming-of-age novel by ⟨A1.1⟩? → Q3.1: In what year was
⟨A2.1⟩ published by Viking Press?
During the generation the subquery and generation are as follows:
Q1.1: Who is the author of Rumble Fish?
A1.1: S. E. Hinton
Q2.1: What is the coming-of-age novel by S. E. Hinton?
A2.1: The Outsiders
Q3.1: In what year was The Outsiders published by Viking Press?
A3.1: 1967
Final generation: 1967
Therefore, Plan∗ RAG is the only method that arrives at the correct answer of 1967.

E.2. Performance Experiment

Table 2 and Table 3 demonstrate the performance limitations of traditional prompting RAG methods and recently proposed
advanced RAG approaches, while highlighting the superior performance of the proposed method (Plan∗ RAG).
For Table 2, all methods use same set of retrievals obtained via Contriever with k=5 on the main query. The prompts
employed by each model are detailed in App. B. To eliminate stochasticity, we set temperature=0 across all models. For
Table 3, we evaluate against recently proposed advanced RAG methods: Self-RAG (Asai et al., 2023), RQ-RAG (Chan et al.,
2024), and ReAct (Yao et al., 2023). We utilize the official implementations for Self-RAG and RQ-RAG, while adapting
ReAct’s prompt for the RAG framework (detailed in App. C).
For comparison with state-of-the-art methods, we employ Plan∗ RAG SubQ , which retrieves a fresh set of documents for each
sub-query, maintaining consistency with the experimental setup of other state-of-the-art approaches.
For both experiments, we employ accuracy contains as the evaluation metric. This metric represents a relaxed version of the
exact-match score, considering a prediction correct when the true answer appears as a substring within the prediction. This
approach accounts for variations in answer phrasing across different language models.

18
 Efficient Test-Time Planning for Retrieval Augmented Generation

E.3. Retrieval Effectiveness of Plan∗ RAG

In this experiment, we evaluate the effectiveness of
Plan∗ RAG SubQ ’s reasoning plan nodes, which generate
targeted subatomic queries leading to improved retrievals. Table 6. Information Gain (IG): IG at different depths of the reason-
Using a random sample of 1000 queries from the Hot- ing DAG, demonstrating systematic information accumulation across
potQA distractor dataset, which provides golden sen- reasoning steps on HotpotQA, validating the structural coherence of
the reasoning DAG.
tences for evaluation, we assess retrieval performance
against ReAct and QD-RAGSubQ . As shown in Ta- Depth 0 1 2 3 4 5 6
ble 4, Plan∗ RAG SubQ demonstrates superior performance IG 7.23 8.85 8.91 9.02 9.47 9.75 10.0
across all threee metrics: Precision, Recall, and Accuracy.
To account for potential fragmentation of golden sentences during embedding generation, we segment the golden sentences
into chunks of size 50. A retrieval is considered successful if any chunk of the golden sentence is present in the retrieval.

E.4. Information Gain (IG)

We evaluate the effectiveness of Plan∗ RAG’s reasoning plan by analyzing the cumulative information gain (IG) across
DAG-depth. A valid reasoning plan should demonstrate increasing IG with depth, as each subquery contributes additional
relevant information toward answering the main query. Conversely, stagnant or decreasing IG would indicate subqueries at
that depth fail to contribute meaningful information. We formally define IG in Eq. (6).
Using the HotpotQA dataset (7,405 queries), we employ GPT-3.5turbo to compute and return IG scores at each depth of the
reasoning DAG. We showcase the average IG score at each depth in Table 6. The evaluation prompt is
Suppose you are an expert in quantifying information gain. Given a main query and a
set of subqueries quantify how much knowledge is gained by this whole set of
subqueries. Range of information gain: 1-10. 1 being the least and 10 being the most.
The special tag <AI.J> is used to denote the place where the answer to QI.J will be
placed.
NOTE: Only output the Information Gain number that can be passed to eval() directly.
No explanations or anyother tag. Strictly follow the below template.

Example:

Main Query: What is the DoB of the current President of Finland?

Subqueries: [Q1.1: Who is the current President of Finland?]
Information Gain: 5

Main Query: What is the DoB of the current President of Finland?

Subqueries: [Q1.1: Who is the current President of Finland?, Q2.1: When was <A1.1>
born?]
Information Gain: 10

Main Query: What is the DoB of the current President of Finland?

Subqueries: [Q1.1: What is the capital of Finland?]
Information Gain: 0

E.5. Reasoning Plan Generation

While our primary experiments use GPT-4o to generate the reasoning plan, we demonstrate that Plan∗ RAG’s performance is
not dependent on the large language models. For this, we finetune the Llama3.1-instruct8b model with LoRA adapters on a
small subset of HotpotQA-train dataset (3,700 query-plan pairs), using plans generated by GPT-4 as training data. The
results show that a relatively small set of training data is sufficient to finetune a language model to generate structure output
required to construct the DAG.
The finetuning process employs the Parameter-Efficient Fine-Tuning (PEFT) library with the following hyperparameters:
5,000 training steps, LoRA rank of 8, LoRA alpha of 16, batch size of 4, and gradient accumulation steps of 2. Table 5
demonstrates the competitive accuracy of the finetuned model compared to other methods on a subset of HotpotQA-fullwiki
dataset, confirming that Plan∗ RAG’s enhanced performance is not dependent on a large language model.

19