RouterKGQA: Specialized–General Model Routing for Constraint-Aware

Knowledge Graph Question Answering

Bo Yuan1 * Hexuan Deng1,2 * Xuebo Liu1† Min Zhang1

1
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China
2
Zhongguancun Academy, Beijing, China
23s051006@[Link], hxuandeng@[Link]
{liuxuebo,zhangmin2021}@[Link]

Abstract Current approaches to integrating LLMs with

Knowledge graph question answering (KGQA) KGs fall into two paradigms. Agent-based meth-
arXiv:2603.20017v1 [[Link]] 20 Mar 2026

is a promising approach for mitigating LLM ods utilize general LLMs to form an agent work-
hallucination by grounding reasoning in struc- flow that explores the KG through multi-round in-
tured and verifiable knowledge graphs. Ex- teractions to accumulate evidence (Sun et al., 2024;
isting approaches fall into two paradigms: Chen et al., 2024; Dong et al., 2025). While ef-
retrieval-based methods utilize small special- fective, this incurs high computational overhead,
ized models, which are efficient but often pro-
leading to significant inference latency and cost.
duce unreachable paths and miss implicit con-
straints, while agent-based methods utilize Generation-based methods train specialized
large general models, which achieve stronger LLMs to generate structured retrieval plans for KG
structural grounding at substantially higher execution (Luo et al., 2024a,b, 2025). While more
cost. We propose RouterKGQA, a framework efficient, they may generate hallucinated relations
for specialized–general model collaboration, absent from the target KG (Tian et al., 2025), es-
in which a specialized model generates rea- pecially under domain shift. Existing specialized
soning paths and a general model performs
LLMs also often overlook implicit constraints in
KG-guided repair only when needed, improv-
ing performance at minimal cost. We further the question. For example, “Who is the U.S. pres-
equip the specialized with constraint-aware an- ident?” may correspond to multiple tail entities
swer filtering, which reduces redundant an- unless a temporal constraint is specified.
swers. In addition, we design a more efficient To address these issues, we propose
general agent workflow, further lowering in- RouterKGQA, a framework of specialized–
ference cost. Experimental results show that general model collaboration, which achieves
RouterKGQA outperforms the previous best by
better performance with lower cost. Specifically,
3.57 points in F1 and 0.49 points in Hits@1
on average across benchmarks, while requir-
RouterKGQA dynamically routes queries between
ing only 1.15 average LLM calls per ques- a specialized model, which is efficient and per-
tion. Codes and models are available at https: forms well in-domain, and a general model, which
//[Link]/Oldcircle/RouterKGQA. offers stronger generalization. We further improve
both components. For the specialized model, we
1 Introduction
train it to capture implicit constraint semantics in
Recent advances in large language models (LLMs) questions, enabling more precise answer filtering.
have enabled remarkable progress across natural For the general agent workflow, we replace costly
language processing tasks (OpenAI, 2023; Team, multi-round exploration with a cheaper exploration
2024). However, LLMs remain prone to fac- only on relations, which further reduces cost. As
tual hallucination—generating plausible but fac- a result, RouterKGQA achieves strong accuracy
tually incorrect content (Wagner et al., 2025). A while substantially lowering cost compared with
promising direction for mitigating hallucination is conventional agent workflows. Our contributions
to ground LLM reasoning in knowledge graphs are:
(KGs), which store large-scale factual knowledge
in a structured, interpretable form (Bollacker et al., • We propose RouterKGQA, a specialized–
2008; Ji et al., 2024; Sui et al., 2025). general routing framework that combines ef-
* Equal contribution. ficient specialized generation with general re-
†
Corresponding author. pair for KGQA.

1
 • We improve both systems by introducing relation sequences with a fine-tuned LLM and ex-
Constraint-aware Reasoning Paths (CRPs) for ecutes them via breadth-first search. GCR (Luo
specialized models, and by designing a more et al., 2025) further constrains decoding with a KG-
efficient agent workflow for general models. Trie to ensure path validity. GNN-RAG (Mavro-
matis and Karypis, 2025) scores candidate entities
• Experiments on WebQSP and CWQ show that with GNNs and extracts shortest paths as reason-
RouterKGQA achieves the highest F1 on both ing evidence. Logical-form methods instead gen-
benchmarks with only 1.15 average LLM calls erate executable queries, such as S-expressions or
per question. SPARQL. ChatKBQA (Luo et al., 2024a) gener-
ates S-expressions that are converted to SPARQL,
2 Related Work while MemQ (Xu et al., 2025) reconstructs rea-
soning steps into query statements through tool
Factual Hallucination of LLMs. Despite strong
calling. However, both paradigms struggle with im-
language capabilities, LLMs remain prone to fac-
plicit constraints within the question: path-based
tual hallucination, producing fluent yet incorrect
methods cannot explicitly encode them in special-
text (Ji et al., 2023; Huang et al., 2025). Prompting-
ized models, while logical-form methods handle
based methods such as chain-of-thought (Wei et al.,
them in nested formal languages that are harder to
2022) and self-consistency (Wang et al., 2023)
learn (Tian et al., 2025).
improve coherence but cannot guarantee factual
A central challenge of KGQA is relation hallu-
correctness (Ye and Durrett, 2022). Retrieval-
cination, i.e., generating relations absent from the
augmented generation (Lewis et al., 2020; Gao
target KG (Tian et al., 2025). Existing methods
et al., 2023) grounds outputs in external evidence,
mitigate this mainly by restricting the search space,
but standard RAG retrieves unstructured text and
such as KG-Trie-constrained decoding (Luo et al.,
lacks the relational structure needed for complex
2025), subgraph-based path extraction (Mavroma-
reasoning. KG-augmented approaches offer a more
tis and Karypis, 2025), and training-set relation li-
structured alternative by constraining reasoning to
braries (Xu et al., 2025). However, these strategies
verifiable triples (Pan et al., 2024; Agrawal et al.,
do not address the weak out-of-domain general-
2024).
ization of specialized models, which limits their
Agent-based KGQA. Agent-based methods typ- performance upper bound. ChatKBQA (Luo et al.,
ically utilize general LLMs, treat the LLM as an in- 2024a) instead repairs erroneous relations via ex-
teractive agent that explores the KG through multi- haustive post-hoc search, which is prohibitively
round tool calls. ToG (Sun et al., 2024) performs it- slow for complex multi-hop queries.
erative beam-search expansion over entities and re- Our method builds on reasoning-path methods
lations. StructGPT (Jiang et al., 2023) provides spe- and equips them with explicit constraint genera-
cialized interfaces for structured data interaction. tion, improving accuracy. To further address OOD-
DoG (Ma et al., 2025) introduces multi-role de- induced hallucination, we route to a general model
bate for question decomposition. PoG (Chen et al., when specialized models fail to handle the ques-
2024) decomposes questions into sub-objectives tion.
with backtracking and self-correction. While of-
fering strong structural grounding, these methods 3 Method
require complex search over entities and relations
3.1 Problem Formulation
and are often time-consuming. In contrast, we use
a more efficient search scheme that operates only A knowledge graph is defined as
over relations, significantly reducing cost. We fur-
ther route simple questions to a small specialized G = {(s, r, o) | s ∈ E, r ∈ R, o ∈ E ∪ L}, (1)
model, reducing cost even further.
where E, R, and L denote the sets of entities, re-
Retrieval-based KGQA. These methods typi- lations, and literal values, respectively. Given an
cally train specialized models to generate retrieval N -hop question q over G, the goal is to recover the
plans in a single pass, substantially reducing cost. reasoning path
Reasoning-path methods generate relation paths for
KG execution. RoG (Luo et al., 2024b) predicts P = ⟨(s1 , r1 , o1 ), . . . , (sN , rN , oN )⟩, (2)

2
where oN is the final answer to q, and adjacent For example, for the question “Which U.S.
triples satisfy si+1 = oi for i = 1, . . . , N − 1. president graduated from Harvard and assumed
To solve this problem, we propose RouterKGQA, office after 2000?”, the main path R =
as illustrated in Figure 1. The framework con- ⟨[Link], president.office_holder⟩ re-
sists of three stages. First, we train a specialized trieves candidate presidents. We then generate
model, which maps a natural-language question to C = (2, [Link], Harvard) to re-
a Constraint-aware Reasoning Path (CRP), which quire the candidate to be linked to Harvard, and
contains a main path and its associated constraints C = (2, [Link], ≥“2000”) to retain only
(§3.2). Second, if the main path is unreachable, an those whose term started in or after 2000.
agent-based workflow repairs it via beam search,
which requires fewer LLM calls than other meth- Specialized Model Training. To construct in-
ods (§3.3). Finally, the corrected CRP is converted struction fine-tuning data, we convert the SPARQL
into a SPARQL query to retrieve the final answer annotations in the training set into gold CRPs us-
(§3.4). The prompts we used are in Appendix A. ing the deterministic procedure described in Ap-
pendix B. Each training instance pairs a natural-
3.2 Constraint-Aware Path Generation with language question as input with the processed gold
Specialized LLMs CRP as output, following the instruction template
in Appendix A.1. We fine-tune the specialized
We describe how the specialized model generates a
model with LoRA (Hu et al., 2022); implementa-
CRP from a natural-language question, and reacha-
tion details are provided in Section 4.1.
bility verification that determines whether repair is
needed (Stage 1). Reachability verification. The generated main
path may still be inconsistent with the KG because
Initial entity. A CRP starts from a topic entity
of relation errors, direction mismatches, or struc-
name s1 . We let the specialized model first generate
tural mistakes. We therefore execute the skeleton
the initial entity, then ground this predicted name
query induced by R on G. If it returns a non-empty
to its corresponding entity using ELQ (Li et al.,
result set, the main path is reachable. Otherwise,
2020), yielding the grounded topic entity s1 ∈ E.
we assume that the specialized model has failed on
Main path. Following Luo et al. (2024b), we this case and invoke the general model.
represent the reasoning path as a flat sequence
of relations rather than as a nested logical form. 3.3 Efficient Path Repair with General LLMs
This format is easier for small LLMs to learn be- The specialized model is accurate and cost-
cause it avoids recursive syntax and function to- effective on in-domain questions, but often fails
kens, and it keeps the main path separate from con- under domain shift. When reachability verification
straints, enabling independent reachability verifica- fails, we treat the case as beyond the capability of
tion and progressive constraint relaxation. Specif- the specialized model and route the question to a
ically, from P we extract the relation sequence stronger agent workflow (Stage 2). To keep this
R = ⟨r1 , r2 , . . . , rN ⟩ as the main path. Starting stage efficient, we avoid iterative search over enti-
from s1 , we follow the relations in R hop by hop ties and relations. Instead, we first ask the general
to reach the answer oN . model to generate a sequence of high-level sub-
tasks in one shot, and then recover an executable
Constraint set. However, a relation path alone relation path via KG-guided beam search.
is often insufficient because, in a KG, a pair (s, r)
may correspond to multiple objects, yielding mul- Reasoning blueprint generation. Given a ques-
tiple candidate answers. To resolve this ambigu- tion q, the general model decomposes it into a se-
ity, building on prior work, we equip the special- quence of textual subtasks
ized model with the ability to generate constraints.
We generate constraints over intermediate entities, T = ⟨t1 , t2 , . . . , tM ⟩, (4)
which is defined as C = (i, ric , oci ), requiring oi
to satisfy relation ric with value or condition oci . where each tk describes one semantic step in the
Formally, this restricts oi to reasoning process. These blueprints provide se-
mantic guidance for search, but are not assumed to
Eci = {e ∈ E | (e, ric , oci ) ∈ G}. (3) align one-to-one with KG relations.

3
 Stage 1: Specialized LLM Stage 2: General LLM
Step1: find the key actor featured in the film

Step2: find the basketball team he played for.

General LLM
Knowledge
Graph Depth 2
Depth 1 Depth 3
from Step1 from Step1
Generated Multi-Constraint Reasoning Path from Step1 actor ...
actor
subject roster team starring
?c ?y ?x from Step2
director cast team
from Step2
?from <= "2015-01-01" ?to >= "2015-12-31"
writer
position
roster
KG genre
Reachabilit Check sport number
(Main Path)
No (Unreachable) Final
from Step2 league
Yes (Reachable) Selection
sport
school
Stage 3: Progressive Query Execution
SPARQL Query Construction LLM filtering + SBERT filtering

Answer: Constraint Corrected Main Path

Chicago Execute Relaxation actor roster team
on KG ?c ?y ?x
Bulls Success Failure Strategy

Figure 1: Overview of the RouterKGQA framework.

Beam search over relation paths. To map the sentence-BERT (SBERT; Reimers and Gurevych,
above subtasks to a relation path, we search over 2019) and keep the top-x relations:
relation sequences rather than joint entity–relation
e i,j,k = Topx Rnext ; sim(tk , r) ,


states. Let R i,j (9)

where Topx (S; f (r)) denotes the subset of S con-

Bi−1 = {Ri−1,1 , . . . , Ri−1,w } (5)
taining the x elements with the largest values of
denote the beam at depth i − 1, where w is the f (r), where r ∈ S. sim(tk , r) is the cosine simi-
beam width and larity between the SBERT embeddings of tk and
r.
Ri−1,j = ⟨r1,j , r2,j , . . . , ri−1,j ⟩ (6) We then expand each partial path with the fil-
tered relations and obtain the candidate set
is the j-th partial relation path. w
[
For each partial path Ri−1,j , we first compute P
ei = { Ri−1,j ⊕ r | ∃k, r ∈ R
e i,j,k }, (10)
the set of entities reachable from the topic entity j=1

s1 :
where ⊕ appends a relation to a path.
Ei−1,j = Reach(s1 , Ri−1,j ), (7)
Path selection. To further reduce the search
where Reach(s1 , R) denotes the set of entities space, we score each expanded path against the
reached by following the relation sequence R from full question q and retain the top-y candidates:
s1 on G. For initialization, E0,1 = {s1 }. We then
enumerate all outgoing relations from these reach- P
bi = Topy (P
ei ; sim(q, R)), (11)
able entities:
where each path R is linearized by concatenating
Rnext = { r | ∃e ∈ Ei−1,j , ∃o, (e, r, o) ∈ G }. its relation names.
i,j
(8) Finally, the general model further reduces the
candidate set by selecting the best w paths from
Relation selection. We first perform coarse filter- Pbi , forming the next beam Bi . Each search step
ing to retain relations that are most relevant to the therefore requires only one LLM call, while the
blueprint. Because blueprint steps and KG relations remaining filtering is handled by small SBERT at
are not always perfectly aligned, we score each negligible cost. Including the initial blueprint gen-
candidate relation against every blueprint tk using eration, the total number of LLM calls is at most

4
d+1, where d is the maximum search depth. More- 4 Experiments
over, because every expansion is derived from KG-
4.1 Experimental Setup
reachable entities, all candidate paths in the beam
remain executable. Datasets. We evaluate on two widely used
KBQA benchmarks: WebQuestionsSP (We-
Stopping criterion. To determine when to stop bQSP) (Yih et al., 2016) and Complex WebQues-
the search, we use the hop count predicted by the tions (CWQ) (Talmor and Berant, 2018), using
specialized model as the maximum search depth Freebase (Bollacker et al., 2008) as the underlying
d. Even when the specialized model predicts an KG. Compared with WebQSP, CWQ generally con-
incorrect relation path, its depth prediction remains tains questions with greater hop depth and more
highly reliable: with Llama-2-7B, the predicted complex constraints, and is therefore substantially
depth matches the ground-truth depth in 97.20% more challenging. More detailed dataset statistics
of cases on CWQ and 98.70% on WebQSP. After and descriptions are provided in Appendix C.1.
depth d, we reduce the beam size to 1 and return
the remaining path as the corrected main path R. Metrics and baselines. Following prior work,
we report Hits@1 and F1. Hits@1 measures the
3.4 Progressive Query Execution proportion of questions for which the predicted
In this section, we explain how the corrected CRP answer set overlaps with the gold answer set, while
is converted into an executable SPARQL query and F1 is the mean F1 score between the predicted
retrieve answers from the KG. We also describe the and gold answer sets. We compare against three
progressive constraint relaxation strategy for im- categories: (1) LLM-only methods; (2) IR-based
proving robustness when full execution fails (Stage methods using subgraph retrieval; and (3) LLM-
3). based methods, including agent-based and retrieval-
based approaches. Detailed metric definitions are
Execution with full constraints. After correc- provided in Appendix C.2.
tion, we deterministically convert the CRP into
Configuration of RouterKGQA. RouterKGQA
an executable SPARQL query. The main path
uses a LoRA-tuned Llama-2-7B (Touvron et al.,
R = ⟨r1 , . . . , rN ⟩ is mapped to a chain of triple
2023) or Llama-3.1-8B (Team, 2024) special-
patterns rooted at the topic entity s1 , with each
ized model for CRP generation, and GPT-4o-mini
ri corresponding to one hop. Each constraint
(T =0) for path repair. The specialized model is
C = (i, ric , oci ) is then translated into an addi-
trained on (question, gold CRP) pairs derived from
tional condition: entity constraints are converted
SPARQL annotations. Unless otherwise noted, we
into triple patterns requiring the intermediate entity
use the same optimization settings for the special-
oi to connect to oci via ric , numeric constraints are
ized model, including a learning rate of 5 × 10−5 ,
converted into FILTER clauses imposing inequal-
cosine scheduler, batch size 4, gradient accumula-
ity conditions on numeric or temporal attributes,
tion 4, and bf16. For Stage 2, we set the relation
and string constraints, when present, are converted
filter size to x=4, the path filter size to y=10, and
into literal-matching FILTER clauses. Executing
the beam width to w=3, and use SBERT (nomic-
the resulting query yields the answer set
embed-text-v1; Nussbaum et al., 2025) for seman-
A ← Execute(ConvertCRP(CRP), G). tic matching. In Stage 3, constraints are relaxed in
the order string → numeric → entity. All experi-
If execution succeeds, we return A directly. Details ments use seed 42. Additional implementation and
on constructing gold CRPs from SPARQL annota- reproduction details are provided in Appendix D.
tions for training are provided in Appendix B. 4.2 Main Results
Constraint relaxation. If execution with all con- RouterKGQA improves F1 while preserving
straints fails, e.g., due to noisy constraint predic- strong Hits@1. Table 1 shows that RouterKGQA
tions or KG incompleteness, we progressively re- consistently improves F1 without sacrificing
lax lower-priority constraints while keeping the Hits@1, indicating that constraint-aware execution
corrected main path fixed, and re-execute the sim- effectively filters spurious candidates. Compared
plified query. This strategy improves robustness with the strongest prior F1 baseline, RouterKGQA
without discarding the core reasoning chain. improves F1 by 4.30 points on WebQSP and 2.84

5
 Category Method WebQSP CWQ
Hits@1 F1 Hits@1 F1
LLM-only Llama-2-7B (Touvron et al., 2023) 53.87 35.67 22.74 18.11
Llama-3.1-8B (Team, 2024) 55.96 31.61 30.81 21.83
GPT-4o-mini 64.86 48.59 40.89 40.89
IR-based KV-Mem (Miller et al., 2016) 46.70 34.50 21.10 15.70
PullNet (Sun et al., 2019) 68.10 – 47.20 –
EmbedKGQA (Saxena et al., 2020) 66.60 – 44.70 –
NSM+h (He et al., 2021) 74.30 67.40 48.80 44.00
TransferNet (Shi et al., 2021) 71.40 – 48.60 –
Subgraph Retrieval (Sun et al., 2020) 69.50 64.10 50.20 47.10
LLM-based ToG (GPT-4o-mini) (Sun et al., 2024) 77.20 50.04 60.67 41.90
PoG (GPT-4o-mini) (Chen et al., 2024) 83.13 61.15 64.67 55.56
InteractiveKBQA (GPT-4) (Xiong et al., 2024) 72.47 – 59.17 –
EffiQA (Dong et al., 2025) 82.90 – 69.50 –
RoG (Llama-2-7B) (Luo et al., 2024b) 86.18 70.24 61.71 54.98
ChatKBQA (Llama-2-7B) (Luo et al., 2024a) 86.33 82.13 85.16 80.94
MemQ (Xu et al., 2025) 88.74 77.90 88.16 75.56
GNN-RAG (Mavromatis and Karypis, 2025) 90.73 73.49 68.65 60.45
GCR (Llama-3.1-8B + GPT-4o-mini) (Luo et al., 2025) 91.68 75.27 74.64 62.94
Ours RouterKGQA (Llama-2-7B + GPT-4o-mini) 90.73 86.43 88.96 83.72
RouterKGQA (Llama-3.1-8B + GPT-4o-mini) 91.15 86.38 89.66 83.78

Table 1: Performance comparison on WebQSP and CWQ (%). Results for methods without public code are taken
from the original papers; all other results are reproduced in our environment using the same Freebase snapshot. See
Appendix C.2 for unified metric definitions and Appendix D for reproduction settings and implementation details.

points on CWQ. At the same time, its Hits@1 re- 4.3 Ablation Study
mains comparable to the strongest prior baseline on
We examine the effects of model routing, stopping
WebQSP and surpasses it on CWQ, showing that
criteria, constraint modeling, and representation
the F1 gains come from better candidate filtering
design.
while preserving the correct answer in most cases.

Agent-based methods are often more effec- Agent-based repair brings marginal gains. Ta-
tive, but substantially more expensive. Table 2 ble 3 shows that Stage 1 is the primary source of
shows that representative agent-based methods, performance, while Stage 2 provides reliable addi-
such as ToG and PoG, outperform simple LLM- tional improvements. Removing Stage 2 leads to
only baselines on average, reflecting the benefit of smaller drops than stage 1 on both benchmarks, but
explicit multi-step KG exploration. However, this the drop is consistent. This shows that agent-based
gain comes at a substantial efficiency cost: com- path repair delivers meaningful marginal gains by
pared with retrieval-based methods, agent-based correcting unreachable relation paths that the spe-
methods require 5.80× more LLM calls and incur cialized model fails to recover.
26.67× higher cost on average.
Depth prediction is robust. In the main experi-
Our agent-based path repair is much cheaper, ments, we use the hop count predicted by the spe-
and model routing reduces the cost further. cialized model as the stopping criterion. We com-
RouterKGQA (Stage 2+3), i.e., the agent-based- pare it with two alternative variants as stopping
only configuration, already outperforms represen- criterion. Using blueprint step count consistently
tative agent-based baselines while reducing cost by underperforms, indicating that semantic subtasks
3.14× relative to PoG and 7.53× relative to ToG. do not align perfectly with KG hop structure. We
Adding model routing further improves the trade- further let the general LLM self-judge when to
off: compared with the agent-based-only setting, stop. This variant performs closer to the default
RouterKGQA (Full) reduces cost by 4.51× and setting, but is still slightly worse and requires an
LLM calls by 2.88×, while improving F1 by 15.70 additional LLM call. These results support using
points. This shows that routing between retrieval- the specialized model to control the depth of agent-
based generation and agent-based repair is more based search, achieving the best performance with
effective than relying on either strategy alone. minimal cost.

6
 Avg. # Avg. # Avg. Cost
Category Method Hits@1 F1
LLM Calls Tokens (USD) / 10K Qs
GNN-RAG 90.73 73.49 2.00 778.17 0.56
RoG 86.18 70.24 2.00 592.55 0.45
Retrieval-based ChatKBQA 86.33 82.13 1.00 208.08 0.46
GCR 91.68 75.27 2.00 725.57 0.89
MemQ 88.74 77.90 1.00 175.62 0.21
ToG 77.20 50.04 10.91 8,426.59 19.35
Agent-based
PoG 83.13 61.15 7.64 4,704.47 8.07
RouterKGQA (Stage 2+3) 86.07 70.68 3.31 1,353.78 2.57
Ours
RouterKGQA (Full) 91.15 86.38 1.15 612.68 0.57

Table 2: Efficiency and performance comparison on WebQSP (%). RouterKGQA (Full) achieves the best F1 at a cost
comparable to retrieval-based methods, while RouterKGQA (Stage 2+3), i.e., the agent-based-only configuration,
remains substantially cheaper than representative agent-based baselines. Costs are estimated using Llama-2-7B
pricing ($0.05/1M input, $0.25/1M output) and Llama-3.1-8B pricing ($0.10/1M input, $0.10/1M output) from
Artificial Analysis, and official GPT-4o-mini pricing ($0.15/1M input, $0.60/1M output) from OpenAI, all accessed
2026-02-14.

Variants WebQSP CWQ Variants WebQSP CWQ

Hits@1 F1 Hits@1 F1 Hits@1 F1 Hits@1 F1
RouterKGQA 91.15 86.38 89.66 83.78 GCR (w/o LLM Filter) 92.13 57.00 74.99 49.65
GCR 91.68 75.27 74.64 62.94
Stage-level ablations
GCR (w/ RouterKGQA Filter) 87.13 83.95 80.46 68.05
w/o Stage 1 (Stage 2+3) 86.07 70.68 70.67 59.30
w/o Stage 2 (Stage 1+3) 89.02 84.68 88.05 82.12 RouterKGQA 91.15 86.38 89.66 83.78
Stopping criterion variants
w/ blueprint step count 88.47 83.96 88.59 82.81 Table 4: Reference comparison with GCR variants (%).
w/ LLM self-judge 90.76 86.11 89.44 83.40 RouterKGQA is included for reference.
Constraint ablations (on Stage 1+3)
w/o Entity Constraint 88.65 78.21 87.20 58.32
w/o Numeric Constraint 89.62 81.66 88.67 71.15 Dataset Metric ChatKBQA RouterKGQA
w/o String Constraint 89.02 84.58 87.96 80.72 Skeleton Accuracy 82.43 85.78
w/o All Constraints 89.45 75.56 88.28 46.60 WebQSP
Exact Match 62.42 69.68
Skeleton Accuracy 74.00 75.62
Table 3: Ablation study of RouterKGQA (%). Con- CWQ
Exact Match 54.23 54.63
straint ablations are conducted on Stage 1+3. Stage 1:
retrieval-based CRP generation; Stage 2: agent-based Table 5: First-prediction generation accuracy (%) of
KG-guided repair; Stage 3: progressive query execu- ChatKBQA (S-expression) and RouterKGQA (CRP),
tion. both using Llama-2-7B.

Constraints mainly improve precision. Within ficient. Besides, even the best GCR variant remains
Stage 1+3, removing all constraints leaves Hits@1 clearly below RouterKGQA, suggesting that our
nearly unchanged but substantially reduces F1. explicit CRP formulation is more effective than
This shows that constraints primarily improve pre- post-hoc filtering over monolithic outputs.
cision by filtering spurious candidates rather than
changing top-1 reachability. Among the constraint CRPs are easier to generate. We further com-
types, entity constraints contribute the most, fol- pare whether CRP generation offers intrinsic ad-
lowed by numeric constraints. vantages over the S-expression format, e.g., ChatK-
To further verify the effectiveness of our con- BQA (Luo et al., 2024a). Using the same back-
straints, we apply them to the baseline GCR, with bone (Llama-2-7B), we measure: (1) skeleton ac-
results shown in Table 4. The results show that re- curacy—whether the predicted structure matches
moving GCR filtering yields relatively high Hits@1 the gold, ignoring entity/literal values; and (2) ex-
but much lower F1, while adding our generated act match—whether the prediction is fully correct.
constraint filtering further strengthens this effect, As shown in Table 5, CRP achieves higher skele-
especially on the harder CWQ benchmark. This ton accuracy on both benchmarks (+3.35% on We-
suggests that its filtering mechanism remains insuf- bQSP, +1.62% on CWQ), indicating that the flat

7
 Dimension Category N Full S1+S3 ∆F1 GCR WebQSP CWQ
Role Model
Conjunction 1575 85.95 83.18 +2.77 67.52 Hits@1 F1 Hits@1 F1
Composition Composition 1546 82.34 81.91 +0.43 62.39
Qwen2.5-7B 88.04 84.67 88.19 81.32
Type Comparative 213 82.02 80.88 +1.14 49.62 Specialized Qwen3-8B 88.04 84.80 89.32 83.32
Superlative 197 79.70 76.53 +3.17 45.05
(Stage 1) Llama-2-7B 90.73 86.43 88.96 83.72
0 1176 85.26 84.70 +0.56 65.33 Llama-3.1-8B 91.15 86.38 89.66 83.78
Constraint 1 1511 84.90 82.94 +1.96 61.03
Complexity 2 Qwen-2.5-7B 90.79 86.25 89.49 83.63
633 78.72 75.76 +2.96 63.73
Llama-2-7B 89.63 85.30 88.25 82.67
3+ 211 82.76 80.88 +1.88 61.04 General
(Stage 2) Llama-3.1-8B 89.57 86.90 89.47 83.35
ChatGPT 89.38 85.20 89.21 83.49
Table 6: CWQ performance by query complexity (F1, GPT-4o-mini 91.15 86.38 89.66 83.78
%). ∆F1 = Full − Stage 1+3 (i.e., the contribution of
Stage 2 correction). GCR results are included for cross- Table 7: Impact of model choice in RouterKGQA’s
method comparison. specialized and general roles.

triple-sequence format is structurally easier for a 4.5 Specialized–General Model Analysis

7B model to learn than nested S-expressions with Table 7 examines the impact of model choice in
function keywords. The advantage is more pro- each role. When varying the specialized model (fix-
nounced on exact match (+7.26% on WebQSP), ing GPT-4o-mini as the general model), stronger
confirming that CRP’s explicit constraint separa- models produce substantially better results: Llama-
tion also improves value-level prediction. 3.1-8B outperforms Qwen2.5-7B by +3.11 Hits@1
on WebQSP and +2.46 F1 on CWQ. This confirms
4.4 Analysis by Query Complexity that CRP generation quality—which benefits from
fine-tuning on KG-structured data—is the primary
Table 6 jointly analyzes RouterKGQA’s perfor-
driver of end-to-end performance.
mance across two complexity dimensions on CWQ.
When varying the general model (fixing Llama-
3.1-8B as specialized), the gap between models is
Advantage over GCR across composition types.
notably smaller. Even Llama-2-7B as the general
RouterKGQA achieves consistent and large im-
achieves 89.63% Hits@1 on WebQSP, a gap of only
provements over GCR across all four CWQ compo-
1.52 points with GPT-4o-mini. Among stronger
sition types. The advantage is most pronounced on
generals, performance differences are within 1.8
comparative (+32.40 F1) and superlative (+34.65
points. This is because RouterKGQA’s correction
F1) queries—precisely the types that demand ex-
task is relatively simple—selecting the best path
plicit numeric and ordering constraints, where
from a small candidate set—rather than generating
GCR’s constraint-free path generation is most in-
answers from scratch, making RouterKGQA de-
adequate. Even on simpler conjunction and com-
ployable with fully open-source models at minimal
position types, RouterKGQA maintains substantial
performance loss. Appendix E further shows that
margins (+18.43 and +19.95 F1), demonstrating
RouterKGQA yields reasoning paths that are more
broad improvements across all query structures.
semantically consistent with the original question.
Stage 2 contribution amplified by constraints.
5 Conclusion
Stage 2’s correction contributes most on superla-
tive queries (∆F1 = +3.17), where precise paths We proposed RouterKGQA, a framework for
are essential for executing ordering constraints, and knowledge graph question answering based on
on conjunction queries (∆F1 = +2.77), which in- specialized–general model routing, which com-
volve multiple joined conditions. The breakdown bines efficient specialized generation with general
by constraint complexity reveals a clear trend: the model repair. We further improve both components:
gain is largest on questions of moderate difficulty, for the specialized model, we introduce Constraint-
i.e., those with 2 constraints. For overly simple aware Reasoning Paths for deterministic answer
questions, the specialized model already handles filtering. For the general workflow, we design an
them well, while for overly difficult questions, the efficient KG-guided repair procedure that reduces
repair success rate also decreases. This analysis the cost of multi-round exploration. Experiments
further provides insight into when Stage 2 repair is on WebQSP and CWQ show that RouterKGQA
most effective. achieves strong performance gains, while requiring

8
only 1.15 average LLM calls per question. Fur- Papers), NAACL 2024, Mexico City, Mexico, June
ther analysis shows that the repair stage provides 16-21, 2024, pages 3947–3960, Mexico City, Mexico.
Association for Computational Linguistics.
consistent gains at low cost, and that the overall
framework remains effective across different query Kurt Bollacker, Colin Evans, Praveen K. Paritosh, Tim
types and model choices. Sturge, and Jamie Taylor. 2008. Freebase: a collabo-
ratively created graph database for structuring human
Limitations knowledge. In Proceedings of the ACM SIGMOD
International Conference on Management of Data,
RouterKGQA has several limitations. First, the SIGMOD 2008, Vancouver, BC, Canada, June 10-12,
2008, pages 1247–1250. ACM.
current CRP representation supports only flat en-
tity, numeric, and string constraints. This keeps Liyi Chen, Panrong Tong, Zhongming Jin, Ying Sun,
the representation simple and easier for specialized Jieping Ye, and Hui Xiong. 2024. Plan-on-graph:
models to learn, but does not explicitly capture Self-correcting adaptive planning of large language
model on knowledge graphs. In Advances in Neural
more nested or compositional constraint structures. Information Processing Systems 38: Annual Confer-
Second, the current routing decision uses main- ence on Neural Information Processing Systems 2024,
path reachability as a simple and efficient signal. NeurIPS 2024, Vancouver, BC, Canada, December
However, a reachable path may still be semantically 10 - 15, 2024.
incorrect. Incorporating finer-grained signals, such Zixuan Dong, Baoyun Peng, Yufei Wang, Jia Fu, Xi-
as generation confidence or semantic consistency, aodong Wang, Xin Zhou, Yongxue Shan, Kangchen
could further improve robustness. Third, like other Zhu, and Weiguo Chen. 2025. Effiqa: Efficient
KGQA pipelines, our framework benefits from ac- question-answering with strategic multi-model col-
laboration on knowledge graphs. In Proceedings
curate entity linking and a sufficiently complete of the 31st International Conference on Computa-
underlying KG; errors in topic-entity disambigua- tional Linguistics, COLING 2025, Abu Dhabi, UAE,
tion or missing relations or literals may still affect January 19-24, 2025, pages 7180–7194, Abu Dhabi,
both correction and final answer filtering. Fourth, UAE. Association for Computational Linguistics.
our experiments are limited to English benchmarks Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia,
on Freebase (WebQSP and CWQ). Although the Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo,
framework is general, its transferability to other Meng Wang, et al. 2023. Retrieval-augmented gen-
knowledge graphs, domains, or multilingual set- eration for large language models: A survey. CoRR,
abs/2312.10997.
tings remains to be validated. Finally, although pro-
gressive constraint relaxation improves robustness Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and
when full query execution fails, it may occasion- Ji-Rong Wen. 2021. Improving multi-hop knowledge
ally return broader answer sets than desired when base question answering by learning intermediate
supervision signals. In WSDM ’21, The Fourteenth
important constraints are removed. ACM International Conference on Web Search and
Data Mining, Virtual Event, Israel, March 8-12, 2021,
Ethical considerations pages 553–561. ACM.

Our work adheres to the ACL Ethics Policy and Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan
uses publicly available datasets for reproducibility. Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and
Weizhu Chen. 2022. Lora: Low-rank adaptation of
LLMs may exhibit racial and gender biases, so we
large language models. In The Tenth International
strongly recommend users assess potential biases Conference on Learning Representations, ICLR 2022,
before applying the models in specific contexts. Virtual Event, April 25-29, 2022. [Link].
Additionally, due to the difficulty of controlling
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong,
LLM outputs, users should be cautious of issues Zhangyin Feng, Haotian Wang, Qianglong Chen,
arising from hallucinations. Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2025.
A survey on hallucination in large language models:
Principles, taxonomy, challenges, and open questions.
References ACM Trans. Inf. Syst., 43(2):42:1–42:55.

Garima Agrawal, Tharindu Kumarage, Zeyad Alghamdi, Yixin Ji, Kaixin Wu, Juntao Li, Wei Chen, Mingjie
and Huan Liu. 2024. Can knowledge graphs reduce Zhong, Jia Xu, and Min Zhang. 2024. Retrieval and
hallucinations in llms? : A survey. In Proceedings reasoning on kgs: Integrate knowledge graphs into
of the 2024 Conference of the North American Chap- large language models for complex question answer-
ter of the Association for Computational Linguistics: ing. In Findings of the Association for Computa-
Human Language Technologies (Volume 1: Long tional Linguistics: EMNLP 2024, Miami, Florida,

9
 USA, November 12-16, 2024, volume EMNLP 2024 Jie Ma, Zhitao Gao, Qi Chai, Wangchun Sun, Pinghui
of Findings of ACL, pages 7598–7610, Miami, Wang, Hongbin Pei, Jing Tao, Lingyun Song, Jun
Florida, USA. Association for Computational Lin- Liu, Chen Zhang, et al. 2025. Debate on graph: A
guistics. flexible and reliable reasoning framework for large
language models. In AAAI-25, Sponsored by the
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Association for the Advancement of Artificial Intelli-
Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea gence, February 25 - March 4, 2025, Philadelphia,
Madotto, and Pascale Fung. 2023. Survey of halluci- PA, USA, pages 24768–24776. AAAI Press.
nation in natural language generation. ACM Comput.
Surv., 55(12):248:1–248:38. Costas Mavromatis and George Karypis. 2025. GNN-
RAG: graph neural retrieval for efficient large lan-
Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Xin guage model reasoning on knowledge graphs. In
Zhao, and Ji-Rong Wen. 2023. Structgpt: A general Findings of the Association for Computational Lin-
framework for large language model to reason over guistics, ACL 2025, Vienna, Austria, July 27 - August
structured data. In Proceedings of the 2023 Con- 1, 2025, volume ACL 2025 of Findings of ACL, pages
ference on Empirical Methods in Natural Language 16682–16699, Vienna, Austria. Association for Com-
Processing, EMNLP 2023, Singapore, December 6- putational Linguistics.
10, 2023, pages 9237–9251, Singapore. Association
for Computational Linguistics. Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir-
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Hossein Karimi, Antoine Bordes, and Jason Weston.
Petroni, Vladimir Karpukhin, Naman Goyal, Hein- 2016. Key-value memory networks for directly read-
rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- ing documents. In Proceedings of the 2016 Confer-
täschel, et al. 2020. Retrieval-augmented generation ence on Empirical Methods in Natural Language Pro-
for knowledge-intensive NLP tasks. In Advances cessing, EMNLP 2016, Austin, Texas, USA, Novem-
in Neural Information Processing Systems 33: An- ber 1-4, 2016, pages 1400–1409, Austin, Texas. The
nual Conference on Neural Information Processing Association for Computational Linguistics.
Systems 2020, NeurIPS 2020, December 6-12, 2020,
Zach Nussbaum, John Xavier Morris, Andriy Mulyar,
virtual, pages 9459–9474.
and Brandon Duderstadt. 2025. Nomic embed: Train-
Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar ing a reproducible long context text embedder. Trans.
Mehdad, and Wen-tau Yih. 2020. Efficient one-pass Mach. Learn. Res., 2025.
end-to-end entity linking for questions. In Proceed-
ings of the 2020 Conference on Empirical Methods in OpenAI. 2023. GPT-4 technical report. CoRR,
Natural Language Processing, EMNLP 2020, Online, abs/2303.08774.
November 16-20, 2020, pages 6433–6441, Online.
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Ji-
Association for Computational Linguistics.
apu Wang, and Xindong Wu. 2024. Unifying large
Haoran Luo, Haihong E, Zichen Tang, Shiyao Peng, language models and knowledge graphs: A roadmap.
Yikai Guo, Wentai Zhang, Chenghao Ma, Guanting IEEE Trans. Knowl. Data Eng., 36(7):3580–3599.
Dong, Meina Song, Wei Lin, et al. 2024a. Chatk-
bqa: A generate-then-retrieve framework for knowl- Nils Reimers and Iryna Gurevych. 2019. Sentence-bert:
edge base question answering with fine-tuned large Sentence embeddings using siamese bert-networks.
language models. In Findings of the Association In Proceedings of the 2019 Conference on Empiri-
for Computational Linguistics, ACL 2024, Bangkok, cal Methods in Natural Language Processing and
Thailand and virtual meeting, August 11-16, 2024, the 9th International Joint Conference on Natural
volume ACL 2024 of Findings of ACL, pages 2039– Language Processing, EMNLP-IJCNLP 2019, Hong
2056, Bangkok, Thailand. Association for Computa- Kong, China, November 3-7, 2019, pages 3980–3990,
tional Linguistics. Hong Kong, China. Association for Computational
Linguistics.
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and
Shirui Pan. 2024b. Reasoning on graphs: Faithful Apoorv Saxena, Aditay Tripathi, and Partha P. Talukdar.
and interpretable large language model reasoning. In 2020. Improving multi-hop question answering over
The Twelfth International Conference on Learning knowledge graphs using knowledge base embeddings.
Representations, ICLR 2024, Vienna, Austria, May In Proceedings of the 58th Annual Meeting of the As-
7-11, 2024. [Link]. sociation for Computational Linguistics, ACL 2020,
Online, July 5-10, 2020, pages 4498–4507, Online.
Linhao Luo, Zicheng Zhao, Gholamreza Haffari, Yuan- Association for Computational Linguistics.
Fang Li, Chen Gong, and Shirui Pan. 2025. Graph-
constrained reasoning: Faithful reasoning on knowl- Jiaxin Shi, Shulin Cao, Lei Hou, Juanzi Li, and Han-
edge graphs with large language models. In Forty- wang Zhang. 2021. Transfernet: An effective and
second International Conference on Machine Learn- transparent framework for multi-hop question an-
ing, ICML 2025, Vancouver, BC, Canada, July 13- swering over relation graph. In Proceedings of the
19, 2025, volume 267 of Proceedings of Machine 2021 Conference on Empirical Methods in Natural
Learning Research, pages 41540–41565. PMLR / Language Processing, EMNLP 2021, Virtual Event
[Link]. / Punta Cana, Dominican Republic, 7-11 November,

10
 2021, pages 4149–4158, Online and Punta Cana, Do- Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti
minican Republic. Association for Computational Bhosale, et al. 2023. Llama 2: Open foundation and
Linguistics. fine-tuned chat models. CoRR, abs/2307.09288.
Yuan Sui, Yufei He, Nian Liu, Xiaoxin He, Kun Wang, Robin Wagner, Emanuel Kitzelmann, and Ingo Boersch.
and Bryan Hooi. 2025. Fidelis: Faithful reasoning 2025. Mitigating hallucination by integrating knowl-
in large language models for knowledge graph ques- edge graphs into LLM inference - a systematic litera-
tion answering. In Findings of the Association for ture review. In Proceedings of the 63rd Annual Meet-
Computational Linguistics, ACL 2025, Vienna, Aus- ing of the Association for Computational Linguistics
tria, July 27 - August 1, 2025, volume ACL 2025 of (Volume 4: Student Research Workshop), ACL 2025,
Findings of ACL, pages 8315–8330, Vienna, Austria. Vienna, Austria, July 27 - August 1, 2025, pages 795–
Association for Computational Linguistics. 805, Vienna, Austria. Association for Computational
Linguistics.
Haitian Sun, Andrew O. Arnold, Tania Bedrax-Weiss,
Fernando Pereira, and William W. Cohen. 2020.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V.
Faithful embeddings for knowledge base queries. In
Le, Ed H. Chi, Sharan Narang, Aakanksha Chowd-
Advances in Neural Information Processing Systems
hery, and Denny Zhou. 2023. Self-consistency
33: Annual Conference on Neural Information Pro-
improves chain of thought reasoning in language
cessing Systems 2020, NeurIPS 2020, December 6-
models. In The Eleventh International Conference
12, 2020, virtual.
on Learning Representations, ICLR 2023, Kigali,
Haitian Sun, Tania Bedrax-Weiss, and William W. Co- Rwanda, May 1-5, 2023. [Link].
hen. 2019. Pullnet: Open domain question answering
with iterative retrieval on knowledge bases and text. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
In Proceedings of the 2019 Conference on Empiri- Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le,
cal Methods in Natural Language Processing and and Denny Zhou. 2022. Chain-of-thought prompt-
the 9th International Joint Conference on Natural ing elicits reasoning in large language models. In
Language Processing, EMNLP-IJCNLP 2019, Hong Advances in Neural Information Processing Systems
Kong, China, November 3-7, 2019, pages 2380–2390, 35: Annual Conference on Neural Information Pro-
Hong Kong, China. Association for Computational cessing Systems 2022, NeurIPS 2022, New Orleans,
Linguistics. LA, USA, November 28 - December 9, 2022, pages
24824–24837.
Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo
Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, Heung- Guanming Xiong, Junwei Bao, and Wen Zhao. 2024.
Yeung Shum, and Jian Guo. 2024. Think-on-graph: Interactive-kbqa: Multi-turn interactions for knowl-
Deep and responsible reasoning of large language edge base question answering with large language
model on knowledge graph. In The Twelfth Inter- models. In Proceedings of the 62nd Annual Meeting
national Conference on Learning Representations, of the Association for Computational Linguistics (Vol-
ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- ume 1: Long Papers), ACL 2024, Bangkok, Thailand,
[Link]. August 11-16, 2024, pages 10561–10582, Bangkok,
Thailand. Association for Computational Linguistics.
Alon Talmor and Jonathan Berant. 2018. The web as
a knowledge-base for answering complex questions. Mufan Xu, Gewen Liang, Kehai Chen, Wei Wang, Xun
In Proceedings of the 2018 Conference of the North Zhou, Muyun Yang, Tiejun Zhao, and Min Zhang.
American Chapter of the Association for Computa- 2025. Memory-augmented query reconstruction for
tional Linguistics: Human Language Technologies, llm-based knowledge graph reasoning. In Findings of
NAACL-HLT 2018, New Orleans, Louisiana, USA, the Association for Computational Linguistics, ACL
June 1-6, 2018, Volume 1 (Long Papers), pages 641– 2025, Vienna, Austria, July 27 - August 1, 2025, vol-
651, New Orleans, Louisiana. Association for Com- ume ACL 2025 of Findings of ACL, pages 24068–
putational Linguistics. 24084, Vienna, Austria. Association for Computa-
tional Linguistics.
Llama Team. 2024. The llama 3 herd of models. CoRR,
abs/2407.21783.
Xi Ye and Greg Durrett. 2022. The unreliability of ex-
Yuhang Tian, Dandan Song, Zhijing Wu, Pan Yang, planations in few-shot prompting for textual reason-
Changzhi Zhou, Jun Yang, Hao Wang, Huipeng Ma, ing. In Advances in Neural Information Processing
Chenhao Li, and Luan Zhang. 2025. Compkbqa: Systems 35: Annual Conference on Neural Informa-
Component-wise task decomposition for knowledge tion Processing Systems 2022, NeurIPS 2022, New
base question answering. In Proceedings of the 2025 Orleans, LA, USA, November 28 - December 9, 2022,
Conference on Empirical Methods in Natural Lan- pages 30378–30392.
guage Processing, EMNLP 2025, Suzhou, China,
November 4-9, 2025, pages 293–309, Suzhou, China. Wen-tau Yih, Matthew Richardson, Christopher Meek,
Association for Computational Linguistics. Ming-Wei Chang, and Jina Suh. 2016. The value of
semantic parse labeling for knowledge base question
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- answering. In Proceedings of the 54th Annual Meet-
bert, Amjad Almahairi, Yasmine Babaei, Nikolay ing of the Association for Computational Linguistics,

11
 ACL 2016, August 7-12, 2016, Berlin, Germany, Vol-
Here is the question.
ume 2: Short Papers, pages 201–206, Berlin, Ger- Q: <question>
many. The Association for Computer Linguistics. A:

A Prompt Templates A.3 Stage 2: Path Selection

This appendix provides the full prompt templates At each beam search step, after SBERT-based
used in each stage of RouterKGQA. Placeholders coarse filtering, the general LLM selects the most
are shown in <angle brackets>. promising paths from the candidate set. The same
prompt template is used for both intermediate steps
A.1 Stage 1: CRP Generation (specialized) (selecting up to w paths) and the final step (select-
The specialized model is fine-tuned on (question, ing 1 path), with only the value of n varying:
gold CRP) pairs using the following instruction
Path Selection Prompt
format:
Q: ‘<question>’
CRP Generation Prompt Paths from ‘<start_entity_names>’:
Path 1: <entity> -> <relation_1> -> . . . ->
Instruction: Generate a reasoning path that retrieves <end_entity>. . .
the information corresponding to the given question. Path 2: <entity> -> <relation_1> -> . . . ->
Input: Question: { <question> } <end_entity>. . .
Output: <structured CRP> ...
Select up to <n> paths most likely to reach the FINAL
answer (not intermediate entities). Reply with path
A.2 Stage 2: Reasoning Blueprint Generation numbers only, e.g.: Path 1, Path 2
(general)
During intermediate beam search steps, n is set
When the specialized-generated main path is un-
to the beam width w; at the final step (maximum
reachable on the KG, the general LLM decomposes
depth reached), n=1 to select a single best path.
the question into a sequence of single-hop reason-
ing steps via the following few-shot prompt: B SPARQL-to-CRP Conversion
Reasoning Blueprint Prompt Gold CRPs used for specialized training are deter-
ministically derived from SPARQL annotations via
Given a question, generate the reasoning steps with-
out providing the final answer or specific entities. the following rule-based procedure.
Q: What countries are located in Eastern Europe and
have the country calling code +373? Step 1: Subgraph construction and main path
A: #1 Identify the countries located in Eastern Eu- extraction. We construct a directed graph from
rope. all triple patterns in the WHERE clause, where each
#2 Determine which of these countries has the coun-
try calling code +373. variable or named entity is a node and each triple
Q: What country that trades with Turkey has an ISO pattern (s, r, o) forms a directed edge. The topic
numeric code lower than 012? entity s1 is identified as the Freebase MID that lies
A: #1 Identify the countries that are trade partners of
Turkey. on the path to the SELECT target variable oN . We
#2 Determine which of these countries has an ISO then extract the path from s1 to oN in this graph;
numeric code lower than 012.
Q: Who were the inspirations for the author of This
the relation sequence along this path forms the
Side of Paradise? main path R = ⟨r1 , . . . , rN ⟩. Since constraint
A: #1 Identify the author of This Side of Paradise. branches attach to intermediate variables but do not
#2 Determine the individuals or works that influenced
the author. connect to oN , this path is unique in the constructed
Q: What countries border the location where the film graph.
Amen is set?
A: #1 Identify the location where the film Amen is Step 2: Constraint classification. All triple pat-
set. terns and FILTER clauses not on the main path are
#2 Determine the countries that border this location.
Q: Lou Seal is the mascot for the team that last won classified into three constraint types, each repre-
the World Series when? sented as C = (i, ric , oci ):
A: #1 Identify the team for which Lou Seal is the
mascot. • Entity restrictions. Starting from each inter-
#2 Determine the year this team last won the World
Series. mediate entity oi on the main path, we search
over the remaining subgraph. Any branch that

12
 terminates at a named KG entity (i.e., a Free- Aspect Category CWQ WebQSP
base MID not on the main path) yields an # % # %
entity restriction, where oci is the target entity. 1-hop 854 24.19 1,042 63.58
Depth 2-hop 1,859 52.65 592 36.12
• Numeric restrictions. Triples whose ob- 3-hop+ 818 23.17 5 0.31
ject is a numeric or temporal literal (e.g., 0 1,176 33.31 1,169 71.32
1 1,511 42.79 332 20.26
xsd:dateTime, xsd:float) yield a numeric Constraints
2 633 17.92 92 5.61
restriction, where oci encodes both the compar- 3+ 211 5.98 46 2.81
ison operator (=, ≥, ≤, >) and the threshold
value, extracted from the associated FILTER Table 8: Test-set distributions of question depth and
clause. ORDER BY clauses with LIMIT 1 are constraint complexity on WebQSP and CWQ.
also converted to numeric restrictions with oci
set to argmax or argmin. Query-based methods. These methods retrieve
answers by executing a structured query (e.g.,
• String restrictions. Triples whose object is a SPARQL) against the KG. The returned answer
string literal (optionally with a language tag) set is inherently unordered. ChatKBQA (Luo et al.,
yield a string restriction, where oci is the literal 2024a) counts Hits@1=1 if the answer set of the
string value. first executable predicted query intersects with the
gold answer set. In contrast, MemQ (Xu et al.,
Round-trip verification. We verify the fidelity 2025) considers only the first returned entity—
of this conversion by round-trip testing (SPARQL whose position is determined by the query engine
→ CRP → SPARQL): both directions use deter- and does not carry ranking semantics—and checks
ministic rules. whether it belongs to the gold answer set.

C Experimental Setup Details LLM-filtering methods. These methods use an

LLM to select or generate answers from candi-
C.1 Dataset Details date entities. Both RoG (Luo et al., 2024b) and
GCR (Luo et al., 2025) compute the metric identi-
We evaluate RouterKGQA on two standard
cally: checking whether the LLM-filtered answer
Freebase-based KBQA benchmarks: WebQues-
set has a non-empty intersection with the gold an-
tionsSP (WebQSP) (Yih et al., 2016) and Com-
swer set. However, RoG labels this metric Hits@1,
plex WebQuestions (CWQ) (Talmor and Berant,
while GCR refers to it as Hit, despite the underlying
2018). WebQSP contains 4,737 questions and is
computation being the same.
commonly used to evaluate KBQA systems on rel-
atively simpler semantic parsing and multi-hop rea- Unified evaluation protocol. To ensure a fair
soning cases. CWQ is substantially larger, with comparison across methods, we standardize the
34,689 questions, and is designed to emphasize computation as follows:
more complex compositional reasoning. Compared
with WebQSP, CWQ contains a much larger propor- • For query-based methods, Hits@1=1 if the
tion of deeper multi-hop questions and questions answer set returned by the first executable
with richer constraints, making it a more challeng- query has a non-empty intersection with the
ing benchmark for constraint-aware KGQA. Ta- gold answer set.
ble 8 reports the distributions of question depth and
constraint complexity on the test sets of the two • For LLM-filtering methods, Hits@1=1 if
benchmarks. the answer set generated by the first LLM rea-
soning step has a non-empty intersection with
C.2 Metric Definitions the gold answer set.

Prior KBQA methods adopt inconsistent defini- F1 is computed using standard set-level preci-
tions of Hits@1, which complicates fair compar- sion and recall between the predicted and gold
ison across approaches. We identify two answer- answer sets. This unified protocol ensures that
retrieval paradigms and describe how each has com- Hits@1 reflects answer correctness rather than
puted this metric. implementation-specific ordering effects.

13
D Implementation Details Category Method Reprod. Notes
Llama-2-7B ✓ Direct inference
D.1 Training Details LLM-only Llama-3.1-8B ✓ Direct inference
GPT-4o-mini ✓ Direct inference
The specialized CRP generation model is fine- KV-Mem ✗ Original paper
tuned using LLaMA-Factory1 with LoRA (Hu PullNet ✗ Original paper
et al., 2022). We train dataset-specific special- EmbedKGQA ✗ Original paper
IR-based
NSM+h ✗ Original paper
ized models for evaluation on WebQSP and CWQ. TransferNet ✗ Original paper
We apply the same training recipe to both Llama- Subgraph Retrieval ✗ Original paper
2-7B and Llama-3.1-8B backbones unless other- ToG ✓ GPT-4o-mini
PoG ✓ GPT-4o-mini
wise specified. The main optimization settings are InteractiveKBQA ✗ Original paper
shared across datasets, while the number of train- EffiQA ✗ Original paper
ing epochs differs between WebQSP and CWQ. LLM-based RoG ✓ Official code
ChatKBQA ✓ Modified†
Table 9 summarizes the main training hyperparam- MemQ ✓ Official code
eters. GNN-RAG ✓ Official code
GCR ✓ Official code
Hyperparameter WebQSP CWQ
Table 10: Baseline reproduction status. ✓: reproduced
Stage 1: CRP Generation (Specialized Model) in our environment; ✗: results from the original paper.
Fine-tuning Type LoRA LoRA †
Learning Rate 5e-5 5e-5
The original ChatKBQA uses Llama-2-13B on CWQ
LR Scheduler Cosine Cosine and Llama-2-7B on WebQSP; we unify both to Llama-
Train Batch Size 4 4 2-7B for fair comparison under the same model scale.
Gradient Accum. Steps 4 4
Effective Batch Size 16 16
Train Epochs 50 10 we reconstruct each predicted path into a natural-
Precision bf16 bf16
language question qr using an LLM and decom-
Stage 2: Path Repair (General Model)
General Model GPT-4o-mini GPT-4o-mini pose both qr and the original question q into sub-
Temperature T 0 0 question sets, denoted by A and B, respectively,
Beam Width w 3 3 following the decomposition style of CWQ (Tal-
Relation Filter Size x 4 4
Path Filter Size y 10 10 mor and Berant, 2018). Based on the set rela-
Semantic Matching nomic-embed-text-v1 tionship between A and B, we classify each ex-
Stage 3: Progressive Query Execution ample into four categories: EXACT_EQUAL
Relaxation Order string → numeric → entity (A = B), RECON_SUBSET (A ⊂ B), PAR-
TIAL_OVERLAP (A ∩ B ̸= ∅, with neither set
Table 9: Implementation details of RouterKGQA. containing the other), and DISJOINT (A∩B = ∅).
Stage 1 settings are shared across Llama-2-7B and
Figure 2(b) confirms that semantic alignment
Llama-3.1-8B unless otherwise noted.
directly determines downstream performance: EX-
ACT_EQUAL yields the best results across all
methods, while DISJOINT yields the worst. Fig-
D.2 Baseline Reproduction
ure 2(a) shows that RouterKGQA substantially in-
Table 10 summarizes the reproduction status of creases the proportion of EXACT_EQUAL pre-
all compared methods. For methods with publicly dictions (34.7% vs. 10.8% for GCR and 5.9% for
available code, we reproduce results in our environ- GNN-RAG) while reducing DISJOINT to 1.4%
ment using the same Freebase snapshot to ensure (vs. 24.9% and 46.0%). This demonstrates that
fair comparison. All reproduced methods follow RouterKGQA generates reasoning paths that are
the original paper’s settings unless otherwise noted. semantically equivalent or highly consistent with
For methods without public code, we report results the original question intent.
directly from the original papers.

E Sub-Question Consistency Analysis

To assess the semantic alignment between pre-
dicted reasoning paths and the original question,
1
[Link]

14
 (a) Distribution of Sub-Question Categories 100
(b) Hits@1 and F1 by Category
70 RouterKGQA
GCR
GNN-RAG 61.4 90
60
52.0
50 80
46.0
Proportion (%)

Score (%)
41.1
40 70
34.7
30 60
24.9

20 RouterKGQA
12.4
50 GCR
10.8 GNN-RAG
10 5.9 6.9 Hits@1 (solid)
40 F1 (dotted)
0
qual Subset verlap Disjoin
t qual bset erlap Disjoin
t
Exact E Recon Partial
O Exact E Recon
Su
Partial
Ov

Figure 2: Sub-question consistency analysis on CWQ. (a) Distribution of sub-question set relationship categories
across methods. (b) Hits@1 (solid) and F1 (dotted) by sub-question relationship category.

15