Multi Agent RAG
Multi Agent RAG
Graphs
Anton Gusarov1, 2 , Anastasia Volkova2 , Valentin Khrulkov1 , Andrey Kuznetsov1, 2, 3 ,
Evgenii Maslov1, 2 , Ivan Oseledets1, 2
1
AIRI, 2 Skoltech, 3 Innopolis University
Corresponding author: gusarov@[Link]
Preprint
sist with sub-database retrieval and SQL refinement. Alpha- to graph databases. However, this is based on the Chinese
SQL (Li et al. 2025) tackles the challenges of zero-shot Text- language OwnThink knowledge graph and not available
to-SQL by combining Monte Carlo Tree Search (MCTS) for direct download. (Feng, Papicchio, and Rahman 2025)
with an LLM-based action model and self-supervised re- present CypherBench, a benchmark for evaluating LLM-
ward. based question answering over full-schema property graphs
using Cypher, built on Wikidata-derived datasets with di-
Agent-Based Graph Reasoning. Several recent papers on verse query types aligned to realistic graph schemas. (Tiwari
iterative agent-based methods aimed at graph data. et al. 2025) introduce Auto-Cypher, a fully automated LLM-
The Graph Chain-of-Thought solution (Jin et al. 2024) supervised pipeline for generating and verifying synthetic
tackles the LLM graph reasoning problem with three-step Text2Cypher training data (SynthCypher) and introduces an
iterations: reasoning, interaction, and execution. adapted SPIDER-Cypher benchmark to address the lack of
Based on Graph-CoT ideas, (Gao et al. 2025) proposes a evaluation standards in this domain.
multi-agent GraphRAG approach addressing QA tasks with In comparison to prior work that primarily focus on static
cooperative agentic KG exploration and extraction strategies prompt-based Cypher generation or dataset construction for
search supported by a multi-perspective self-reflection mod- benchmarking, our approach introduces a modular multi-
ule that refines reasoning strategy. agent GraphRAG workflow that emphasizes iterative refine-
Text-to-Cypher. (Hornsteiner et al. 2024) develop a mod- ment, named entity verification, and aggregated semantic-
ular natural language interface for Neo4j using GPT-4 Turbo syntactic feedback. Crucially, our system integrates runtime
to perform Cypher query generation, database selection, interaction with the graph database to detect and correct
and error correction, employing the design science research both hallucinations and logical errors. In contrast to (Horn-
methodology and reporting high accuracy in few-shot sce- steiner et al. 2024), which targets Neo4j, we implement our
narios for practical Text-to-Cypher tasks on CLEVR graph workflow on Memgraph, demonstrating broader applicabil-
dataset (Mack and Jefferson 2018), representing a transport ity to LPG-compatible backends where runtime efficiency
transit network. is critical. Additionally, we test the system on industry-
(Chatterjee and Dethlefs 2022) present an automated QA grade use cases, specifically IFC-based digital building rep-
system for wind turbine maintenance that uses a semantic resentations, extending beyond general-purpose or synthetic
parser to convert natural language queries into Cypher for datasets into the AEC (Architecture, Engineering, and Con-
interactive decision-making support, and release a domain- struction) domain.
specific dataset of question-Cypher pairs for this purpose.
Quite a few question-to-Cypher datasets are publicly Implementation
available. SpCQL (Guo et al. 2022) introduces the first We propose an LLM workflow that adopts a modular agentic
large-scale Text-to-Cypher (Text-to-CQL) semantic pars- architecture with a feedback-driven refinement loop to over-
ing dataset containing 10,000 natural language and Cypher come the limitations of standalone LLMs in text-to-Cypher
query pairs over a Neo4j graph, highlighting the unique generation by leveraging their reasoning capabilities for it-
challenges of Cypher compared to SQL and exposing the erative improvement (Qu et al. 2024). Figure 1 provides an
limitations of existing Text-to-SQL models when applied overview of the system components and data flow.
The implementation code along with the experimental re- queries to the database. For any entity not found, indicat-
sults is publicly available at: [Link] ing likely hallucination, the module initiates a two-step
recovery process. First, it retrieves candidate replace-
Agent Roles and Responsibilities. The workflow is com- ments based on the normalized Levenshtein similarity ra-
posed of seven cooperating agents and one graph database tio (Bachmann 2024). Second, it leverages an LLM to
query executor module, each specializing in a distinct sub- semantically rank all existing entities of the same type in
task. the database, selecting the most contextually appropriate
1. Query Generator formulates a Cypher query based on alternative.
the user question in natural language. The query should 6. Instructions Generator synthesizes revision instruc-
be both syntactically correct and semantically aligned tions based on the verification results targeting halluci-
with the user intent. The generation is grounded on the nated or misnamed entities. It takes as input the struc-
provided context graph schema to the model, including tured feedback from the Verification Module, including
node and pairwise relationship descriptions. In subse- which entities failed to match the data content. For each
quent iterations, if the first-shot generation was not ac- invalid component, it generates a correction proposal by
cepted, the agent takes aggregated feedback from the combining: (i) edit-based suggestions derived from high-
Evaluator and Verification agents to refine the previous similarity alternatives, and (ii) semantically grounded
revision of the Cypher query. recommendations ranked by the LLM. The output is a
2. Graph Database Executor interfaces with the underly- concise instruction to guide the Query Generator on how
ing graph database engine (Memgraph in our case) to ex- to revise the query.
ecute the generated Cypher query and retrieve the out- 7. Feedback Aggregator integrates the outputs of both the
come, which may consist of structured result data, an er- Query Evaluator and the Verification Module into a uni-
ror message, or an empty result set. The latter is com- fied correction strategy. It consolidates signals such as
mon when the Query Generator’s LLM incorrectly refer- semantic inconsistencies, execution errors, and naming
ences attribute names, often due to hallucinations or in- mismatches to produce structured and prioritized feed-
sufficient data in the graph schema. back. This aggregated feedback is the basis for guiding
3. Query Evaluator is responsible for assessing the seman- the subsequent correction of the query, ensuring that both
tic and logical adequacy of the generated Cypher query logical soundness and schema compliance are addressed.
relative to the user’s intent and the correctness of the 8. Accepted queries are passed to the Interpreter, which
query results. Functionally, it serves as an LLM-based finally generates a concise, domain-relevant natural lan-
critic (McAleese et al. 2024; Yang et al. 2025) within guage answer.
our pipeline. The evaluator is prompted to analyze three
key aspects: (1) consistency between the user’s intended Each agent in our pipeline is guided by a system prompt
semantics and its natural language explanation, (2) align- tailored to its role. The complete prompt templates are
ment of the query logic with the user question, and (3) va- provided in the public implementation: [Link]
lidity and informativeness of the returned results. Based your-repo.
on this assessment, it outputs both structured feedback Self-Correction Loop. The iterative schema-aware cor-
and a discrete query grade from the set: rection and normalization is a critical mechanism for boost-
• Accept: the query is error-free and the returned results ing query accuracy and semantic alignment. It operates over
fully and logically answer the user question. a maximum of four iterations, progressively improving the
Cypher query by incorporating feedback from both seman-
• Incorrect: the query executes without error and re-
tic validation and data-level verification. At each step, the
turns data, but is semantically misaligned, logically
system analyzes execution outcomes, detects logical flaws
flawed, or incomplete.
or schema mismatches, and generates targeted correction in-
• Error or Empty: the query either fails to execute due structions. The refinement loop is detailed in Algorithm 1.
to a runtime error or returns no results.
Incorporating Graph Schema into LLM Prompts. The
The last case commonly arises from misidentified named
LLM is data-informed such that the graph database schema
entities, overly restrictive conditions, or invalid traversal
is explicitly incorporated into the system prompt of the
paths in the graph often due to LLM hallucinations.
Query Generator agent. We experimentally observed that
4. Named Entity Extractor is an LLM-based component generation quality significantly improves when the schema
that identifies elements within the query such as node la- is presented in a format closely resembling actual Cypher
bels, property-value pairs, and relationship types that are query syntax, as this provides better structural grounding
susceptible to hallucination. Its primary function is to de- and improves token-level alignment with expected outputs.
compose the query to enable subsequent verification of For each node type, representative attribute–value exam-
their existence in the underlying graph data. ples are also provided to guide the generation of property-
5. Verification Module follows the Named Entity Extrac- based query conditions. Detailed listings of the node and
tor and for verifies the existence and correctness of the relationship schema used for the fictional character graph
extracted schema elements against the actual graph data. from the CypherBench dataset (Feng, Papicchio, and Rah-
This is done first programmatically via auxiliary Cypher man 2025) are included in Appendix A.
Figure 2: Single-storey Sample House IFC model (xBIM Team 2024), first utilized for GraphRAG-based information extraction
in (Iranmanesh, Saadany, and Vakaj 2025). Left: 3D representation of the building. Right: fragment illustrating the cross-
relations of IFC entities mapped into a labeled property graph, including spatial hierarchies, property sets, and quantitative
attributes.
Table 1: Accuracy comparison between linear-pass LLM baseline and our multi-agent text-to-Cypher generation system across
CypherBench domains and models.
varying structural complexity. This evaluation setup allows line across all models and domains in the experiment.
us to demonstrate the applicability of our method beyond On average, the proposed agentic workflow yields notice-
open-domain knowledge graphs and into structured engi- able improvements: on average +10.23% for Gemini 2.5
neering data contexts. A visualization of the sample building Pro, +6.79% for GPT-4o, +7.67% for Qwen3 Coder and
and IFC data structure is shown in Figure 2. +10.01% for GigaChat 2 MAX. The results indicate that in-
corporating iterative refinement, verification, and semantics-
Baseline and Models. We evaluate our proposed multi- syntax feedback aggregation enhances structured query gen-
agent Cypher generation pipeline against a baseline imple- eration capabilities in LLM-based information retrieval sys-
mented using four state-of-the-art LLM backbones: Gem- tems over the property graphs.
ini 2.5 Pro, GPT-4o (2024-11-20), GigaChat 2 MAX, and
Qwen3 Coder. Each model is tested under two configura- Sample IFC data. Table in Appendix C presents the re-
tions: (i) “Single”, a linear-pass baseline where only the sults of our workflow using Gemini 2.5 Pro on the architec-
Query Generator → Executor → Interpreter sequence per- tural Sample House IFC dataset, which contains ten ground-
forms the full answer generation task without iterative feed- truth question–answer pairs. Following the evaluation pro-
back but (up to four attempts also allowed); and (ii) “Agen- tocol of (Iranmanesh, Saadany, and Vakaj 2025), we re-
tic”, our Multi-Agent GraphRAG setup, in which the same port all questions and answers in full. Their method and
model drives the proposed multi-agent workflow with dis- results are used as the baseline for comparison. Compared
tinct roles for evaluation, verification, and iterative Cypher to (Iranmanesh, Saadany, and Vakaj 2025), our Multi-Agent
query refinement. GraphRAG system correctly answers the last three questions
(previously unanswered or answered partially) and shows
Evaluation Metrics. We report accuracy as the percent- the ability to express uncertainty (e.g., Question 2) and
age of correctly answered questions in natural language grounding responses on the graph database contents (e.g.,
form. Answer correctness is assessed using a reference- Questions 7 and 8).
based evaluation procedure, where a dedicated LLM com-
pares the generated natural language answer against the Discussion and limitations
given in a form of JSON database outcome ground truth.
In this section, we analyze the performance of the proposed
Following the LLM-as-a-judge framework (Tiwari et al.
workflow by tracing how its components process data, make
2025), we employ GigaChat 2 MAX as the evaluation model
decisions and refine Cypher query from iteration to iteration.
to judge semantic equivalence between the answers. This au-
Drawing on qualitative analysis of system traces (see exam-
tomated evaluation approach allows consistent and scalable
ple in Appendix B), we highlight factors contributing to suc-
evaluation across all tested models and domains. The prompt
cessful query generation and identify remaining limitations.
of a judge LLM apart from instructions contains few-shot
All experimental traces are publicly available together with
examples and available in the project’s repository.
the system’s code implementation.
The effectiveness of the Multi-Agent GraphRAG stems
Experimental Results from several key design components. First of all, schema
CypherBench dataset. Table 1 presents the accuracy re- validation and query names normalization play a crucial
sults following LLM-as-a-judge metric across five selected role. Database-grounded feedback enables correction of
CypherBench dataset domains for four foundation models structural errors such as misused relationship directions or
in both single-pass (without iterative refinement) and multi- types, while entity verification mitigates LLM hallucinations
agent settings. The final row reports the average perfor- by enforcing consistency with actual graph database con-
mance across all evaluated datasets. The pipeline was exe- tent. For example, when the system generates a query on
cuted once per dataset graph to obtain accuracy results, al- employees and their managers, database-informed feedback
lowing up to four refinement attempts per query. pushes correct relationship traversal (e.g., from employee to
According to these results, Multi-Agent GraphRAG manager via Reports to relationship), while entity veri-
pipeline consistently outperforms the linear-pass LLM base- fication ensures that department names or employee iden-
tifiers match actual entries via auxiliary database queries. structured querying. Furthermore, while previous systems
Together, they support multi-hop reasoning and allow the often rely on static schemas, our refinement loop dynami-
system to iteratively converge to schema-compliant and ex- cally interacts with the database through auxiliary Cypher
ecutable queries. queries, enabling more adaptive and context-aware correc-
The workflow also benefits from reformulating Cypher’s tion strategies.
strict comparisons into explicit value retrieval. Rather than These systems show promise as natural interfaces to com-
relying on binary equality checks that may yield empty plex domain-specific data. In our study, we demonstrated
or ambiguous results, returning relevant property values performance on IFC (Industry Foundation Classes) sample
for each entity allows it to infer the answer more trans- data – a widely adopted format for representing buildings
parently. For instance, when asked whether two characters in the AEC (Architecture, Engineering, and Construction)
share the same creator, the system first attempted a direct sector, highlighting the value of such pipelines for enabling
equality check ([Link] = [Link]), which simplified access to complex structured technical data.
returned nothing due to a mismatch. A subsequent refor- Future work includes extending this approach to multi-
mulation MATCH (c:Character) WHERE [Link] IN turn dialogue scenarios, incorporating explicit subgoal plan-
["Morbius, the Living Vampire", "Giganto"] ning for compositional queries, and developing larger
RETURN [Link], [Link] explicitly retrieves each domain-specific datasets, particularly for IFC-linked Cypher
character’s creator, avoiding empty results and making the queries, to support the adoption of AI-driven solutions in
comparison observable in the output. digital construction and operation.
Despite these strengths, several limitations remain, which
we outline below based on failure cases observed in the References
traces. One observed limitation is the difficulty in handling
compositional queries involving disjunctions (e.g., ”Who Angles, R. 2018. The Property Graph Database Model. In
are married to Cersei Lannister or have Cassana Baratheon Alberto Mendelzon Workshop on Foundations of Data Man-
as their mother?”, which requires the union of two struc- agement.
turally distinct subqueries) and symmetric relationships (e.g. Angles, R.; Bonifati, A.; Garcı́a, R.; and Vrgoč, D. 2024.
(:Character)-[:hasSpouse]-(other:Character), Path-based Algebraic Foundations of Graph Query Lan-
which can match from either side and therefore complicates guages. arXiv:2407.04823.
schema validation and query formulation) even with multi- Bachmann, M. 2024. RapidFuzz: Fuzzy String Matching for
step feedback. Addressing these cases may require explicit Python. [Link]
query planning or intermediate symbolic representations of Accessed: 2025-07-30.
query intent.
Chatterjee, J.; and Dethlefs, N. 2022. Automated Question-
The Multi-Agent GraphRAG also struggles with multi-
Answering for Interactive Decision Support in Operations
intent questions that require decomposing and aligning dis-
and Maintenance of Wind Turbines. IEEE Access.
tinct subgoals such as e.g. listing children and counting their
descendants (in CypherBench’s fictional character) leading Donkers, A.; Yang, D.; and Baken, N. 2020. Linked data for
to semantic conflation and misaligned answer structure. This smart homes: comparing RDF and labeled property graphs.
highlights a limitation in handling compositional queries In LDAC.
where sub-intents must be separated and resolved indepen- Edge, D.; Trinh, H.; Cheng, N.; Bradley, J.; Chao, A.; Mody,
dently. A.; Truitt, S.; Metropolitansky, D.; Ness, R. O.; and Larson,
These findings suggest that the system’s success is tied to J. 2025. From Local to Global: A Graph RAG Approach to
its ability to integrate database-aware verification, semantic- Query-Focused Summarization. arXiv:2404.16130.
syntactic feedback, and iterative refinement. At the same Feng, Y.; Papicchio, S.; and Rahman, S. 2025. Cypher-
time, addressing the remaining limitations, particularly for Bench: Towards Precise Retrieval over Full-scale Modern
compositional and structurally complex Cypher queries, Knowledge Graphs in the LLM Era. arXiv:2412.18702.
opens a way for future improvements in agentic query gen-
eration over PLG graphs. Francis, N.; Green, A.; Guagliardo, P.; Libkin, L.; Lindaaker,
T.; Marsault, V.; Plantikow, S.; Rydberg, M.; Selmer, P.; and
Conclusion Taylor, A. 2018. Cypher: An evolving query language for
property graphs. In Proceedings of the 2018 international
We presented the Multi-Agent GraphRAG system for conference on management of data, 1433–1445.
text-to-Cypher question answering over property graph
databases. Our approach combines modular LLM-agentic Gao, J.; Zou, X.; Ai, Y.; Li, D.; Niu, Y.; Qi, B.; and
components for query generation, query entities verification, Liu, J. 2025. Graph Counselor: Adaptive Graph Explo-
execution, and feedback aggregation into an iterative refine- ration via Multi-Agent Synergy to Enhance LLM Reason-
ment loop. Experimental results across CypherBench and ing. arXiv:2506.03939.
IFC-derived datasets demonstrate that this design improves Guo, A.; Li, X.; Xiao, G.; Tan, Z.; and Zhao, X. 2022.
query accuracy and robustness compared to existing LLM Spcql: A semantic parsing dataset for converting natural lan-
baselines. guage into cypher. In Proceedings of the 31st ACM Inter-
Unlike prior work focused primarily on Neo4j, our system national Conference on Information & Knowledge Manage-
targets Memgraph, expanding the landscape of LLM-based ment, 3973–3977.
Han, H.; Wang, Y.; Shomer, H.; Guo, K.; Ding, J.; Lei, Y.; McAleese, N.; Pokorny, R. M.; Uribe, J. F. C.; Nitishin-
Halappanavar, M.; Rossi, R. A.; Mukherjee, S.; Tang, X.; skaya, E.; Trebacz, M.; and Leike, J. 2024. LLM Critics
He, Q.; Hua, Z.; Long, B.; Zhao, T.; Shah, N.; Javari, A.; Help Catch LLM Bugs. arXiv:2407.00215.
Xia, Y.; and Tang, J. 2025. Retrieval-Augmented Generation Memgraph Ltd. 2025. Memgraph Database. https://
with Graphs (GraphRAG). arXiv:2501.00309. [Link]. Accessed: 2025-07-31.
Hornsteiner, M.; Kreussel, M.; Steindl, C.; Ebner, F.; Empl, Peng, B.; Zhu, Y.; Liu, Y.; Bo, X.; Shi, H.; Hong, C.; Zhang,
P.; and Schönig, S. 2024. Real-time text-to-cypher query Y.; and Tang, S. 2024. Graph Retrieval-Augmented Genera-
generation with large language models for graph databases. tion: A Survey. arXiv:2408.08921.
Future Internet, 16(12): 438.
Pourreza, M.; Li, H.; Sun, R.; Chung, Y.; Talaei, S.;
Iranmanesh, S.; Saadany, H.; and Vakaj, E. 2025. LLM- Kakkar, G. T.; Gan, Y.; Saberi, A.; Ozcan, F.; and Arik,
assisted Graph-RAG Information Extraction from IFC Data. S. O. 2024. CHASE-SQL: Multi-Path Reasoning and
arXiv:2504.16813. Preference Optimized Candidate Selection in Text-to-SQL.
ISO/IEC. 2024. ISO/IEC 39075:2024 Information technol- arXiv:2410.01943.
ogy – Database languages – Graph Query Language (GQL). Qu, Y.; Zhang, T.; Garg, N.; and Kumar, A. 2024. Recur-
Technical Report ISO/IEC 39075:2024, ISO/IEC. Standard. sive introspection: Teaching language model agents how to
Jiang, J.; Zhou, K.; Zhao, W. X.; Song, Y.; Zhu, C.; Zhu, H.; self-improve. Advances in Neural Information Processing
and Wen, J.-R. 2024. KG-Agent: An Efficient Autonomous Systems, 37: 55249–55285.
Agent Framework for Complex Reasoning over Knowledge Rubin, O.; and Berant, J. 2021. SmBoP: Semi-
Graph. arXiv:2402.11163. autoregressive Bottom-up Semantic Parsing.
Jiang, J.; Zhou, K.; Zhao, W. X.; and Wen, J.-R. 2023. arXiv:2010.12412.
UniKGQA: Unified Retrieval and Reasoning for Solving Solar-Lezama, A. 2009. The sketching approach to program
Multi-hop Question Answering Over Knowledge Graph. synthesis. In Asian symposium on programming languages
arXiv:2212.00959. and systems, 4–13. Springer.
Jin, B.; Xie, C.; Zhang, J.; Roy, K.; Zhang, Y.; Li, Z.; Li, Tiwari, A.; Malay, S. K. R.; Yadav, V.; Hashemi, M.; and
R.; Tang, X.; Wang, S.; Meng, Y.; and Han, J. 2024. Graph Madhusudhan, S. T. 2025. Auto-Cypher: Improving LLMs
Chain-of-Thought: Augmenting Large Language Models by on Cypher generation via LLM-supervised generation-
Reasoning on Graphs. In Ku, L.-W.; Martins, A.; and Sriku- verification framework. arXiv:2412.12612.
mar, V., eds., The 62nd Annual Meeting of the Associa-
Vanlande, R.; Nicolle, C.; and Cruz, C. 2008. IFC and
tion for Computational Linguistics, Proceedings of the An-
building lifecycle management. Automation in construction,
nual Meeting of the Association for Computational Linguis-
18(1): 70–78.
tics, 163–184. Association for Computational Linguistics
(ACL). Publisher Copyright: © 2024 Association for Com- Walsh, B.; Mohamed, S. K.; and Nováček, V. 2020. BioKG:
putational Linguistics.; Findings of the 62nd Annual Meet- A Knowledge Graph for Relational Learning On Biological
ing of the Association for Computational Linguistics, ACL Data. Proceedings of the 29th ACM International Confer-
2024 ; Conference date: 11-08-2024 Through 16-08-2024. ence on Information & Knowledge Management.
Li, B.; Zhang, J.; Fan, J.; Xu, Y.; Chen, C.; Tang, N.; and Wang, B.; Ren, C.; Yang, J.; Liang, X.; Bai, J.; Chai, L.; Yan,
Luo, Y. 2025. Alpha-SQL: Zero-Shot Text-to-SQL using Z.; Zhang, Q.-W.; Yin, D.; Sun, X.; and Li, Z. 2025. MAC-
Monte Carlo Tree Search. arXiv:2502.17248. SQL: A Multi-Agent Collaborative Framework for Text-to-
SQL. arXiv:2312.11242.
Li, H.; Zhang, J.; Li, C.; and Chen, H. 2023a. Resdsql: De-
coupling schema linking and skeleton parsing for text-to-sql. Wang, B.; Shin, R.; Liu, X.; Polozov, O.; and Richardson,
In Proceedings of the AAAI Conference on Artificial Intelli- M. 2021. RAT-SQL: Relation-Aware Schema Encoding and
gence, volume 37, 13067–13075. Linking for Text-to-SQL Parsers. arXiv:1911.04942.
Li, J.; Li, Y.; Li, G.; Jin, Z.; Hao, Y.; and Hu, X. 2023b. xBIM Team. 2024. [Link]: Example Building
Skcoder: A sketch-based approach for automatic code gen- Model in IFC Format. [Link]
eration. In 2023 IEEE/ACM 45th International Conference com/xBimTeam/XbimEssentials/refs/heads/master/Tests/
on Software Engineering (ICSE), 2124–2135. IEEE. TestFiles/[Link]. Accessed: 2025-07-31.
Luo, H.; E, H.; Tang, Z.; Peng, S.; Guo, Y.; Zhang, W.; Ma, Xu, X.; Liu, C.; and Song, D. 2017. SQLNet: Generating
C.; Dong, G.; Song, M.; Lin, W.; Zhu, Y.; and Luu, A. T. Structured Queries From Natural Language Without Rein-
2024. ChatKBQA: A Generate-then-Retrieve Framework forcement Learning. arXiv:1711.04436.
for Knowledge Base Question Answering with Fine-tuned Yang, R.; Ye, F.; Li, J.; Yuan, S.; Zhang, Y.; Tu, Z.; Li,
Large Language Models. In Findings of the Association for X.; and Yang, D. 2025. The Lighthouse of Language:
Computational Linguistics ACL 2024, 2039–2056. Associa- Enhancing LLM Agents via Critique-Guided Improvement.
tion for Computational Linguistics. arXiv:2503.16024.
Mack, D.; and Jefferson, A. 2018. CLEVR graph: A dataset Yu, T.; Li, Z.; Zhang, Z.; Zhang, R.; and Radev, D. 2018a.
for graph question answering. [Link] TypeSQL: Knowledge-based Type-Aware Neural Text-to-
ai/clevr-graph. Accessed: 2025-07-29. SQL Generation. arXiv:1804.09769.
Yu, T.; Yasunaga, M.; Yang, K.; Zhang, R.; Wang, D.; Li,
Z.; and Radev, D. 2018b. SyntaxSQLNet: Syntax Tree Net-
works for Complex and Cross-DomainText-to-SQL Task.
arXiv:1810.05237.
Zhong, V.; Xiong, C.; and Socher, R. 2017. Seq2SQL: Gen-
erating Structured Queries from Natural Language using Re-
inforcement Learning. arXiv:1709.00103.
Appendix A Listing 2: Example relationships schema from the fictional
The listings below provide example schemas used to guide character graph in CypherBench
the Query Generator agent during Cypher query generation. Type: basedIn
They include representative node and relationship defini- - (:Organization)-[:basedIn]->(:Location)
tions for the fictional character graph in the CypherBench
dataset (Feng, Papicchio, and Rahman 2025), note that the Type: bornIn
- (:Character)-[:bornIn]->(:Location)
schemas are formatted to closely resemble Cypher syntax.
Type: diedIn
Listing 1: Example node schema with properties and sam- - (:Character)-[:diedIn]->(:Location)
pled values from the fictional character graph in Cypher-
Bench dataset Type: fromUniverse
Each node type includes properties - (:Character)-[:fromUniverse]->
hierarchy divided by ’.’ and sampled (:FictionalUniverse)
examples of each property values. - (:Organization)-[:fromUniverse]->
(:FictionalUniverse)
Node Type: Character
Properties: Type: hasFather
.aliases: "Ibuki Suika" - (:Character)-[:hasFather]->(:Character)
.birth_name: Thomas Merlyn
.country_of_citizenship: "Denmark" Type: hasMother
.creator: Ake Holmberg - (:Character)-[:hasMother]->(:Character)
.description: adventure time character
.gender: trans woman Type: hasSpouse
.name: Thunderbolt (DC Comics) - (:Character)-[:hasSpouse]->(:Character)
.occupation: "prophet, psychic"
Type: hasStudent
Node Type: FictionalUniverse - (:Character)-[:hasStudent]->(:Character)
Properties:
.aliases: "SoD universe" Type: killedBy
.copyright_holder: Sony Group - (:Character)-[:killedBy]->(:Character)
.creator: CD Projekt RED
.description: self-contained narrative Type: memberOf
universe of the SCP web-based - (:Character)-[:memberOf]->
collaborative writing project (:Organization)
.inception_year: 1999
.name: The Black Hole universe
(Query execution attempts made: 1) Node property value used in the query:
"corlys velaryon"
Query Evaluator
Evaluation: Error or empty Suggested node property corrections:
Property: "Corlys Velaryon"
Feedback: Levenstein similarity score: 86.66
The query failed because it uses a pattern expression, such
Property: "Lucerys Velaryon"
as (c)-[:hasFather]->(father), directly inside a Levenstein similarity score: 77.41
WHERE clause. This syntax is not supported by Mem-
graph, as indicated by the error message. To check for Property: "Jacaerys Velaryon"
the existence of a relationship pattern, it must be wrapped Levenstein similarity score: 75.0
in the EXISTS() function. The query should be rewrit-
ten to use OPTIONAL MATCH or separate MATCH clauses Other semantically relevant suggestions:
with WITH to combine the results, which is a more robust "Laenor Velaryon",
pattern. "Laena Velaryon",
"Jacaerys Velaryon"
For example: ---
Node label: "Character"
MATCH (child:Character)-[:hasFather]->(f:
Character)
Node property value used in the query:
WHERE toLower([Link]) = ’corlys velaryon’
"daemon targaryen"
WITH COLLECT(child) AS children MATCH
(spouse_char:Character)-[:hasSpouse]-(s:
Suggested node property corrections
Character)
Property: "Daemon Targaryen"
WHERE toLower([Link]) = ’daemon targaryen’
Levenstein similarity score: 87.5
WITH children + COLLECT(spouse_char) as
allPeople!
Property: "Aemon Targaryen"
UNWIND allPeople as person
Levenstein similarity score: 83.87
RETURN count(DISTINCT person).
Query Evaluator
Property: "Aemond Targaryen" Evaluation: Accept
Levenstein similarity score": 81.25
Feedback:
Other semantically relevant suggestions:
"Daemon Targaryen", The query correctly implements the required logic. It uses
"Aemond Targaryen", two separate OPTIONAL MATCH clauses to find characters
"Daeron Targaryen" satisfying each condition (child of Corlys, spouse
of Daemon), combines the results into a single list, and
Relationships verification results: then counts the distinct characters in that list. This is a
--------------------------------- valid, though somewhat complex, way to implement an
[] OR condition and correctly answers the user’s question.
Feedback Aggregator
Your query incorrectly used an expression in the WHERE
clause. Instead, use two separate MATCH clauses for each
condition: one to find characters :hasFather Corlys
Velaryon, and another to find characters :hasSpouse
Daemon Targaryen. Combine the results from both
matches, then unwind the list and return the count of dis-
tinct characters. Ensure you use the exact, case-sensitive
names Corlys Velaryon and Daemon Targaryen for
the ‘name‘ property.
(Query generation attempt: 2)
Query Generator
OPTIONAL MATCH
(child:Character)-[:hasFather]->
(:Character{name: ’Corlys Velaryon’})
WITH collect(DISTINCT child) AS children
OPTIONAL MATCH
(spouse_char:Character)-
[:hasSpouse]-
(:Character{name: ’Daemon Targaryen’})
WITH children + collect(
DISTINCT spouse_char
)
AS all_chars UNWIND all_chars AS c
RETURN count(DISTINCT c)
Graph DB Executor
Query outcome:
[
{
"count(DISTINCT c)": 3
}
]
Appendix C
Below are the predicted answers and generated correspond-
ing Cypher queries on the Sample House IFC data using
our Multi-Agent GraphRAG pipeline. The user questions
and corresponding expected answers are sourced from (Iran-
manesh, Saadany, and Vakaj 2025).