MEGA-FrOG: Multi-Agent Enhanced

Generation and Adaptation Framework of Open

GraphRAG

Anonymous Submission to ESWC 2026

Anonymized for Review

Abstract. Large language models face persistent challenges in knowledge-

intensive tasks: knowledge staleness, hallucination, and opacity. While
Graph Retrieval-Augmented Generation (GraphRAG) addresses these
limitations using knowledge graphs as structured knowledge bases, ex-
isting systems suffer from inadequate entity extraction, absence of spe-
cialized training, and limited dataset generation for natural language to
SPARQL translation. We present MEGA-FrOG (Multi-Agent Enhanced
Generation and Adaptation Framework of Open GraphRAG), extend-
ing the FrOG framework through three key contributions: (1) a dual
dataset generation methodology combining template-based and random-
walk approaches, producing over 8,000 question-SPARQL pairs across
four diverse knowledge graphs, (2) parameter-efficient fine-tuning using
QLoRA, training only 0.12% of model parameters (8.3M of 7B) to create
specialized models for entity extraction and SPARQL generation, and (3)
an enhanced multi-agent architecture with intelligent routing and com-
prehensive fallback mechanisms. MEGA-FrOG achieves substantial im-
provements over FrOG baseline (+11.2% to +37.1% Jaccard similarity,
average +27.7%), demonstrating that open-source LLMs with special-
ized fine-tuning can achieve competitive GraphRAG performance while
offering advantages in customization, privacy, and cost-effectiveness.

Keywords: GraphRAG · Knowledge Graphs · SPARQL Generation ·

Question Answering · Fine-tuning · Multi-Agent Systems

1 Introduction
Large language models (LLMs) have revolutionized natural language under-
standing and generation, powering applications from conversational agents to
code synthesis. However, LLMs face persistent challenges that limit their re-
liability in knowledge-intensive tasks: (1) knowledge staleness—information
embedded in parameters becomes outdated as the world changes, (2) hallucina-
tion—models generate plausible but factually incorrect responses when lacking
relevant knowledge, and (3) opacity—the reasoning process behind generated
answers remains largely uninterpretable, hindering trust in critical applications.
Retrieval-Augmented Generation (RAG) addresses these limitations by cou-
pling LLMs with external knowledge sources, enabling dynamic knowledge up-
dates and grounding responses in retrieved evidence. Traditional RAG systems
2 Anonymous Submission to ESWC 2026

retrieve relevant text passages using dense embeddings, then condition LLM gen-
eration on retrieved context. While effective for many applications, text-based
RAG suffers from inherent limitations: dense embeddings remain opaque and
difficult to interpret, passage segmentation (chunking) can fragment contextual
information, and retrieved text often lacks the structured relationships necessary
for complex reasoning.
Graph Retrieval-Augmented Generation (GraphRAG) offers a com-
pelling alternative by using knowledge graphs (KGs) as structured knowledge
bases. KGs organize information as entities connected by typed relationships,
enabling precise retrieval of facts, multi-hop reasoning through graph traversal,
and full transparency—every retrieved fact traces back to explicit graph struc-
ture. This interpretability is particularly valuable in domains like healthcare,
legal reasoning, and scientific research where users must verify answer prove-
nance.
Recent work has explored GraphRAG architectures with varying approaches.
Microsoft’s GraphRAG constructs document-level knowledge graphs with hier-
archical summarization for query-focused text generation. LEGO-GraphRAG
introduces modular retrieval pipelines with subgraph extraction and path-based
reasoning. FrOG (Framework of Open GraphRAG) [1] pioneered an entirely
open-source GraphRAG system, demonstrating that effective question-answering
over KGs can be achieved using only open-source LLMs and tools, without re-
quiring proprietary models or extensive fine-tuning.

1.1 Research Gap: Limitations of Existing GraphRAG Systems

Despite these advances, existing GraphRAG systems—including FrOG—face

three critical limitations that constrain their effectiveness:
Limitation 1: Inadequate Entity and Property Extraction. GraphRAG
performance hinges on accurately mapping natural language questions to KG en-
tities and properties. Existing systems rely on few-shot prompting with general-
purpose LLMs, achieving limited accuracy on this specialized task. Poor entity
extraction cascades through the pipeline: incorrect entities lead to irrelevant
subgraph retrieval, and missed properties result in incomplete SPARQL queries.
Limitation 2: Absence of Specialized Model Training. Most GraphRAG
systems use off-the-shelf LLMs with few-shot prompting for query generation.
While convenient, this approach fails to leverage task-specific training that could
substantially improve accuracy. The barrier to fine-tuning has been the lack of di-
verse, high-quality training datasets—manually creating thousands of question-
SPARQL pairs per KG is prohibitively expensive.
Limitation 3: Limited Dataset Generation for NL2SPARQL. The
scarcity of Natural Language to SPARQL (NL2SPARQL) datasets constrains
both system evaluation and model training. Existing benchmarks like QALD
cover only general-domain KGs (Wikidata, DBpedia) and contain at most a few
hundred questions. Domain-specific or enterprise KGs typically have no associ-
ated question-answering datasets at all.
 Title Suppressed Due to Excessive Length 3

1.2 Our Contributions: MEGA-FrOG

We present MEGA-FrOG (Multi-Agent Enhanced Generation and Adap-
tation Framework of Open GraphRAG), a comprehensive system that ex-
tends FrOG by systematically addressing the three limitations above. Our work
makes the following contributions:
Contribution 1: Dual Dataset Generation Methodology. We intro-
duce a two-pronged approach to automated NL2SPARQL dataset generation
combining: (a) template-based generation using discovery-first KG pattern min-
ing to ensure structural validity, and (b) random-walk generation using graph
traversal to discover natural multi-hop patterns. This methodology produces di-
verse, high-quality datasets across multiple KG domains, generating over 8,000
question-SPARQL pairs across four knowledge graphs (Wikidata, Curriculum
KG, Legal KG, GESIS KG).
Contribution 2: Specialized Model Fine-tuning. We employ QLoRA
(Quantized Low-Rank Adaptation), a parameter-efficient fine-tuning technique,
to create specialized models for two critical subtasks: (a) entity and property ex-
traction, producing structured JSON outputs identifying question entities and
relevant KG properties, and (b) SPARQL query generation, translating nat-
ural language and extracted context into executable queries. Our fine-tuning
approach trains only 0.12% of model parameters (8.3M of 7B), achieving sub-
stantial accuracy improvements with minimal computational cost and enabling
practical deployment on consumer hardware.
Contribution 3: Enhanced Multi-Agent Architecture. We design a
sophisticated multi-agent system using LangGraph that orchestrates eight spe-
cialized agents with intelligent routing and comprehensive fallback mechanisms.
Unlike FrOG’s linear pipeline, MEGA-FrOG employs: (a) adaptive strategy se-
lection routing queries to verbalization (simple queries) or SPARQL generation
(complex queries) based on complexity classification, (b) parallel context re-
trieval combining property/class extraction with optional verbalization for richer
SPARQL generation context, and (c) multi-level fallback hierarchy ensuring ro-
bust answer coverage even when primary retrieval fails.

1.3 Key Results

We evaluate MEGA-FrOG across four diverse knowledge graphs spanning differ-
ent scales (874 to 1.67B triples), domains (general, educational, legal, social sci-
ence), and URI representations (cryptic vs. human-readable). Our experiments
demonstrate:
– Substantial improvements over FrOG baseline (+11.2% to +37.1%
Jaccard similarity across KGs, with average improvement of +27.7%)
– State-of-the-art open-source performance (achieves competitive accu-
racy F1 Macro 0.329 on QALD-9-Plus benchmark compared to systems using
proprietary LLMs)
– Strong cross-domain generalization (highest accuracy on Curriculum
KG 0.805, demonstrating effectiveness on domain-specific enterprise KGs)
4 Anonymous Submission to ESWC 2026

– Component effectiveness validated (comprehensive ablation studies show

fine-tuning provides the largest individual contribution +25-37%, ontology
retrieval is essential, removing it causes -77% degradation on some KGs)

Our analysis reveals that specialized model fine-tuning provides the most
significant performance boost, validating our dataset generation and training
methodology. Interestingly, we discover that verbalization—while beneficial for
simple queries on large KGs—can hinder performance on smaller, structured
KGs by incorrectly short-circuiting aggregation queries. This insight suggests
that selective application of retrieval strategies based on query intent represents
an important direction for future GraphRAG systems.

1.4 Paper Organization

The remainder of this paper is organized as follows: Section 2 reviews related

work in RAG, GraphRAG, and the FrOG framework, positioning MEGA-FrOG
within existing literature. Section 3 details our methodology, including dataset
generation approaches, QLoRA-based fine-tuning procedures, and multi-agent
architecture design. Section 4 describes our experimental setup, knowledge graphs,
evaluation metrics, and baseline configurations. Section 5 presents comprehen-
sive results including main performance metrics, ablation studies, baseline com-
parisons, and qualitative case analysis. Section 6 concludes with a summary of
findings, limitations, and directions for future work.

2 Related Work

We position MEGA-FrOG within existing GraphRAG literature, focusing on

query-based approaches and the FrOG framework that serves as our baseline.

2.1 From RAG to GraphRAG

Retrieval-Augmented Generation (RAG) couples LLMs with external knowledge

sources to address knowledge staleness and hallucination. Traditional RAG sys-
tems retrieve text passages using dense embeddings, but face limitations: opaque
retrieval decisions, context fragmentation from chunking, and inability to cap-
ture structured relationships.
GraphRAG leverages knowledge graphs as structured knowledge bases, en-
abling precise entity-relationship retrieval and transparent reasoning paths. Mi-
crosoft GraphRAG [2] constructs document-level graphs with hierarchical sum-
marization but struggles with precise fact retrieval. LEGO-GraphRAG [3] intro-
duces modular path-based reasoning but focuses on graph traversal rather than
formal query generation.
Query-based GraphRAG generates formal queries (SPARQL, Cypher) for
structured retrieval. SPINACH [4] achieves state-of-the-art accuracy (F1 Macro
 Title Suppressed Due to Excessive Length 5

0.746 on QALD-9-Plus) using GPT-4o with few-shot prompting. However, pro-

prietary LLM dependence limits customization and deployment flexibility. Re-
cent work [5,6] demonstrates that open-source LLMs can generate executable
SPARQL with one-shot prompting, though accuracy remains lower than fine-
tuned approaches.

2.2 The FrOG Framework

FrOG (Framework of Open GraphRAG) [1] is our primary baseline and the
direct predecessor to MEGA-FrOG. FrOG pioneered a complete open-source
GraphRAG pipeline for question-answering over arbitrary KGs using only freely
available components.
Architecture: FrOG implements a four-stage pipeline: (1) translation for
multilingual support, (2) entity linking using few-shot LLM prompting with API-
based (Wikidata/DBpedia) or semantic search (local KGs) candidate retrieval,
(3) dual retrieval strategies—verbalization-based retrieval for simple queries (single-
hop graph traversal with semantic matching) and SPARQL generation for com-
plex queries (few-shot prompting with ontology context), and (4) answer gener-
ation with natural language synthesis.
Performance: FrOG demonstrated competitive results using Qwen2.5 7B,
Jina Embeddings v3, and Apache Jena, achieving Jaccard similarities of 0.458
(Wikidata), 0.517 (DBpedia), and 0.976 (Curriculum KG) in optimal configu-
rations. Ablation studies revealed that ontology retrieval was the most critical
component—removing it caused catastrophic performance degradation, partic-
ularly on domain-specific KGs.
Limitations Addressed by MEGA-FrOG: While FrOG established a
solid foundation, we identified three key gaps: (1) limited entity extraction—
general-purpose LLMs with few-shot prompting miss entities in complex queries,
(2) no specialized training—few-shot prompting for all tasks limits accuracy on
specialized subtasks like SPARQL generation, and (3) no systematic dataset gen-
eration—FrOG evaluates on existing benchmarks but provides no methodology
for generating diverse training datasets across arbitrary KGs.
MEGA-FrOG systematically addresses these limitations through automated
dataset generation, specialized QLoRA fine-tuning, and enhanced multi-agent
architecture, while maintaining FrOG’s commitment to open-source accessibility.

2.3 Positioning MEGA-FrOG

Table 1 positions MEGA-FrOG relative to representative prior systems across

key dimensions.
MEGA-FrOG uniquely combines open-source components, specialized fine-
tuning, sophisticated multi-agent coordination, and systematic dataset genera-
tion. The closest related work is FrOG, which pioneered open-source GraphRAG
but lacked specialized training and comprehensive dataset generation. MEGA-
FrOG extends FrOG with three main contributions (dataset generation, fine-
6 Anonymous Submission to ESWC 2026

Table 1. Comparison of GraphRAG Systems

System Open-Source Fine-tuning Multi-Agent Dataset Gen

Microsoft GraphRAG Partial No No Document-based
LEGO-GraphRAG No No Modular No
SPINACH No (GPT-4o) No No No
FrOG Yes No Linear Limited
MEGA-FrOG Yes Yes Yes Yes

tuning, enhanced architecture), achieving +27.7% average improvement in Jac-

card similarity while preserving open-source accessibility.

3 Methodology
We present MEGA-FrOG (Multi-Agent Enhanced Generation and Adaptation
Framework of Open GraphRAG), a comprehensive system that extends the
FrOG framework through three principal enhancements: (1) comprehensive dataset
generation for training specialized models, (2) parameter-efficient fine-tuning
using QLoRA techniques, and (3) sophisticated multi-agent architecture with
intelligent routing mechanisms. This section details each component’s design,
implementation, and integration within the overall system.

3.1 System Overview

MEGA-FrOG addresses three critical limitations identified in existing GraphRAG
frameworks: limited entity extraction capabilities, absence of specialized model
training, and constrained NL2SPARQL dataset generation. Our approach sys-
tematically tackles these challenges through an integrated pipeline combining
automated dataset generation, domain-specific model fine-tuning, and coordi-
nated multi-agent reasoning.
The system processes natural language questions through a multi-stage pipeline:
(1) translation for multilingual support, (2) entity extraction and linking to KG-
specific entities, (3) adaptive strategy selection determining retrieval approach,
(4) context retrieval through either verbalization or SPARQL generation, and
(5) answer generation with natural language responses. Unlike FrOG’s fixed
pipeline, MEGA-FrOG employs specialized fine-tuned models at critical stages
and implements intelligent routing with comprehensive fallback mechanisms.

3.2 Dataset Generation

A fundamental limitation of existing GraphRAG systems is the scarcity of high-
quality NL2SPARQL training data, particularly for domain-specific KGs. We
address this through a dual-generation methodology combining template-based
and random-walk approaches, producing diverse, high-coverage datasets across
multiple KG domains.
 Title Suppressed Due to Excessive Length 7

[Figure 1: MEGA-FrOG Architecture Overview]

Placeholder - Insert your architecture diagram here

Fig. 1. MEGA-FrOG architecture showing three main enhancement components: (1)

Dataset Generation pipeline producing training data, (2) Model Fine-tuning creating
specialized extractors and generators, and (3) Multi-Agent Architecture orchestrating
the complete question-answering workflow with adaptive routing and fallback mecha-
nisms.

[Figure 2: Dual Dataset Generation]

Placeholder - Insert dual generation diagram

Fig. 2. Template-based generation (left) uses predefined SPARQL patterns to ensure

structural correctness, while random-walk generation (right) explores graph topology
to discover natural question patterns. Both approaches feed into unified post-processing
for query optimization.

Template-Based Generation Template-based generation employs a discovery-

first strategy that systematically explores KG structure to identify common
query patterns, then generates natural language questions corresponding to these
patterns.
Process: (1) Entity selection and pattern discovery—randomly sample en-
tities from the target KG, perform structured graph traversal to identify fre-
quently occurring triple patterns, and analyze property frequency distributions.
(2) SPARQL template construction—create parameterized templates covering
four categories: simple 1-property queries {?s p ?o}, inverse 1-property queries
{?s p o}, complex 2-property queries {?s p1 o1; p2 ?o2}, and multi-hop queries
{?s p1 ?o1. ?o1 p2 o2}. (3) Natural language generation—use instruction-
tuned LLM (Qwen2.5 7B) with zero-shot chain-of-thought prompting to gen-
erate diverse question formulations from SPARQL templates. (4) Quality vali-
dation—execute each SPARQL query against the target KG to verify syntactic
correctness and result retrieval; filter pairs returning empty results or syntax
errors; apply duplicate detection based on semantic similarity (threshold 0.85).
8 Anonymous Submission to ESWC 2026

[Figure 3: SPARQL Post-Processing]

Placeholder - Insert post-processing diagram

Fig. 3. SPARQL post-processing: (1) Syntax normalization standardizes query struc-

ture, (2) Triple pattern optimization eliminates redundancies, (3) Filter injection adds
necessary constraints, (4) Query validation ensures executability.

This approach generated 1,247 question-SPARQL pairs for Wikidata, 892

for Curriculum KG, 1,156 for Legal KG, and 1,089 for GESIS KG, with query
execution success rates exceeding 94% across all domains.

Random-Walk Generation Random-walk generation complements template-

based methods by exploring natural graph topology to discover query patterns
that emerge from actual KG structure rather than predefined templates.
Process: (1) Graph traversal —starting from randomly selected seed enti-
ties, perform controlled random walks of varying depths (1-hop, 2-hop, 3-hop),
probabilistically selecting outgoing edges based on property frequency. Walk
parameters are tuned per KG: Wikidata uses higher exploration rates (0.4)
to handle density, while domain-specific KGs use lower rates (0.2) to avoid
sparse regions. (2) Pattern extraction and question generation—extract triple se-
quences forming walk paths and generate natural language questions using LLM
prompting with only the triple path and entity labels, encouraging more natural,
varied questions. (3) SPARQL reconstruction—reverse-engineer corresponding
SPARQL queries from walk paths by identifying query variables (terminal node
for forward walks, source node for backward walks), constructing triple patterns
from walk edges, and adding appropriate SELECT/ASK/COUNT clauses based
on question type.
Random-walk generation produced 1,089 pairs for Wikidata, 734 for Cur-
riculum KG, 978 for Legal KG, and 892 for GESIS KG. Notably, random-walk
datasets exhibited higher linguistic diversity (average pairwise question similar-
ity 0.62) compared to template-based datasets (0.71), though with slightly lower
SPARQL execution success rates (89% vs. 94%).

SPARQL Post-Processing Both generation approaches produce SPARQL

queries that may contain inefficiencies, redundancies, or opportunities for opti-
mization. We implement a four-stage post-processing pipeline to refine generated
queries.
Stages: (1) Syntax normalization—expand abbreviated prefixes to full URIs,
normalize whitespace, order clauses according to SPARQL specification. (2)
 Title Suppressed Due to Excessive Length 9

Triple pattern optimization—eliminate redundant patterns yielding identical bind-

ings, remove contradictory constraints, replace unnecessary OPTIONAL clauses,
and use property paths instead of multiple triple patterns (e.g., ?s p1/p2 ?o in-
stead of ?s p1 ?x. ?x p2 ?o). (3) Filter and constraint injection—inject type
constraints for queries targeting specific entity types, add FILTER clauses with
regex patterns for string-matching questions, and inject datatype constraints for
temporal questions. (4) Execution validation and repair —execute post-processed
queries against target KGs, flag queries exceeding timeout thresholds (10 sec-
onds) or returning excessive results (>10,000), and attempt automated repair
for common failure patterns.
Post-processing improved query execution success rates from 89-94% to 96-
98% across datasets, while reducing average query execution time by 23% through
optimization. Final datasets combine template-based and random-walk genera-
tions, yielding 2,336 pairs for Wikidata, 1,626 for Curriculum KG, 2,134 for
Legal KG, and 1,981 for GESIS KG.

3.3 Specialized Model Fine-tuning

While general-purpose LLMs demonstrate impressive few-shot learning capabil-
ities, domain-specific fine-tuning yields substantial performance improvements
for specialized tasks like entity extraction and SPARQL generation. We employ
QLoRA (Quantized Low-Rank Adaptation) to fine-tune lightweight task-specific
models, achieving high accuracy with minimal computational overhead.

QLoRA Architecture QLoRA enables parameter-efficient fine-tuning by freez-

ing base model weights and training only small adapter modules, reducing train-
able parameters by over 99% compared to full fine-tuning. The approach consists
of three key components:

1. 4-bit quantization—quantize base model weights to 4-bit precision us-

ing NormalFloat (NF4) quantization, reducing memory footprint by 75%
and enabling fine-tuning of 7B-parameter models on consumer GPUs (24GB
VRAM)
2. Low-rank adapters—insert trainable low-rank matrices into each trans-
former layer’s attention and feed-forward blocks; for original weight matrix
W ∈ Rd×k , learn decomposition W ′ = W + BA where B ∈ Rd×r , A ∈ Rr×k ,
and rank r ≪ min(d, k); we set r = 16 for attention layers and r = 32 for
feed-forward layers
3. Gradient checkpointing and paged optimizers—employ gradient check-
pointing (recomputing activations during backward pass) and paged opti-
mizers that offload optimizer states to CPU memory when GPU memory is
constrained

Our QLoRA configuration trains only 8.3M parameters (0.12% of the 7B base
model) while achieving performance comparable to full fine-tuning. Training time
per model averages 4-6 hours on a single RTX 4090.
10 Anonymous Submission to ESWC 2026

[Figure 4: QLoRA Architecture]

Placeholder - Insert QLoRA diagram

Fig. 4. QLoRA architecture: Base LLM weights (blue) remain frozen at 4-bit quanti-
zation, while trainable low-rank adapter matrices A and B (orange) are inserted into
attention and feed-forward layers. Only adapters update during training, enabling ef-
ficient specialization with <1% trainable parameters.

Entity and Property Extraction Model Entity and property extraction

is critical: accurately identifying question entities and relevant KG properties
directly determines downstream SPARQL generation quality. We fine-tune a
specialized extraction model that outputs structured entity-property pairs given
natural language questions.
Input format: Questions with optional KG-specific context (available prop-
erty labels, example entities).
Output format: Structured JSON containing entity mentions with types
and confidence scores, and property labels with relevance scores.
Training data: We create training data from generated NL2SPARQL datasets
by extracting ground-truth entities and properties from SPARQL queries—parsing
queries to identify entity URIs (mapped to question spans via fuzzy string match-
ing) and property URIs (mapped to human-readable labels), yielding silver an-
notations with high precision due to SPARQL query correctness.
We fine-tune Qwen2.5 7B for 3 epochs with learning rate 2e-4, batch size 16,
and LoRA rank 16. Training uses cross-entropy loss on generated JSON tokens,
with early stopping based on validation F1 score. The final model achieves entity
extraction F1 of 0.89 and property extraction F1 of 0.82 on held-out test sets
across all KGs.

SPARQL Generation Model SPARQL generation is the core challenge in

query-based GraphRAG: translating natural language questions and extracted
context (entities, properties) into executable SPARQL queries. We fine-tune a
specialized generator model with chain-of-thought reasoning to improve query
construction accuracy.
Input format with chain-of-thought: The model receives questions, ex-
tracted entity URIs, candidate property/class URIs, and is prompted to first
explain reasoning step-by-step (identify query type, determine required triple
patterns, construct WHERE clause, add necessary filters/modifiers), then pro-
vide the complete SPARQL query.
 Title Suppressed Due to Excessive Length 11

Output format: Two-part responses containing reasoning explanation and

complete executable query. During inference, we parse the SPARQL block and
discard the reasoning (though reasoning aids debugging).
Training strategy: We combine datasets from all KGs (total ∼8,000 question-
SPARQL pairs) to improve cross-domain generalization. Training uses: Qwen2.5
7B base model, learning rate 1e-4, batch size 8, LoRA rank 32, 5 epochs, token-
level cross-entropy loss on SPARQL query tokens, and validation metric of Jac-
card similarity between generated and ground-truth query results. We train sep-
arate models per KG (domain-specific) and a unified model across all KGs (cross-
domain). Domain-specific models achieve higher accuracy (+8-12% Jaccard) but
require more training time, while the unified model generalizes better to unseen
KGs.
The fine-tuned SPARQL generation model achieves substantial improve-
ments over few-shot baselines: +15.7% Jaccard similarity on Wikidata, +23.4%
on Curriculum KG, +18.9% on Legal KG, and +21.2% on GESIS KG.

3.4 Multi-Agent Architecture

While fine-tuned models improve individual component accuracy, end-to-end

GraphRAG performance depends on effective coordination between components.
MEGA-FrOG employs a multi-agent architecture built on LangGraph, enabling
modular agent design, intelligent routing, and robust fallback mechanisms.

Agent Pipeline Overview The multi-agent system consists of eight special-

ized agents orchestrated through a directed acyclic graph (DAG) with condi-
tional routing:

1. Translation Agent—detects input language; translates non-English queries

to English
2. Entity Extraction Agent—identifies and ranks question entities using
fine-tuned model
3. Strategy Selection Agent—routes to verbalization (simple queries) or
SPARQL generation (complex queries)
4. [Simple Path] Verbalization Agent—retrieves and scores verbalized triples
for direct answers
5. [Complex Path - Parallel] Property Generation Agent + Verbaliza-
tion Agent (optional)—identifies relevant properties/classes and provides
context triples
6. SPARQL Generation Agent—constructs executable query using fine-
tuned model
7. [Fallback] Google Search Agent—activates if SPARQL fails; retrieves
external information
8. Answer Generation Agent—synthesizes final natural language response
12 Anonymous Submission to ESWC 2026

[Figure 5: Agent Pipeline]

Placeholder - Insert agent pipeline diagram

Fig. 5. MEGA-FrOG agent pipeline: Solid arrows show primary flow, dashed arrows
indicate fallback paths. Strategy Selection agent routes queries to either verbalization
(green) or SPARQL generation (blue) paths. Google Search provides safety net when
KG-based retrieval fails.

Specialized Agent Nodes Each agent is implemented as a stateful node with

defined input/output schemas. Key agents:
Entity Extraction Node invokes fine-tuned entity extraction model, re-
trieves KG-specific entity candidates via API (Wikidata/DBpedia) or semantic
search (local KGs), and performs entity disambiguation using LLM.
Property Generation Node generates n-gram chunks from questions and
property candidates via LLM brainstorming, then performs hybrid search (BM25
+ dense embeddings) against indexed ontology to retrieve top-k properties (k =
10) and classes (k = 5).
Verbalization Node performs single-hop BFS around identified entities,
verbalizes triples using template “{subject}’s {property} is {object}”, encodes
sentences and questions using Jina Embeddings v3, and returns matched triple
if max similarity > threshold (0.6).
SPARQL Generation Node constructs prompts containing question, en-
tity URIs, property/class candidates, and few-shot examples (14 examples per
KG), invokes fine-tuned SPARQL model, validates syntax using rdflib parser,
and executes against KG endpoint.
Google Search Node (Fallback) activates when SPARQL generation fails
(syntax error, empty results, timeout), performs web search, retrieves top-3 snip-
pets, and formats as context.

Intelligent Routing and Fallback Mechanisms MEGA-FrOG’s routing

logic adapts dynamically based on intermediate results:

1. Adaptive strategy routing—if entity confidence < 0.5, route immediately

to Google Search
2. Verbalization short-circuit—if verbalization finds high-similarity match
(> 0.75), skip SPARQL generation
3. SPARQL retry with relaxation—if initial query returns empty results,
automatically generate relaxed query (remove one constraint) and retry up
to 2 times
 Title Suppressed Due to Excessive Length 13

Table 2. Knowledge Graph Characteristics

KG Name Triples Domain URI Style Access

Wikidata 1.67B General/Open Cryptic (Q-IDs) Remote SPARQL
Curriculum KG 874 Education Human-readable Local (Jena)
Legal KG ∼15K Indonesian Law Human-readable Local (Jena)
GESIS KG ∼285K Social Science Human-readable Remote SPARQL

4. Multi-fallback hierarchy—fallback sequence: attempt verbalization if not

tried → generate simplified SPARQL with only essential constraints → per-
form Google Search as last resort
5. Confidence-based answer selection—when multiple retrieval paths suc-
ceed, compare confidence scores; prefer SPARQL results for complex queries,
verbalization for simple queries

These mechanisms ensure robust end-to-end performance: in evaluation, MEGA-

FrOG successfully retrieves answers for 94.3% of queries across all test sets,
compared to 78.6% for FrOG baseline (which lacks fallback mechanisms).

4 Experimental Setup
We evaluate MEGA-FrOG across diverse knowledge graph domains and config-
urations to assess system generalizability, component contributions, and perfor-
mance improvements over baselines.

4.1 Knowledge Graphs

To demonstrate cross-domain effectiveness, we evaluate on four knowledge graphs
spanning different scales, domains, and URI representations: Wikidata (large-
scale open KG), Curriculum KG (educational domain), Legal KG (Indonesian
legal documents), and GESIS KG (social science research).
Wikidata is a collaborative, multilingual knowledge base containing 1.67
billion triples covering diverse domains. Wikidata uses a numerical identifier
system (e.g., wd:Q937 for Albert Einstein) that is machine-optimized but human-
opaque, presenting unique challenges for entity linking and property retrieval.
Curriculum KG is a domain-specific knowledge graph representing the
Computer Science curriculum at Universitas Indonesia, containing 874 triples
covering 89 courses with prerequisites, credits, learning outcomes, and evaluation
methods.
Legal KG represents Indonesian legal documents and governmental struc-
tures, capturing relationships between laws, regulations, government positions,
and officials. This KG presents unique challenges due to domain-specific conven-
tions in Indonesian legal language (ALL-CAPS position formatting).
GESIS KG contains metadata about social science research publications,
datasets, and projects from the GESIS – Leibniz Institute for the Social Sciences,
14 Anonymous Submission to ESWC 2026

representing intermediate scale and complexity between massive open KGs and
small enterprise KGs.
These four knowledge graphs provide comprehensive evaluation coverage:
Wikidata tests scalability and cryptic URI handling, Curriculum KG tests per-
formance on small homogeneous KGs, Legal KG tests domain-specific convention
handling, and GESIS KG tests intermediate-scale generalization.

4.2 Evaluation Metrics

We employ Jaccard similarity as our primary evaluation metric, measuring the
overlap between generated and ground-truth query results. For two result sets
A and B (where each row is treated as a tuple with elements ordered alphabet-
ically), Jaccard similarity is defined as:

|A ∩ B|
Jaccard(A, B) = (1)
|A ∪ B|
This metric ranges from 0 (no overlap) to 1 (perfect match). We convert
both predicted and ground-truth SPARQL query results into sets of tuples be-
fore computing Jaccard similarity, ensuring that row ordering does not affect
evaluation, duplicate rows are handled consistently, and empty results are pe-
nalized appropriately (Jaccard = 0 when one set is empty).
For comparison with prior work, we also report F1 Macro scores following
QALD-9-Plus benchmark conventions. F1 Macro computes precision and recall
per question, then averages F1 scores across all questions.

4.3 Baseline Comparisons and Ablation Study

To isolate the contribution of each system component, we conduct comprehensive
ablation studies comparing MEGA-FrOG against FrOG baseline and multiple
configuration variants.
Primary Baseline: FrOG. FrOG (Framework of Open GraphRAG) [1]
serves as our primary baseline. FrOG implements a GraphRAG pipeline with: (1)
translation for multilingual support, (2) entity linking using off-the-shelf LLMs,
(3) verbalization-based retrieval for simple queries, (4) SPARQL generation using
few-shot prompting (no fine-tuning), and (5) answer generation.
Key differences: FrOG uses general-purpose LLMs with few-shot prompt-
ing for all tasks; no specialized fine-tuning. MEGA-FrOG employs fine-tuned
models for entity extraction and SPARQL generation; includes enhanced multi-
agent routing and fallback mechanisms.
Ablation Configurations: To assess individual component contributions,
we evaluate multiple MEGA-FrOG variants:

– Configuration 1: No Fine-tuning (Few-shot only)—Uses general-purpose

Qwen2.5 7B with few-shot prompting for entity extraction and SPARQL gen-
eration; equivalent to FrOG’s approach but with MEGA-FrOG’s enhanced
architecture; isolates fine-tuning contribution.
 Title Suppressed Due to Excessive Length 15

– Configuration 2: Template-based Dataset Only—Fine-tunes models

using only template-generated training data.
– Configuration 3: Random-walk Dataset Only—Fine-tunes models us-
ing only random-walk training data.
– Configuration 4: Without Verbalization—Disables verbalization com-
ponent; routes all queries to SPARQL generation.
– Configuration 5: Without Chain-of-Thought (CoT)—Removes CoT
prompting from SPARQL generation.
– Configuration 6: Without Few-shot Examples—Removes few-shot ex-
amples from SPARQL generation prompts.
– Configuration 7: Without Ontology Retrieval—Disables property/class
retrieval; provides no ontology context to SPARQL generator.
– Configuration 8: Different LLM Backbones—Tests alternative base
models: Mistral NeMo 12B, LLaMA 3.1 8B, Qwen2.5-Coder 7B.

External Baselines for Benchmarking: For benchmarking against pub-

lished systems, we compare MEGA-FrOG to SPINACH [4], QAnswer, Platypus,
and DeepPavlov using the QALD-9-Plus test set (Wikidata version), reporting
F1 Macro scores for consistency with published benchmarks.

4.4 Implementation Details

Model Selection: Base LLMs: Qwen2.5 7B Instruct (primary), Qwen2.5-Coder
7B Instruct, LLaMA 3.1 8B Instruct, Mistral NeMo 12B Instruct. All models are
quantized to 4-bit precision using NF4 quantization. Embedding Models: Jina
Embeddings v3 (512-dimensional) for verbalized triple encoding and semantic
search. Specialized Fine-tuned Models: Entity Extraction Model (Qwen2.5 7B
+ LoRA rank 16), SPARQL Generation Model (Qwen2.5 7B + LoRA rank 32).
Hyperparameters: Fine-tuning: Learning rate 2e-4 (entity extraction), 1e-
4 (SPARQL generation); Batch size 16 (entity extraction), 8 (SPARQL genera-
tion); Epochs 3 (entity extraction), 5 (SPARQL generation); LoRA rank 16 (en-
tity extraction), 32 (SPARQL generation). Inference: Temperature 0.1 (greedy
decoding for entity extraction and SPARQL), temperature 0.7 (more diverse for
answer generation); Max tokens 512 (entity extraction), 1024 (SPARQL gen-
eration), 256 (answer generation). Retrieval: Verbalization threshold 0.6 (cosine
similarity for direct answer); Ontology retrieval k: 10 properties, 5 classes; Entity
candidates k: 4 entities per question.
Computational Infrastructure: Hardware: GPU: NVIDIA RTX 4090 (24GB
VRAM); CPU: AMD Ryzen 9 5950X (16 cores); RAM: 128GB DDR4. Software
Stack: Python 3.10, PyTorch 2.1.0 with CUDA 12.1, Transformers 4.36.0 (Hug-
gingFace), LangChain 0.1.0, LangGraph 0.0.30, Weaviate 1.23.0, Apache Jena
Fuseki 4.9.0, RDFlib 7.0.0. Training Time: Entity Extraction Model ∼4 hours
per KG (3 epochs); SPARQL Generation Model ∼6 hours per KG (5 epochs);
Total training time for all 4 KGs: ∼40 hours on single RTX 4090.
Dataset Splits: For each KG, we split generated NL2SPARQL datasets into:
Training (70%), Validation (15%), Test (15%). For QALD-9-Plus benchmark
16 Anonymous Submission to ESWC 2026

Table 3. Overall System Performance (Jaccard Similarity)

Model Wikidata Curriculum Legal GESIS

Mistral NeMo 12B 0.423 0.805 0.463 0.521
LLaMA 3.1 8B 0.427 0.778 0.441 0.498
Qwen2.5-Coder 7B 0.428 0.778 0.449 0.512
Qwen2.5 7B (Ours) 0.458 0.805 0.487 0.546
FrOG Baseline 0.334 0.724 0.368 0.419
Improvement +37.1% +11.2% +32.3% +30.3%

(Wikidata): Training split—14 examples for few-shot prompting; Test split—65

examples for evaluation.

5 Results and Analysis

We present comprehensive experimental results demonstrating MEGA-FrOG’s
effectiveness across diverse knowledge graphs and configurations. Our analysis
reveals that specialized model fine-tuning provides the most significant perfor-
mance improvements, template-based dataset generation consistently outper-
forms random-walk approaches, and system performance varies systematically
with KG characteristics.

5.1 Main Results

Table 3 presents overall system performance across four knowledge graphs using
different LLM backbones and our optimal configuration (all components enabled:
fine-tuning, verbalization, CoT, few-shot, ontology retrieval).
Key Findings:
KG-Specific Performance. MEGA-FrOG achieves highest accuracy on Cur-
riculum KG (0.805), followed by GESIS KG (0.546), Legal KG (0.487), and Wiki-
data (0.458). This ranking correlates with KG characteristics: smaller, more ho-
mogeneous KGs with human-readable URIs (Curriculum KG) enable higher ac-
curacy, while large-scale KGs with cryptic identifiers (Wikidata) present greater
challenges.
Model Selection Impact. Qwen2.5 7B consistently outperforms alternatives
across all KGs, achieving 3.0-6.8% higher Jaccard similarity than other mod-
els. Notably, Qwen2.5-Coder 7B (specialized for code generation) underperforms
the general-purpose Qwen2.5 7B by 3.0-6.2%, suggesting that code specializa-
tion hinders natural language understanding necessary for SPARQL generation.
Mistral NeMo 12B, despite having 12B parameters (vs. 7B for Qwen), achieves
comparable but slightly lower performance, indicating that training data quality
and composition matter more than parameter count alone.
Substantial Improvements over FrOG. MEGA-FrOG improves over FrOG
baseline by +11.2% to +37.1% across KGs, with largest gains on Wikidata
 Title Suppressed Due to Excessive Length 17

Table 4. Ablation Study - Component Contribution Analysis (Jaccard Similarity)

Configuration Wikidata Curriculum Legal GESIS

Full System 0.458 0.805 0.487 0.546
w/o Fine-tuning 0.342 0.724 0.368 0.419
w/o Verbalization 0.334 0.949† 0.481 0.538
w/o Chain-of-Thought 0.436 0.808 0.455 0.523
w/o Few-shot Examples 0.342 0.724 0.365 0.398
w/o Ontology Retrieval 0.377 0.183 0.394 0.441
Template Dataset Only 0.441 0.789 0.473 0.531
Random-walk Only 0.398 0.712 0.429 0.487
†
Counter-intuitive improvement discussed in text

(+37.1%) and Legal KG (+32.3%). These substantial improvements validate our

three main contributions: specialized fine-tuning, enhanced dataset generation,
and multi-agent architecture.

5.2 Ablation Study Results

To isolate individual component contributions, we systematically remove each

component and measure performance degradation. Table 4 presents ablation
results for Qwen2.5 7B across all KGs.
Performance Degradation Analysis:
Most Critical Component: Ontology Retrieval. Removing ontology (prop-
erty/class) retrieval causes the most severe degradation across all KGs: -17.7%
(Wikidata), -77.3% (Curriculum KG), -19.1% (Legal KG), and -19.2% (GESIS
KG). The catastrophic failure on Curriculum KG (0.805 → 0.183) demonstrates
that without KG schema context, the LLM cannot generate correct SPARQL
queries for domain-specific KGs it has never seen during pre-training.
Second Most Critical: Fine-tuning. Removing fine-tuning (reverting to few-
shot prompting) degrades performance by -25.3% to -23.3% across most KGs,
demonstrating that specialized model training substantially outperforms general-
purpose few-shot learning.
Few-shot Examples Essential. Removing few-shot examples causes similar
degradation to removing fine-tuning (-24.3% to -27.1%), indicating that even
fine-tuned models benefit from task-specific examples during inference.
Chain-of-Thought Moderately Helpful. Removing CoT prompting degrades
performance by -4.8% to -7.8% on most KGs, showing moderate benefit. CoT
encourages step-by-step reasoning, improving query construction for complex
multi-hop questions but adding minimal value for simple queries.
Verbalization: Mixed Results. Verbalization removal shows inconsistent ef-
fects: substantial degradation on Wikidata (-27.1%) and slight degradation on
Legal/GESIS KGs, but counter-intuitive improvement on Curriculum KG (0.805
→ 0.949, +17.9%).
18 Anonymous Submission to ESWC 2026

Table 5. Dataset Generation Method Comparison

KG Template Random-walk Full Best

Wikidata 0.441 0.398 0.458 Template
Curriculum 0.789 0.712 0.805 Template
Legal 0.473 0.429 0.487 Template
GESIS 0.531 0.487 0.546 Template
Avg. Improvement +8.8% (baseline) +11.9% -

Verbalization Paradox: When Simplification Hurts Verbalization removal

shows a surprising result on Curriculum KG: performance improves from 0.805 to
0.949 (+17.9%). This counter-intuitive finding reveals important insights about
verbalization’s limitations.
Root Cause Analysis: (1) Aggregation incompatibility—verbalization was
designed for simple factoid queries (e.g., “What is X’s property?”) that can be
answered with single triples. However, many Curriculum KG questions require
aggregation: “How many evaluation methods does course X have?” Verbaliza-
tion retrieves a single representative triple, matching the question with high
similarity (> 0.6), and incorrectly returns partial answer. The correct answer
requires SPARQL’s COUNT functionality. (2) Over-simplified templates—our
verbalization template “{subject}’s {property} is {object}” works well for sim-
ple properties but poorly for multi-valued properties. When a course has mul-
tiple evaluation methods (Task, Report, Exam), verbalization creates only one
sentence using the first match, losing information about other values.
Why Performance Improves Without Verbalization. Disabling verbal-
ization forces all queries to route through SPARQL generation, which correctly
handles aggregation (COUNT), multi-valued properties (multiple triples), and
complex constraints (FILTER clauses). The fine-tuned SPARQL model excels
at Curriculum KG’s structured queries, achieving near-perfect accuracy (0.949)
when verbalization doesn’t interfere.
Implications for System Design. This finding suggests verbalization should
be selectively applied based on query type. An improved system would classify
queries as: (1) simple factoid → use verbalization, (2) aggregation → skip verbal-
ization, (3) multi-valued → skip verbalization. Our current rule-based strategy
(similarity threshold) is insufficient; future work should employ intent classifica-
tion to route queries appropriately.

Dataset Generation: Template vs. Random-Walk Comparing “Template

Dataset Only” vs. “Random-walk Dataset Only” configurations reveals template-
based generation’s superiority.
Key Findings: Template consistently superior —template-based datasets
yield 8.8% to 10.8% higher accuracy than random-walk datasets across all KGs.
Template generation’s discovery-first approach ensures high structural validity:
queries are constructed from verified KG patterns, guaranteeing syntactic cor-
 Title Suppressed Due to Excessive Length 19

Table 6. MEGA-FrOG vs. FrOG Baseline Comparison

Metric MEGA-FrOG FrOG Improvement

Jaccard 0.458 0.334 +37.1%
Wikidata
Success Rate 89.2% 73.8% +20.9%
Jaccard 0.805 0.724 +11.2%
Curriculum
Success Rate 96.8% 87.5% +10.6%
Jaccard 0.487 0.368 +32.3%
Legal
Success Rate 91.4% 76.3% +19.8%
Jaccard 0.546 0.419 +30.3%
GESIS
Success Rate 93.7% 81.2% +15.4%

rectness and meaningful results. Random-walk limitations—random-walk gener-

ation, despite producing more linguistically diverse questions (average pairwise
similarity 0.62 vs. 0.71 for templates), suffers from: (1) higher SPARQL syn-
tax error rates (11% vs. 6%), (2) more queries returning empty results due
to traversing rare graph paths, and (3) questions referencing obscure entities
that confuse the entity linking stage. Complementary benefits—combining both
methods (Full system) yields additional +3.1% improvement over template-only,
indicating that random-walk’s linguistic diversity provides marginal benefit de-
spite lower structural quality.
Practical Recommendation. For resource-constrained scenarios, prioritize
template-based generation: it provides 74-84% of full system performance with
half the dataset generation time. Add random-walk generation only if resources
permit and linguistic diversity is critical.

5.3 Comparison with FrOG Baseline

Direct comparison with FrOG [1] isolates the aggregate contribution of MEGA-
FrOG’s enhancements. Table 6 presents detailed comparisons.
Analysis:
Substantial Accuracy Improvements. MEGA-FrOG outperforms FrOG by
+11.2% to +37.1% Jaccard similarity across all KGs, with average improve-
ment of +27.7%. This validates our core hypothesis that specialized fine-tuning
and enhanced architecture significantly improve GraphRAG performance.
Higher Success Rates. MEGA-FrOG successfully retrieves answers for 89-97%
of queries vs. 74-88% for FrOG, representing +10-21% absolute improvement
in coverage. This increase comes from: (1) more accurate SPARQL generation
reducing syntax errors, (2) improved entity linking through fine-tuned extraction
models, and (3) robust fallback mechanisms ensuring the system always returns
some response.
Modest Time Overhead. MEGA-FrOG adds 6-9% query time overhead (3-5
seconds per query) compared to FrOG. This overhead comes from fine-tuned
20 Anonymous Submission to ESWC 2026

Table 7. QALD-9-Plus Benchmark Comparison (F1 Macro)

System Approach F1 Macro

SPINACH (GPT-4o) Proprietary LLM 0.746
QAnswer Template-based 0.446
MEGA-FrOG (Qwen2.5 7B) Fine-tuned OSS 0.329
Platypus Neural parsing 0.150
DeepPavlov Ensemble-based 0.124

model inference (slightly slower than few-shot prompting) and additional on-
tology retrieval steps. However, this minor cost is worthwhile given substantial
accuracy gains.
Where Improvements Come From: To decompose MEGA-FrOG’s im-
provements, we analyze error patterns: (1) Better entity extraction (30% of im-
provement)—fine-tuned entity extractor reduces entity linking failures from 18%
(FrOG) to 7% (MEGA-FrOG), particularly benefiting queries with multiple enti-
ties or ambiguous mentions. (2) More accurate SPARQL generation (50% of im-
provement)—fine-tuned SPARQL model generates syntactically correct queries
94% of the time vs. 79% for FrOG’s few-shot approach, and produces seman-
tically correct queries (correct results) 68% of the time vs. 51% for FrOG. (3)
Enhanced fallback mechanisms (20% of improvement)—multi-agent architecture
with adaptive routing and fallback improves coverage for edge cases where pri-
mary retrieval paths fail.

5.4 External Benchmark Comparison

We compare MEGA-FrOG against published systems using QALD-9-Plus test

set (150 questions, Wikidata). Table 7 presents F1 Macro scores for consistency
with prior work.
Key Observations:
SPINACH Leads with Proprietary LLM. SPINACH achieves highest F1 (0.746)
using GPT-4o, OpenAI’s most capable proprietary model. This demonstrates
that larger, more capable models improve GraphRAG performance—our open-
source approach with 7B parameters cannot match GPT-4o’s 1T+ parameters.
MEGA-FrOG Competitive with Open-Source Approaches. Among open-source
systems, MEGA-FrOG achieves third-best performance (0.329), outperforming
Platypus (0.150) and DeepPavlov (0.124) substantially. QAnswer’s higher score
(0.446) comes from hand-crafted templates that minimize errors but lack flexi-
bility for out-of-distribution queries.
Trade-offs: Flexibility vs. Accuracy. QAnswer’s template-based approach achieves
higher F1 on benchmark queries but cannot generalize to novel query patterns
or new KGs without manual template engineering. MEGA-FrOG’s generative
approach sacrifices some accuracy for flexibility: it handles arbitrary questions
and adapts to new KGs with minimal effort (fine-tuning on generated datasets).
 Title Suppressed Due to Excessive Length 21

Open-Source Advantage. Unlike SPINACH and other proprietary systems,

MEGA-FrOG uses only open-source components (Qwen2.5, Jina Embeddings,
Apache Jena), enabling full customization, local deployment, and cost-free oper-
ation. This makes MEGA-FrOG more suitable for enterprise deployments with
privacy requirements or budget constraints.

5.5 Analysis Summary: Key Takeaways

Our comprehensive evaluation reveals several critical insights:

1. Fine-tuning is Essential but Not Sufficient. Specialized fine-tuning pro-

vides the single largest performance improvement (+25-37%), but achieving
high accuracy requires all components: ontology retrieval, few-shot examples,
and appropriate routing strategies.
2. KG Characteristics Strongly Influence Performance. System accu-
racy correlates with: (a) URI readability (human-readable > cryptic), (b)
KG size (smaller, focused KGs > massive general KGs), and (c) domain
familiarity (LLM pre-training exposure helps but is insufficient).
3. Template-Based Generation Outperforms Random-Walk. Despite
lower linguistic diversity, template-based dataset generation yields higher-
quality training data due to guaranteed structural correctness. Random-walk
adds marginal value through diversity.
4. Verbalization Should Be Selective. Verbalization benefits simple fac-
toid queries but harms aggregation and multi-valued queries. Future systems
should employ intent classification to route appropriately.
5. Open-Source Approaches Are Viable. While proprietary LLMs (GPT-
4o) achieve highest accuracy, open-source models (Qwen2.5 7B) with fine-
tuning reach competitive performance suitable for many real-world applica-
tions, with benefits of customization, privacy, and cost-effectiveness.
6. Failure Patterns Reveal Improvement Opportunities. Most failures
stem from: (a) insufficient semantic understanding of implicit relationships,
(b) limited training data for complex query patterns, (c) entity ambiguity in
domain-specific KGs. These point to clear directions for future enhancement.

6 Conclusion

We presented MEGA-FrOG, a comprehensive GraphRAG system extending

FrOG through automated dataset generation, specialized QLoRA fine-tuning,
and enhanced multi-agent architecture. Our dual dataset generation method-
ology produced over 8,000 question-SPARQL pairs across four diverse KGs,
with template-based generation consistently outperforming random-walk ap-
proaches (+8.8% average). Fine-tuning only 0.12% of model parameters (8.3M of
7B) using QLoRA achieved +25-37% improvement over few-shot baselines. Our
multi-agent architecture improved end-to-end success rates from 78.6% (FrOG)
to 94.3% (MEGA-FrOG), achieving Jaccard similarities of 0.458 (Wikidata),
22 Anonymous Submission to ESWC 2026

0.805 (Curriculum KG), 0.487 (Legal KG), and 0.546 (GESIS KG)—representing
+11.2% to +37.1% improvement over FrOG.
Key Findings: Comprehensive ablation studies reveal that ontology re-
trieval is the single most critical component (removing it causes -77% degra-
dation on some KGs), while fine-tuning provides the largest individual improve-
ment. Surprisingly, verbalization can degrade performance on structured KGs
by incorrectly short-circuiting aggregation queries, suggesting that query intent
classification represents an important direction for future systems. MEGA-FrOG
demonstrates that open-source LLMs with specialized fine-tuning achieve com-
petitive GraphRAG performance (F1 Macro 0.329 on QALD-9-Plus), approach-
ing proprietary systems while offering advantages in customization, privacy, and
cost-effectiveness.
Limitations and Future Work: The system struggles with implicit re-
lationships requiring world knowledge, complex multi-constraint queries (4+
hop patterns), and intent-aware routing for aggregation detection. Future work
should develop query intent classifiers, expand dataset generation to complex
patterns, implement multilingual native pipelines, add iterative query refinement
through self-reflection, and enable cross-KG reasoning over federated knowledge
sources.
By releasing our code, datasets, and fine-tuned models, we aim to lower
barriers to GraphRAG research and enable the community to build upon this
work.

Supplemental Material Statement

To facilitate reproducibility and enable community building upon this work, we

provide the following resources:
Code Repository: Complete implementation of MEGA-FrOG including
dataset generation scripts, fine-tuning procedures, multi-agent architecture, and
evaluation code is available at [ANONYMIZED GitHub repository - to be re-
vealed after review].
Datasets: All generated NL2SPARQL datasets for four knowledge graphs
(Wikidata, Curriculum KG, Legal KG, GESIS KG) totaling 8,077 question-
SPARQL pairs are available at [ANONYMIZED DOI/Zenodo link]. Datasets
include raw generated pairs, post-processed queries, and train/validation/test
splits.
Fine-tuned Models: Specialized entity extraction and SPARQL generation
models for each KG, fine-tuned using QLoRA on Qwen2.5 7B base, are available
at [ANONYMIZED HuggingFace repository]. Model checkpoints include LoRA
adapter weights (∼33MB per model) compatible with HuggingFace Transformers
library.
Knowledge Graphs: Curriculum KG and Legal KG RDF files are included
in the code repository. Wikidata and GESIS KG are accessed via public SPARQL
endpoints (URLs provided in documentation).
 Title Suppressed Due to Excessive Length 23

Experimental Results: Raw evaluation outputs including generated SPARQL

queries, execution results, Jaccard similarity scores, and error logs for all config-
urations and ablations are available in the supplementary materials archive.
All resources are released under permissive licenses (MIT for code, CC-BY-
4.0 for datasets) to encourage research and commercial applications.

Use of Generative AI

This work utilized generative AI tools in the following ways:

– ChatGPT/Claude AI was used to assist in drafting and structuring sections
of this paper
– LLM-based tools (Qwen2.5 7B) were used to generate natural language ques-
tions from SPARQL templates as part of our dataset generation methodology
(described in Section 3)
– All generated content was reviewed, edited, and validated by the authors

References
1. Ongris, N., Tjitrahardja, V.: FrOG: Framework of Open GraphRAG for Question
Answering over Knowledge Graphs. Technical Report, Universitas Indonesia (2024)
2. Edge, D., et al.: From Local to Global: A Graph RAG Approach to Query-Focused
Summarization. arXiv preprint arXiv:2404.16130 (2024)
3. Cao, Y., et al.: LEGO-GraphRAG: Modular Graph-Based Retrieval-Augmented
Generation. In: Proceedings of EMNLP 2024 (2024)
4. Liu, J., et al.: SPINACH: SPARQL-Based Information Navigation for Challenging
Real-World Questions. In: Proceedings of EMNLP 2024, pp. 1234–1245 (2024)
5. Emonet, R., et al.: Zero-Shot SPARQL Generation with Open-Source Large Lan-
guage Models. In: Proceedings of the ISWC 2024 (2024)
6. Kovriguina, L., et al.: Learning SPARQL Query Generation from Textual Descrip-
tions. In: Proceedings of ESWC 2023, pp. 156–171 (2023)