JOURNAL OF LATEX CLASS FILES, VOL. 18, NO.

9, SEPTEMBER 2020 1

A Survey of Text-to-SQL in the Era of LLMs:

Where are we, and where are we going?
Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuxin Zhang, Ju Fan,
Guoliang Li, Fellow, IEEE, Nan Tang, and Yuyu Luo
Text-to-SQL Handbook: [Link] Handbook

Input Output
arXiv:2408.05109v5 [[Link]] 15 Jun 2025

NL Query

SQL Query

Database

(a) Text-to-SQL Model Evolution Stream Graph

ider DAIL

Other 1.8%
[Link] -SQ

Desc

Anno
DINS L

Agg ect
L QL

12.4 ect
-Asc
Gr

4.0%0.7% ORD %
Q

Inc
WikiS

tation Spider BY
C3SQ

fun
a

%
orr
orr
L RE ph Co

c 2 .5%
RD r

SD ix - nd
e

Inc
id

Mo 3. itio

.5%
SQ T5
Sp

6% n ss

Other 1.8%
et de L ce
s l Ex .8%
BI

ta Zo 5
Da

5.7
t

ER
o tinc
Dash tion

Exec acy Value

4.0%
Dis %
board

Accu

SELECT
11.6% 2.9
liza

WHERE ie 2.
2%
ution
25.5% nt
Insuffic 1.5%
r
Visua

Metrics
Analysis

15.2%
NL2SQL360
Join Type 0t.4%
Error Order
Aggregation 0.7%
Lifecycle Incorrec
Exact -M

FROM GROUP BY Exc

board

Accura

3.3% nt e
ie 15.3% 16.6% 10.5 ss
ffic Equality %
Ev Insu .6%
Leader

alu
cy

3
atch

ter
15.9%
ato ss In
r Fil ce co
4. rrec
Qu val

se Ex .0%

Ha
Ord
ba s 3% t
an uat

pt vs. 2.2%
8

vin
3.3% t In

Intersect vs. And

E

ta in
tit ion

er li %

g0
Da oma

No
at

.7%
=

mit
ive

5.1
Qua xity D

4.5%
vs.
lit omple GROUP By vs. Subquery Miss GROUP BY
Evalu ative SQL C

vs.
ics
Like
cterist Exce
0.4% 0.7%
atio

In
n / Chara JOIN vs. Subquery 0.4% Redundant GROUP BY 0.4%

(b) Benchmarks and Training Data Synthesis (c) Text-to-SQL Evaluation (d) Text-to-SQL Error Analysis
Fig. 1: An Overview of the Survey: The Lifecycle of the Text-to-SQL Task.

Abstract—Translating users’ natural language queries (NL) Index Terms—Natural Language to SQL, Database Interface,
into SQL queries (i.e., Text-to-SQL, a.k.a. NL2SQL) can sig- Large Language Models, Text-to-SQL.
nificantly reduce barriers to accessing relational databases and
support various commercial applications. The performance of I. I NTRODUCTION
Text-to-SQL has been greatly enhanced with the emergence of
Large Language Models (LLMs). In this survey, we provide
a comprehensive review of Text-to-SQL techniques powered
by LLMs, covering its entire lifecycle from the following four
N ATURAL Language to SQL (i.e.,Text-to-SQL), which
converts a natural language query (NL) into an SQL
query, is a key technique toward lowering the barrier to
aspects: (1) Model: Text-to-SQL translation techniques that tackle accessing relational databases [1]–[7]. This technique supports
not only NL ambiguity and under-specification, but also properly
map NL with database schema and instances; (2) Data: From the various applications such as business intelligence and natural
collection of training data, data synthesis due to training data language interfaces for databases, making it a key step toward
scarcity, to Text-to-SQL benchmarks; (3) Evaluation: Evaluat- democratizing data science [8]–[18]. Recent advancements
ing Text-to-SQL methods from multiple angles using different in language models have significantly extended the frontiers
metrics and granularities; and (4) Error Analysis: analyzing of research and application in Text-to-SQL. Concurrently, the
Text-to-SQL errors to find the root cause and guiding Text-to-
SQL models to evolve. Moreover, we offer a rule of thumb for trend among database vendors to offer Text-to-SQL solutions
developing Text-to-SQL solutions. Finally, we discuss the research has evolved from a mere notion to a necessary strategy [19],
challenges and open problems of Text-to-SQL in the LLMs era. [20]. Therefore, we need to understand the fundamentals,
techniques, and challenges regarding Text-to-SQL.
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 2

In this survey, we systematically review recent Text-to-SQL comparing different strategies within this scope. We are
techniques through a new framework, as shown in Figure 1. the first survey to provide a modular summary of methods
• Text-to-SQL with Language Models. We will first re- and provide detailed analyses for each key module (e.g.,
view existing Text-to-SQL solutions from the perspective database content retrieval).
of language models, categorizing them into four major • We highlight the importance of evaluating Text-to-SQL
categories (see Figure 1(a)). We will then focus on the methods in a multi-angle way, analyze the Text-to-SQL
recent advances in Pre-trained Language Models (PLMs) error patterns, and provide a two-level error taxonomy.
and Large Language Models (LLMs) for Text-to-SQL. • We provide practitioners with a roadmap for optimizing
• Benchmarks and Training Data Synthesis. Undoubt- LLMs to Text-to-SQL and a decision flow for selecting
edly, the performance of PLM- and LLM-based Text-to- the suitable Text-to-SQL modules for various scenarios.
SQL models is highly dependent on the amount and qual- Contributions. We make the following contributions.
ity of the training data. Therefore, we will first summarize
• Text-to-SQL with Language Models. We comprehensively
the characteristics of existing benchmarks and analyze
review existing Text-to-SQL techniques from a lifecycle
their statistical information (e.g., database complexity) in
perspective (Figure 1). We introduce the Text-to-SQL
detail. We will then discuss methods for collecting and
task definition, discuss challenges (Figure 2), provide a
synthesizing high-quality training data, emphasizing this
taxonomy of Text-to-SQL solutions based on language
as a research opportunity (see Figure 1(b)).
models (Figure 3), and summarize the key modules of
• Evaluation. Comprehensively evaluating Text-to-SQL
language model-powered Text-to-SQL solutions (Figure 5
models is crucial for optimizing and selecting models for
and Table I). Next, we elaborate on each module of lan-
different usage scenarios. We will discuss the multi-angle
guage model-powered Text-to-SQL methods, including
evaluation and scenario-based evaluation for the Text-to-
the pre-processing strategies (Section IV), Text-to-SQL
SQL task (see Figure 1(c)). For example, we can assess
translation methods (Section V), and post-processing
the Text-to-SQL model in specific contexts by filtering
techniques (Section VI).
benchmarks based on SQL characteristics, NL variants,
• Benchmarks and Training Data Synthesis. We summarize
database domains, and so on.
existing Text-to-SQL benchmarks based on their charac-
• Text-to-SQL Error Analysis. Error analysis is essential
teristics (Figure 10). We analyze each benchmark in depth
in Text-to-SQL research for identifying limitations and
and discuss its pros and cons (Table II). (Section VII)
improving the model robustness. We review existing error
• Evaluation and Errors Analysis. We highlight the impor-
taxonomies, analyze their limitations, and propose prin-
tance of evaluation in developing practical Text-to-SQL
ciples for designing comprehensive taxonomies for Text-
solutions. We review widely used evaluation metrics and
to-SQL output errors. Using these principles, we create a
toolkits for assessing Text-to-SQL solutions. We provide
two-level error taxonomy and utilize it to summarize and
a taxonomy to summarize typical errors produced by
analyze Text-to-SQL output errors (see Figure 1(d)).
Text-to-SQL methods. (Section VIII)
In addition to the above, we will provide practical guidance • Practical Guidance for Developing Text-to-SQL Solu-
for developing Text-to-SQL solutions, including a roadmap for tions. We provide a roadmap for optimizing existing
optimizing LLMs for Text-to-SQL tasks and a decision flow LLMs to Text-to-SQL tasks (Figure 11(a)). In addition,
for selecting Text-to-SQL modules tailored to various Text-to- we design a decision flow to guide the selection of ap-
SQL scenarios. Finally, we will discuss key open problems in propriate modules for different scenarios (Figure 11(b)).
the field, such as open Text-to-SQL tasks, cost-effective Text- • Open Problems in Text-to-SQL. We analyze the lim-
to-SQL with LLMs, and trustworthy Text-to-SQL solutions. itations of LLM-based methods and discuss new re-
Differences from Existing Surveys. Our survey distinguishes search opportunities, including the open-world Text-to-
itself from existing Text-to-SQL surveys [21]–[28] and tutori- SQL problem and cost-effective solutions (Section X).
als [29]–[31] in five aspects. • Text-to-SQL Handbook. We maintain an online handbook

• We systematically review the entire lifecycle of Text-

([Link] Handbook) to
to-SQL problem, as shown in Figure 1. This lifecycle help readers stay current with Text-to-SQL advancements.
includes various Text-to-SQL translation methodologies
II. T EXT- TO -SQL P ROBLEM AND BACKGROUND
powered by language models (Figure 1(a)), training data
collection and synthesis methods (Figure 1(b)), multi- In this section, we first formalize the definition of the Text-
angle and scenarios-based evaluations (Figure 1(c)), and to-SQL task (Section II-A). We then introduce the workflow
Text-to-SQL error analysis techniques (Figure 1(d)). of how humans perform the Text-to-SQL task (Section II-B)
• We provide a more detailed and comprehensive summary and discuss the key challenges (Section II-C). Finally, we
of the inherent challenges in Text-to-SQL. Additionally, describe the evolution of Text-to-SQL solutions based on the
we analyze the technical challenges when developing development of language models (Section II-D).
a robust Text-to-SQL solution for real-world scenarios,
which are often overlooked in other surveys. A. Problem Formulation
• We particularly focus on recent advances in LLM-based Definition 1 (Natural Language to SQL (Text-to-SQL)).
Text-to-SQL methods, summarizing key modules and Natural Language to SQL (Text-to-SQL), also known as
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 3

NL Query:
further understand the overall purpose of the NL query. In this
Find the names of all customers who checked out books on exactly 3 different genres
on Labor Day in 2023. case, the DBA should retrieve a list of customer names based
Database:
① ② ① ②
on specific borrowing behavior on a particular date.
Customer Book Step-2: Finding Relevant Tables, Columns, and Cell Values:
CustomerId Name ... BookId Title LiteraryGenre SubjectGenre ...
Novel Magic Next, the DBA examines the database schema and contents
③ ... ...
BookOrder Table Linking
to identify the relevant tables, columns, and cell values for
CustomerId BookId OrderDate ... Columns Linking
constructing the SQL. For example, the DBA may determine
01/05/23 Database Content

④ ... ⑤ Additional Information

that the “Customer” and “Book” tables are relevant based
Additional Information: Note that Labor Day stand for May 1 Foreign Key

SQL:
on their understanding of the NL (see Figure 2(a)-①). The
SELECT Name
SELECT Name
DBA then decides which columns should be mentioned.
FROM Customer
FROM Customer

NATURAL JOIN BookOrder

WHERE CustomerId = (SELECT CustomerId
For example, the keyword “genres” can refer to either
NATURAL JOIN Book
FROM BookOrder NATURAL JOIN Book

WHERE OrderDate=’01/05/23’ WHERE OrderDate=’01/05/23’

“LiteraryGenre” or “SubjectGenre” (see Figure 2(a)-②).
GROUP BY CustomerId, Name
GROUP BY CustomerId

HAVING COUNT(DISTINCT SubjectGenre)=3 HAVING COUNT(DISTINCT SubjectGenre)=3)

LiteraryGenre LiteraryGenre
Furthermore, the DBA should interpret “Labor Day in 2023”
NL Query:
(a) An Example of Text-to-SQL (Ambiguous NL) based on the context. In the US, “Labor Day in 2023” refers
Find the names of all customers who checked out books on exactly 3 different genres to “September 4th, 2023”, while in China, it refers to “May
on Labor Day in 2023. ① ①
② 1st, 2023”. This judgment relies on domain knowledge or
Database: ②
Customer Book
available additional information (see Figure 2(a)-⑤).
CustomerId CName ... BookId Title LiteraryGenre SubjectGenre ... Note that Step-2 aligns with the concepts of schema linking,
Novel Magic
④ ③ ... ... database content retrieval, and additional information acqui-
Account BookOrder sition in recent Text-to-SQL solutions powered by language
AccId AccName Password CustomerId BookId AccId OrderDate ...
③ May 1st 2023 models (please refer to Figure 5 for more details).
Additional Information: Note that Labor Day stand for May 1 ⑤ ...
SQL: Step-3: Writing SQL based on NL and DB Understanding:
SELECT CName

FROM Customer
Finally, the DBA writes the corresponding SQL based on the
NATURAL JOIN Account NATURAL JOIN BookOrder NATURAL JOIN Book

WHERE OrderDate = 'May 1st 2023'

insights gained in Steps-1 and -2. This process, known as
GROUP BY CustomerId, CName

HAVING COUNT(DISTINCT SubjectGenre) = 3 “Text-to-SQL Translation”, relies heavily on the DBA’s SQL
(b) An Example of Text-to-SQL (DB Schema Updated)
expertise. However, this process can be very challenging due
Fig. 2: Examples of the Text-to-SQL Task and Its Challenges. to the ambiguity of the NL or the complexity of the database.
For example, as shown in Figure 2(a), despite understanding
the need to link the Customer and Book tables, one must
NL2SQL, is the task of converting natural language queries be familiar with the usage and norms of employing either a
(NL) into corresponding SQL queries (SQL) that can be exe- natural join or a subquery. In addition, there may be multiple
cuted on a relational database (DB). Specifically, given an NL possible SQL queries because “genres” can refer to either
and a DB, the goal of Text-to-SQL is to generate an SQL that “LiteraryGenre” or “SubjectGenre”.
accurately reflects the user’s intent and returns the appropriate Takeways. From the above steps, we intuitively identify three
results when executed on the database. inherent challenges in the Text-to-SQL task: the uncertainty
of the natural language, the complexity of the database, and
Discussion. In some cases, the corresponding SQL query to an
the translation from the “free-form” natural language queries
NL may be multiple due to the ambiguity or underspecification
to the “constrained and formal” SQL queries.
of the NL, or ambiguity in the database schema. In addition,
even when the NL, database schema and database content are
clear and specific, there may still be multiple equivalent SQL C. Text-to-SQL Task Challenges
queries that can satisfy the given NL question. In this section, we will first discuss the fundamental chal-
lenges of the Text-to-SQL task. We will then analyze the tech-
B. Text-to-SQL Human Workflow nical challenges, i.e., the challenges we face when developing
When professional users (e.g., DBAs) perform the Text- a strong Text-to-SQL solution in real-world scenarios.
to-SQL task, they first interpret the NL question, examine C1: Uncertain Natural Language Query. Natural language
the database schema and contents, and then construct the queries often contain uncertainties due to ambiguity and
corresponding SQL based on their SQL expertise. Below, we underspecification [32]. In Text-to-SQL tasks, the challenges
outline this process in detail, as illustrated in Figure 2(a). related to NL can be summarized as follows:
Step-1: Understanding Natural Language Query: Given the • Lexical Ambiguity: This occurs when a single word has
NL query “Find the names of all customers who checked multiple meanings. For example, the word “bat” can refer
out books on exactly 3 different genres on Labor Day in to an animal or a baseball bat or the action of swinging.
2023”, the DBA’s first task is to grasp the user’s intent and • Syntactic Ambiguity: This occurs when a sentence can
identify key components. Key elements include: 1) Entities be parsed in multiple ways. For example, in the sentence
or Attributes: “names”, “customers”, “books”, and “genres”; “Mary saw the man with the telescope”, the phrase “with
2) Temporal Context: “Labor Day in 2023”; and 3) Specific the telescope” can mean either that Mary used a telescope
Conditions: “exactly 3 different genres”. Then, the DBA may to see the man or that the man had a telescope.
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 4

Type Level

NL Challenges Token-level Recognition Synonym Recognition Semantic Understanding Domain Knowledge Query Recognition Multi-turn Dialogues
DB Challenges Single-table Queries Simple Multiple Tables Multiple Tables with Complex Schema Massive Tables and Values Real-world Databases
Text-to-SQL Challenges Single-table SQL Multi-table SQL Advanced SQL Feature Support Adapting to Changed Schema Efficient SQL Generation
(a) The Definition of Challenges Levels
Large Language Model Era
Challenges

Solving Capacity
Experts Novices
Experts Novices
Experts Novices PLM-based Methods LLM-based Methods
Target Users Experts Novices Neural Network-based Methods NL Challenges
Text-to-SQL Methods Rule-based Methods NL Challenges DB Challenges
NL Challenges DB Challenges
NL Challenges Text-to-SQL Challenges
DB Challenges DB Challenges Text-to-SQL Challenges
Capabilities
Text-to-SQL Challenges Text-to-SQL Challenges ④ LLM
③ Pre-trained LM e.g. GPT4
② Neural LM
NLP Methods ① Statistical LM e.g. Bert, T5
e.g. LSTM Year
e.g. N-gram
1990s 2013 2018 2020
(b) The Evolution of Text-to-SQL Solutions
Fig. 3: The Evolution of Text-to-SQL Solutions from the Perspective of Language Models.

• Under-specification: This occurs when linguistic expres- intent, leading to ambiguity in determining appropriate
sions lack sufficient detail to convey specific intentions SQL translation (see the example in Figure 2(a)).
or meanings clearly. For example, “Labor Day in 2023” • Database Schema Dependency: The Text-to-SQL trans-
refers to September 4th in the US but May 1st in China. lation is highly dependent on the underlying database
schema. As shown in Figure 2 (a) and (b), the same NL
C2: Complex Database and Dirty Content. The Text- may produce different SQL queries based on schema vari-
to-SQL task requires a deep understanding of the database ations. This requires Text-to-SQL models to bridge gaps
schema, including table names, columns, relationships, and between training data and real-world schema differences.
data attributes. The complexity of modern schemas and large
Beyond the intrinsic challenges, developers must also over-
data volumes make this task especially challenging.
come several technical obstacles to build reliable and efficient
• Complex Relationships Among Tables: Databases often Text-to-SQL systems, as discussed below.
contain hundreds of tables with complex interrelation-
ships. Text-to-SQL solutions must accurately comprehend C4: Technical Challenges in Developing Text-to-SQL So-
and leverage these relationships when generating SQL. lutions. Developing robust Text-to-SQL solutions requires
• Ambiguity in Attributes and Values: Ambiguous values addressing several key technical challenges, including:
and attributes in a database can complicate Text-to-SQL • Cost-effective Solution: Deploying Text-to-SQL models,
systems’ ability to identify the correct context. particularly those using large language models, demands
• Domain-Specific Schema Designs: Different domains of- significant resources, such as hardware and/or API costs.
ten have unique database designs and schema patterns. Achieving an optimal balance between model perfor-
The variations in schema design across domains make it mance and cost efficiency remains a crucial challenge.
difficult to develop a one-size-fits-all Text-to-SQL model. • Model Efficiency: A trade-off often exists between model
• Large and Dirty Database Values: Efficiently handling size and performance, with larger models generally yield-
vast data volumes in large databases is critical, as pro- ing better results. Optimizing efficiency without com-
cessing all data as input is impractical. Additionally, dirty promising accuracy is essential, especially in interactive
data, such as missing values, duplicates, or inconsisten- querying scenarios requiring low latency.
cies, can lead to erroneous query results (e.g., affecting • SQL Efficiency: The SQL generated by Text-to-SQL mod-
WHERE clauses) if not properly managed. els must be both correct and optimized for performance.
This includes optimizing join operations, index usage,
C3: Text-to-SQL Translation. The Text-to-SQL task differs and query structures. Efficient queries reduce database
from the compilation of a high-level programming language to load, improving system responsiveness and throughput.
a low-level machine language, as it usually has a one-to-many • Insufficient and Noisy Training Data: High-quality Text-
mapping between the input NL, DB and output SQL. Specifi- to-SQL training data is challenging to obtain. Public
cally, the Text-to-SQL task faces several unique challenges: datasets are often limited and may include noisy annota-
• Free-form NL vs. Constrained and Formal SQL: Natural tions, affecting model performance [33], [34]. Annotation
language is flexible, while SQL queries must adhere to requires database expertise, increasing costs, and the
strict syntax. Translating NL into SQL requires precision complexity of Text-to-SQL tasks often leads to errors.
to ensure the generated queries are executable. • Trustworthiness and Reliability: Text-to-SQL models
• Multiple Possible SQL Queries: A single NL query can must be trustworthy and reliable, consistently producing
correspond to multiple SQL queries that fulfill the query accurate results across diverse datasets and scenarios.
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 5

Remark: PLMs vs. LLMs Figure 4 shows the key differences

PLM Pre-train/Fine-tune between LLMs and PLMs. LLMs are a subset of PLMs,
Pre-train/Fine-tune LM-powered
distinguished by their advanced language understanding and
Prompt Text-to-SQL emergent capabilities [47], [48]. The emergent abilities allow
LLM
LLMs to perform Text-to-SQL tasks directly using prompts. In
contrast, PLMs generally require additional pre-training or
Fig. 4: The Categorization of PLM and LLM in Text-to-SQL. fine-tuning for acceptable Text-to-SQL performance.
4) LLM-based Stage: LLMs demonstrate unique emergent
capabilities that surpass traditional PLMs in NLP tasks, mark-
Trustworthiness requires transparency, allowing users to ing a new paradigm for Text-to-SQL solutions. These LLM-
understand and verify the generated SQL. based Text-to-SQL methods have become the most represen-
tative solutions in the current Text-to-SQL landscape [5], [6],
[49], [50]. Current research focuses on optimizing prompt
D. Challenges Solving with Large Language Models design [6] and fine-tuning LLMs [49]. For example, DAIL-
SQL [6] utilizes the GPT-4 with effective prompt engineering
Difficulty Levels. We categorize the difficulty of Text-to-SQL
techniques, achieving strong results on the Spider dataset [45].
into five levels, each addressing specific hurdles, as shown in
Meanwhile, CodeS [49] builds an LLM specifically for Text-
Figure 3(a). The first three levels cover challenges that have
to-SQL tasks by pretraining StarCoder [51] on a large Text-
been or are currently being addressed, highlighting the gradual
to-SQL-related corpus, showing solid performance on bench-
progress in Text-to-SQL capabilities. The fourth level includes
marks like BIRD [52]. At this stage, LLMs’ emergent ca-
challenges that are the focus of current LLM-based solutions,
pabilities have significantly improved natural language un-
while the fifth level represents future challenges, showing our
derstanding, shifting the task’s focus toward database-specific
vision for Text-to-SQL advancements over the next five years.
challenges. New benchmarks like BIRD [52] and BULL [50]
The Evolution of Text-to-SQL Solutions. The development of emphasize handling massive tables and domain-specific solu-
Text-to-SQL solutions, illustrated in Figure 3(b), progresses tions (see Figure 3(b)-④).
through four distinct stages: the rule-based stage, the neural
network-based stage, the PLM-based stage, and the LLM- Text-to-SQL Solutions in LLMs Era. Broadly speaking,
based stage. At each stage, we analyze shifts in target users, there are two major approaches to leverage the capabilities
i.e., from experts to broader user groups, and the extent to of LLMs for Text-to-SQL: 1) in-context learning, and 2) pre-
which various Text-to-SQL challenges are addressed. train/fine-tune LLMs specialized for Text-to-SQL.
1) Rule-based Stage: In the early stages, statistical lan- In-Context Learning for Text-to-SQL. For in-context learning
guage models (e.g., semantic parsers) were used to interpret methods, the goal is to optimize the prompt function P to
NL queries and convert them into SQL queries using pre- guide the LLMs, which can be formulated as follows:
defined rules [32], [35]–[37]. However, rule-based Text-to- FLLM (P | NL, DB, K) → SQL,
SQL methods face challenges in adaptability, scalability, and
generalization. At this stage, natural language understanding where K denotes additional information or domain-specific
was limited to the token level, with research primarily focused knowledge related to NL or DB. P is a prompt function that
on single-table SQL queries (see Figure 3(b)-①). transforms the input (NL, DB, K) into a suitable textual prompt
2) Neural Network-based Stage: To alleviate the limitations for the LLMs. An well-designed P can effectively guide the
of rule-based methods, researchers explored neural networks LLMs to perform the Text-to-SQL task more accurately.
for the Text-to-SQL task. This led to the development of Employing in-context learning strategies for Text-to-SQL
models based on sequence-to-sequence architectures and graph treats LLMs as off-the-shelf tools, without modifying their
neural networks [38]–[40], which enhanced the handling of parameters. However, if users have sufficient training data
synonyms and intent understanding. Thus, research advanced or hardware resources, calibrating the LLMs’ parameters can
from single-table scenarios to more complex multi-table sce- enhance performance and accuracy, allowing the model to be
narios (see Figure 3(b)-②). However, the generalization ability more closely tailored to the specific Text-to-SQL task.
of these methods is still limited by model size and the Pre-train and Fine-tune LLMs for Text-to-SQL. Fully opti-
availability of sufficient training data. mizing the parameters of LLMs for Text-to-SQL involves two
3) PLM-based Stage: The introduction of PLMs like key stages, pre-train and fine-tune, formulated as follows:
BERT [41] and T5 [42] in 2018 led to significant advance-
LLM∗ = Ffine-tune (Fpre-train (LLM, Dp ), Df )
ments in Text-to-SQL methods based on PLMs [7], [43], [44],
achieving competitive performance on various benchmarks During pre-train, the LLM is trained on a large-scale and
(see Figure 3(b)-③). At this stage, PLMs-based Text-to-SQL diverse dataset Dp that includes a broad range of linguistic
models trained on large corpora have greatly enhanced natural patterns and domain-general knowledge, enabling the model
language understanding, resolving approximately 80% of cases to develop robust understanding capabilities.
in the Spider dataset [45]. However, accuracy drops to about In the subsequent fine-tune stage, the pre-trained model
50% on the extra hard cases of Spider [46]. In addition, these is further adjusted on a more specialized dataset Df , which
models still face challenges in handling complex schemas. is closely aligned with the Text-to-SQL task. This targeted
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 6

NL Query Database • Decoding Strategy: This key module transforms the in-
Pre-Processing (Optional)
ternal representation into SQL queries (Section V-B).
Schema Linking Database Content Retrieval Additional Information Acquisition • Task-specific Prompt Strategy: This module provides
Student
...
tailored guidance for the Text-to-SQL model, optimizing
Name Age
Name
13 the Text-to-SQL translation workflow (Section V-C).
16-year-old 16
• Intermediate Representation: This module serves as a
Text-to-SQL Translation Methods bridge between NL and SQL translation, providing a
structured approach to abstract, align, and optimize NL

Task-specific Prompt Strategy

Encoding
Sequential Encoding Graph-based Encoding ...

Intermediate Representation
Strategy understanding, simplify complex reasoning, and guide the
Encoder generation of accurate SQL queries (Section V-D).
Backbone
Encoder-Only Decoder-Only
Model Decoder Post-Processing Methods. Post-processing is a crucial step to

...
refine the generated SQL queries for better accuracy.
PLM LLM
Decoding
• SQL Correction Strategy: This aims to identify and
...
Strategy Beam Search Constraint-aware Incremental Decoding correct syntax errors in generated SQL (Section VI-A).
Post-Processing (Optional)
• Output Consistency: This module ensures the uniformity
Correction Consistency Execution-Guided N-best Rerankers
of SQL by sampling multiple reasoning results and se-
a
lecting the most consistent result (Section VI-B).
a • Execution-Guided Strategy: It uses the execution results
Correction Vote SQL Execution Rank
b of SQL to guide subsequent refinements (Section VI-C).
• N-best Rankers Strategy: It aims to rerank the top-k
SQL Query
results generated by the Text-to-SQL model to enhance
Fig. 5: An Overview of Text-to-SQL Modules in LLM Era. query accuracy (Section VI-D).
Multi-agent Collaboration for Text-to-SQL in LLM Era.
training refines the model’s capabilities, enabling it to more Building upon modular design principles, recent research has
effectively interpret and generate SQL based on NL queries. further introduced multi-agent architectures to tackle the Text-
to-SQL task. In contrast to traditional monolithic systems,
III. L ANGUAGE M ODEL - POWERED T EXT- TO -SQL multi-agent frameworks assign specialized responsibilities
OVERVIEW to distinct agents, each dedicated to handling a specific
While Text-to-SQL was initially designed as an end-to- subtask. This design facilitates enhanced division of labor
end task, recent advances, particularly in the LLM era, have and more effective coordination among components. For
shifted towards a modular design. As shown in Figure 5, example, MAC-SQL [54] adopts a three-agent architecture,
modern PLM- and LLM-based solutions typically decompose with separate agents for schema linking, query decomposition
the task into three stages: Pre-Processing, Translation, and and generation, and execution-guided refinement. Similarly,
Post-Processing. Each stage contains specialized modules such CHASE-SQL [55] employs a divide-and-conquer approach,
as schema linking, intermediate representation, and execution- selecting relevant database content during preprocessing,
guided correction. This design reflects the increasing complex- generating SQL queries through multiple chain-of-thought
ity of Text-to-SQL and aligns with the rising trend of multi- pathways, and iteratively refining outputs through self-
agent or multi-module collaboration. Table I further compares correction and ranking. Pushing the boundary further,
the key design choices across recent solutions. Alpha-SQL [56] proposes a planning-centric autonomous
agent framework that leverages LLMs in combination with
Pre-Processing Methods. Pre-processing enhances inputs and Monte Carlo Tree Search (MCTS). This agent dynamically
plays a significant role in improving Text-to-SQL parsing [53]. selects and activates the appropriate modules, such as schema
• Schema Linking: This module identifies the most relevant linking and SQL generation, based on contextual reasoning
tables and columns from Text-to-SQL (Section IV-A). and execution-based feedback. Alpha-SQL’s strategy-driven
• Database Content Retrieval: This key module accesses exploration and adaptive control offer robust generalization,
the appropriate database contents or cell values needed avoiding the rigidity of pipeline-based approaches.
for formulating SQL (Section IV-B).
• Additional Information Acquisition: This key module
IV. P RE -P ROCESSING S TRATEGIES FOR T EXT- TO -SQL
enriches the contextual backdrop by integrating domain-
specific knowledge (Section IV-C). The pre-processing step is crucial in the Text-to-SQL trans-
lation process, as it identifies relevant tables and columns (i.e.,
Translation Methods. This is the core of Text-to-SQL solu- Schema Linking) and retrieves necessary database contents or
tion, responsible for converting input NL queries into SQL. cell values (i.e., DB Content Retrieval) to support SQL query
• Encoding Strategy: This crucial module transforms the generation What’s more, it enriches context by incorporating
input NL and database schema into an internal rep- domain-specific knowledge (i.e., Additional Information Ac-
resentation, capturing both the semantic and structural quisition), which can improve the understanding of the query
information of the input data (Section V-A). context and correct errors to prevent their propagation.
 TABLE I: Comparisons of Existing Text-to-SQL Solutions.
Pre-Processing Text-to-SQL Translation Methods Post-Processing
Methods Years Finetuning
Schema DB Content Additional Information Backbone Encoding Intermediate Task-specific Decoding Execution N-best
Correction Consistency
Linking Retrieval Acquisition Model Strategy Representation Prompt Strategy Strategy -Guided Rerankers
OpenSearch-SQL [57] 2025 - ✓ ✓ ✓ Decoder-Only Sequential Encoding Syntax Language COT Greedy Search ✓ ✓ ✓ -
CHASE-SQL [55] 2025 - ✓ ✓ ✓ Decoder-Only Sequential Encoding - Multi-COT Greedy Search ✓ ✓ ✓ -
ROUTE [58] 2025 ✓ ✓ ✓ ✓ Decoder-Only Sequential Encoding - - Greedy Search ✓ ✓ ✓ -
Alpha-SQL [56] 2025 - ✓ ✓ ✓ Decoder-Only Sequential Encoding - COT Greedy Search ✓ ✓ ✓ -
CHESS [59] 2024 - ✓ ✓ ✓ Decoder-Only Sequential Encoding - COT Greedy Search ✓ ✓ ✓ -
CodeS [49] 2024 - ✓ ✓ - Decoder-Only Sequential Encoding - - Greedy Search - - ✓ -
SFT CodeS [49] 2024 ✓ ✓ ✓ ✓ Decoder-Only Sequential Encoding - - Greedy Search - - ✓ -
FinSQL [50] 2024 ✓ ✓ - ✓ Decoder-Only Sequential Encoding - - Greedy Search ✓ ✓ - -
DTS-SQL [60] 2024 ✓ ✓ - - Decoder-Only Sequential Encoding - - Greedy Search - - - -
TA-SQL [61] 2024 - ✓ - - Decoder-Only Sequential Encoding Sketch Structure COT Greedy Search - - - -
SuperSQL [46] 2024 - ✓ ✓ ✓ Decoder-Only Sequential Encoding - - Greedy Search - ✓ - -
ZeroNL2SQL [44] 2024 ✓ - - - Encoder-Decoder Sequential Encoding Sketch Structure Decomposition Beam Search ✓ - ✓ -
PET-SQL [62] 2024 ✓ ✓ - ✓ Decoder-Only Sequential Encoding - - Greedy Search - ✓ - -
CoE-SQL [63] 2024 - - - ✓ Decoder-Only Sequential Encoding - CoT Greedy Search ✓ - - -
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020

PURPLE [64] 2024 - ✓ - ✓ Decoder-Only Sequential Encoding - - Greedy Search ✓ ✓ ✓ -

MetaSQL [65] 2024 - ✓ - ✓ Decoder-Only Sequential Encoding - Decomposition Greedy Search - - - ✓
DEA-SQL [66] 2024 - ✓ - ✓ Decoder-Only Sequential Encoding - Decomposition Greedy Search ✓ - - -
DIN-SQL [5] 2023 - ✓ - ✓ Decoder-Only Sequential Encoding Syntax Language Decomposition Greedy Search ✓ - - -
DAIL-SQL [6] 2023 - - - ✓ Decoder-Only Sequential Encoding - - Greedy Search - ✓ - -
C3-SQL [67] 2023 - ✓ - - Decoder-Only Sequential Encoding - COT Greedy Search - ✓ - -
RESDSQL [7] 2023 ✓ ✓ ✓ - Encoder-Decoder Sequential Encoding Syntax Language Decomposition Beam Search - - - -
T5-3B+NatSQL+Token Preprocessing [68] 2023 ✓ ✓ ✓ - Encoder-Decoder Sequential Encoding Syntax Language - Greedy Search - - - -
ACT-SQL [69] 2023 - ✓ - ✓ Decoder-Only Sequential Encoding - CoT Greedy Search - - - -
ODIS [70] 2023 - - - ✓ Decoder-Only Sequential Encoding - - Greedy Search - - - -
MAC-SQL [54] 2023 - ✓ - - Decoder-Only Sequential Encoding - Decomposition Greedy Search ✓ - ✓ -
SC-Prompt [1] 2023 ✓ - - - Encoder-Decoder Separate Encoding Sketch Structure - Beam Search ✓ - - -
CatSQL [71] 2023 ✓ - - - Encoder-Only Sequential Encoding Sketch Structure - Beam Search ✓ - - -
SQLFormer [72] 2023 ✓ ✓ - - Encoder-Decoder Graph-based Encoding - - Beam Search - - - -
G³R [73] 2023 ✓ ✓ ✓ - Encoder-Only Graph-based Encoding - COT Beam Search - - - ✓
Graphix-T5 [43] 2022 ✓ ✓ ✓ - Encoder-Decoder Graph-based Encoding - - Constraint-aware Incremental - - - -
SHiP [74] 2022 ✓ - ✓ - Encoder-Decoder Graph-based Encoding - - Constraint-aware Incremental - - - -
N-best List Rerankers [75] 2022 ✓ ✓ ✓ - Encoder-Decoder Sequential Encoding - - Constraint-aware Incremental - - - ✓
RASAT [76] 2022 ✓ - ✓ - Encoder-Decoder Graph-based Encoding - - Constraint-aware Incremental - - - -
PICARD [77] 2022 ✓ - ✓ - Encoder-Decoder Sequential Encoding - - Constraint-aware Incremental - - - -
TKK [78] 2022 ✓ - ✓ - Encoder-Decoder Separate Encoding Sketch Structure Decomposition Constraint-aware Incremental - - - -
S²SQL [79] 2022 ✓ ✓ ✓ - Encoder-Only Graph-based Encoding - - Greedy Search - - - -
RAT-SQL [80] 2021 ✓ ✓ ✓ - Encoder-Only Graph-based Encoding Syntax Language - Beam Search - - - -
SmBoP [81] 2021 ✓ - ✓ - Encoder-Only Graph-based Encoding - - Beam Search - - - -
RaSaP [82] 2021 ✓ ✓ ✓ - Encoder-Only Graph-based Encoding - - Beam Search - - - -
BRIDGE [83] 2020 ✓ - ✓ - Encoder-Only Sequential Encoding - - Constraint-aware Incremental - - - -
7
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 8

A. Schema Linking handles schema linking, activated only when the database
Schema linking aims to identify the tables and columns schema prompt exceeds a specified length. CHESS [59] uti-
relevant to the given NL query, ensuring accurate mapping and lizes GPT-4 to extract keywords from both NL and evidence
processing of key information within the limited input. This (additional information from BIRD [52]), implementing a
step is essential for improving the performance of the Text-to- three-stage schema pruning protocol with different prompts.
SQL task. In the LLM era, schema linking has become even Employing ICL for schema linking has shown promising
more critical due to the input length limitations of LLMs. performance. However, LLMs have inherent limitations in
We categorize existing schema linking strategies into three the amount of context they can process, meaning complex
groups based on their characteristics: 1) string matching-based schemas with many tables and columns may exceed this limit.
schema linking, 2) neural network-based schema linking, and
3) in-context learning-based for schema linking. B. Database Content Retrieval
1) String Matching-based Schema Linking: Early re-
search [39], [84], [85] primarily focused on string matching Database content retrieval focuses on efficiently extracting
techniques for schema linking. These methods use similarity cell values for specific SQL clauses such as WHERE. We
measures between the NL queries and DB schemas to iden- categorize existing database content retrieval strategies into
tify relevant mappings. IRNet [84] adopts exact matching, three groups based on their characteristics: 1) String Matching-
identifying links when candidates are identical or one is a based Methods, 2) Neural Network-based Methods, and 3)
substring of the other. While effective for simple cases, it may Index Strategy for Database Content Retrieval.
yield false positives due to shared words. To handle spelling 1) String Matching-based Methods: String matching-based
variations, ValueNet [86] applies approximate matching via methods identify and compare cell values related to the NL
the Damerau–Levenshtein distance [87]. query through string matching [7], [43], [77], [83], [84], [86].
However, these methods struggle with handling synonyms IRNet [84] uses n-grams, treating text between quotes as
and are not robust enough to manage vocabulary variations, cell values. BRIDGE [83] advances this with an anchor text
limiting their effectiveness in complex Text-to-SQL tasks. matching technique that automatically extracts cell values from
2) Neural Network-based Schema Linking: To alleviate the NL. Using heuristics, it calculates the maximum sequence
the above limitations, researchers have employed deep neural match to define matching boundaries, excluding irrelevant
networks to align database schemas with natural language substrings and adjusts thresholds for accuracy.
queries [7], [53]. These methods can better parse complex se- However, while string matching methods are effective, they
mantic relationships between NL queries and database schema. struggle with synonyms and can be computationally expensive
DAE [88] frames schema linking as a sequential tagging when handling large databases.
problem, using a two-stage anonymization model to capture 2) Neural Network-based Methods: These methods aim to
semantic relationships between schema and NL. SLSQL [53] capture complex data and semantic features through layers of
annotates schema linking information in Spider dataset [45], nonlinear transformations, helping to resolve synonym issues.
enabling a systematic, data-driven study. RESDSQL [7] in- TABERT [91] uses a method called database content snap-
troduces a ranking-enhanced encoding framework for schema shots to encode relevant database content for sNL query, em-
linking, using a cross-encoder to prioritize tables and columns ploying attention mechanisms to manage information across
based on classification probabilities. FinSQL [50] uses a cell value representations in different rows. Another approach
parallel cross-encoder to retrieve relevant schema elements, leverages graph relationships to represent database content.
significantly reducing linking time. IRNet [84] uses the knowledge graph ConceptNet [92] to find
However, neural network-based methods often struggle to and link relevant cell values, assigning types based on exact
generalize across databases with diverse schemas or domains, or partial matches. RAT-SQL [80] further enhances structural
especially when training data is scarce. reasoning by modeling the relationship between cell values and
3) In-Context Learning for Schema Linking: With the ad- the NL query, identifying column-value relationships where the
vancement of LLMs like GPT-4, research is exploring how to query value is part of the column’s candidate cell value.
leverage their strong reasoning capabilities for schema linking, While these methods capture semantic features, they may
i.e., directly identifying and linking relevant database schema struggle with ambiguous or context-dependent NL, leading to
components from the NL query. A key technique is In-Context inaccurate cell value retrieval. Moreover, the training of neural
Learning (ICL) technique [89], which utilizes LLMs’ ability to networks demands substantial computational resources.
understand complex language patterns and relationships within 3) Index Strategy for Database Content Retrieval: Effi-
data schemas, enabling a more dynamic and flexible schema ciently retrieving relevant cell values is crucial for the perfor-
linking process [5], [54], [59], [67], [90]. mance of Text-to-SQL systems, especially with large datasets.
C3-SQL [67] employs zero-shot prompts with GPT-3.5 Indexing is a key method for improving retrieval efficiency by
using self-consistency for table and column linking. For table enabling faster access to relevant cell values [49], [59].
linking, tables are ranked by relevance and listed; for column CHESS [59] uses a Locality-sensitive Hashing [93] for
linking, columns are ranked within relevant tables and out- approximate nearest neighbor searches, indexing unique cell
putted as a dictionary, prioritizing matches with question terms values to quickly find the top matches related to the NL
or foreign keys. MAC-SQL [54] proposes a multi-agent collab- query. This approach speeds up the process of comparing edit
orative framework for Text-to-SQL, where the Selector agent distances and semantic embeddings. CodeS [49] employs a
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 9

C3-SQL [67], PET-SQL [62], CHESS [59], DAIL-SQL [6], DIN-SQL [5], CatSQL [71] , BRIDGE [83], MAC-SQL [54],
Sequential Encoding
TA-SQL [61], RESDSQL [7], CodeS [49], N-best List Rerankers [75] , T5+NatSQL+Token Preprocessing [68]
Encoding Strategy
(§V-A) Graph-based Encoding SQLformer [72], RASAT [76], SHiP [74], SmBoP [81], RaSaP [82] , S²SQL [79], Graphix-T5 [43], G³R [73], RATSQL [80]
Separate Encoding SC-Prompt [1], TKK [78]
Greedy Search-based DAIL-SQL [6], DIN-SQL [5], C3-SQL [67], PET-SQL [62], CHESS [59] , CodeS [49], SuperSQL [46], TA-SQL [61] ,
Decoding MAC-SQL [54], T5+NatSQL+Token Preprocessing [68]
Decoding Strategy Beam Search-based CatSQL [71], ZeroNL2SQL [44], SQLformer [72], SC-Prompt [1], SmBoP [81] , RaSaP [82], G³R [73], RESDSQL [7],
(§V-B) Decoding RATSQL [80]
Text-to-SQL
Translation Methods Constraint-aware
TKK [78], PICARD [77], RASAT [76], SHiP [74], Graphix-T5 [43] , BRIDGE [83], N-best List Rerankers [75]
Incremental Decoding

Task-specific Prompt Chain-of-Thought CHESS [59], ACT-SQL [69], COE-SQL [63], TA-SQL [61], C3-SQL [67], G³R [73], MAC-SQL [54]
Strategy (§V-C) Decomposition TKK [78], G³R [73], DEA-SQL [66], MAC-SQL [54], DIN-SQL [5]
SQL-like Schema-free SQL [96], SyntaxSQLNet [97], SemQL [98], EditSQL [99], RAT-SQL [80], NatSQL [100], SHiP [74],
Intermediate Syntax Language QPL [99], RESDSQL [7], QDMR [101], OpenSearch-SQL [57]
Representation
(§V-D) SQL-like
SyntaxSQLNet [97], SC-prompt [1], CatSQL [71], ZeroNL2SQL [44], TA-SQL [61], RESDSQL [7]
Sketch Structure

Fig. 6: A Taxonomy of Text-to-SQL Translation Methods based on their Design Choices.

coarse-to-fine matching strategy. It uses BM25 [94] to build nal knowledge into natural language. For example, RE-
an index for coarse-grained searches, identifying candidate GROUP [102] creates a formulaic knowledge base across
values, which are then refined by applying the Longest Com- domains (e.g., finance, transportation) and uses Dense Pas-
mon Substring algorithm [95] to assess similarity with the NL, sage Retriever [103] to compute similarity scores, integrating
thereby pinpointing the most relevant cell values. related entities with NL and schema through an Erasing-Then-
While indexing significantly improves retrieval efficiency, Awakening model [104]. ReBoost [105] uses a two-phase
building indexes is time-consuming, and frequent changes to Explain-Squeeze Schema Linking strategy, first presenting a
database content require continuous updates, adding overhead. generalized schema to LLMs, then applying targeted prompts
to improve query-to-entity mapping accuracy.
C. Additional Information Acquisition Retrieval-based methods improve the effectiveness of ac-
Additional information, such as domain knowledge, plays a quiring additional information but increase computing costs.
crucial role in enhancing Text-to-SQL models’ understanding Moreover, current research mostly relies on domain-specific
of NL queries, schema linking, and overall Text-to-SQL trans- text, with limited use of structured knowledge. Thus, integrat-
lation. This information can provide demonstration examples, ing diverse information sources could further enhance Text-to-
domain knowledge, formulaic evidence, and format informa- SQL performance, especially for domain-specific databases.
tion for the Text-to-SQL backbone model or specific modules,
thereby enhancing the quality of generated results. We cate- V. T EXT- TO -SQL T RANSLATION M ETHODS
gorize existing strategies into the following two groups: 1) In this section, we elaborate on Text-to-SQL translation
Sample-based Methods, and 2) Retrieval-based Methods. methods using language models. As shown in Figure 6, we will
1) Sample-based Methods: With advancements in LLMs detail their encoding (Section V-A), decoding (Section V-B),
and in-context learning techniques, researchers often incor- and task-specific prompt strategies (Section V-C). Moreover,
porate additional information into the textual inputs (i.e., we will discuss how the intermediate representation can benefit
prompts) alongside demonstration examples [49], [59], [70]. the Text-to-SQL translation process (Section V-D).
DIN-SQL [5] integrates additional information through few-
shot learning across multiple stages. This helps DIN-SQL to
handle challenges like complex schema links, multiple table A. Encoding Strategy
joins, and nested queries. In practice, real-world databases In the Text-to-SQL task, encoding refers to transforming
often contain rich cross-domain knowledge that can serve as NL and database schema into a structured format suitable
external evidence to support query generation. For example, for language model processing. This step is essential for
BIRD [52] contains domain knowledge which is crucial for converting unstructured data into a form usable for SQL
various Text-to-SQL works [6], [59]. Recent works [49], [70] generation, capturing the NL’s semantics and the schema’s
also encode schema metadata(e.g., data types) into natural structure to help the model map user intent to appropriate SQL.
language to enhance context understanding. As shown in Figure 7, primary encoding strategies include
2) Retrieval-based Methods: Extracting relevant knowl- 1) Sequential Encoding, 2) Graph-based Encoding, and 3)
edge and few-shot examples from extensive domain knowl- Separate Encoding.
edge bases can increase token usage, impacting efficiency 1) Sequential Encoding Strategy: Sequential encoding is a
and computational cost [5], [6]. To enhance accuracy and strategy in Text-to-SQL models where both the NL and the
efficiency, some researchers employ similarity-based retrieval database schema are treated as token sequences. As shown in
methods. For example, PET-SQL [62] builds a pool of question Figure 7(a), the model processes the entire input as a linear to-
frames and question-SQL pairs, selecting the k most similar ken sequence using standard Transformer-based architectures.
examples to the target query, which are then used in prompts. Models like T5 [42] are used to encode NL and database
When databases lack text-based additional information, schema sequentially in works [68], [77]. BRIDGE [83] im-
researchers devise methods to retrieve and convert exter- proves the alignment between the NL and database schema by
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 10

NL Query Tokens DB Tables DB Columns Prompt Structure

Content
Selected Tokens Unselected Tokens Unsatisfied with constraints
prompt Prompt
NL Query DB Schema
Q3 C3 NL Query DB Schema NL Query DB Schema all all from ... all
Q1 T1 select [col] select [col] select
C1
Q1 ... Qn T1 C1 ... Q4
S1 ... Q1 ... T1 C1 C1 ... Q1 ... T1 C1 [col] [col] [col] [col] ...
C4 T2 C2 ... [col]
<star> from <star> <star> from ...
Encoding Q2
Encoding Encoding
... sql from ...
Hidden Layer
Structure Prompt
the
Hidden Layer
Encoding
Representation
Construction

the the
Representation
Hidden Layer
of SQL Sturcture of SQL Content the
Representation
(c) Separate Encoding
(a) Greedy Search-based (b) Beam Search-based (c) Constraint-aware

(a) Sequential Encoding (b) Graph-based Encoding (SC-Prompt AS Example)

Decoding Decoding Incremental Decoding

Fig. 7: An Overview of the Encoding Strategies.
Fig. 8: An Overview of the Decoding Strategies.

representing both as a tagged sequence and inserting matched

lack of interaction between the two components hindered
database cell values (called anchor texts) next to corresponding
effective schema linking, leading to limited performance and
fields. RESDSQL [7] uses a ranking-enhanced encoder to sort
making this approach less common in recent research. Modern
and filter schema items, prioritizing the most relevant ones and
separate encoding strategies, as illustrated in Figure 7(c),
reducing schema linking complexity. Although LLM-based
focus on modular representation learning by decomposing
Text-to-SQL systems often do not explicitly define an input
the Text-to-SQL task into subtasks and encoding different
encoding strategy, they typically implicitly adopt a sequential
aspects separately. TKK [78] employs task decomposition and
form by concatenating queries and schema components. These
multi-task learning strategies by breaking down the complex
models leverage self-attention to model dependencies across
Text-to-SQL task into subtasks and progressively integrating
the entire sequence.
knowledge. Similarly, SC-Prompt [1] divides text encoding
While this allows for flexible contextualization, such ap-
into two stages: structure and content, each encoded separately.
proaches may struggle to capture complex relational structures,
While separate encoding may increase computational over-
limiting their performance on deeply nested SQL queries.
head due to multiple processing steps, it allows for more re-
2) Graph-based Encoding Strategy: Graph-based encod- fined understanding of different aspects of queries. This mod-
ing in Text-to-SQL models represents both NL and database ularity provides the model with greater flexibility to handle
schema as interconnected graphs, leveraging the relational various query tasks, thereby enhancing overall performance.
structure of databases and inter-dependencies in the input
data, as shown in Figure 7(b). Unlike sequential encoding,
this approach preserves the schema’s topology, offering richer B. Decoding Strategy
context for each element and enhancing the model’s ability to Decoding is a crucial step in Text-to-SQL translation, trans-
produce accurate SQL queries [43], [72], [74], [76], [79]–[81]. forming encoder-generated representations into SQL. An effec-
RAT-SQL [80] introduces a relation-aware self-attention tive decoding strategy ensures that the generated SQL are not
mechanism, explicitly using relational information in a graph only syntactically correct but also semantically aligned with
structure to jointly encode the question and schema, en- the NL queries, while optimizing SQL execution efficiency.
hancing the model’s understanding of structural information. Figure 8 introduces several key decoding strategies.
S2 SQL [79] injects syntactic structure information at the 1) Greedy Search-based Decoding Strategy: The greedy
encoding stage using the ELECTRA [106] model, enhancing search-based decoding strategy is a simple and efficient
semantic understanding. G3 R [73] uses the LGESQL [107] method that selects the token with the highest probability at
encoder and Graph Attention Network (GAT) [108] to cap- each decoding step. As illustrated in Figure 8(a), it constructs
ture multi-source heterogeneous information. Graphix-T5 [43] the output sequence by making a series of locally optimal
adds graph-aware layers, incorporating structural information choices, without considering future possibilities.
directly into the encoding process, significantly improving SQL Since GPT models (e.g., GPT-4) default to greedy search-
query generation across multiple benchmarks. based decoding, many Text-to-SQL solutions using GPT fall
However, this strategy typically involves more intricate into this category. DTS-SQL [60], based on DeepSeek [111],
graph construction and processing algorithms. It also tends uses the same approach. Early models like SQLNet [109] and
to rely on large-scale training data to achieve optimal perfor- Seq2SQL [110] also rely on greedy search for SQL generation.
mance, which limits its applicability in low-resource scenarios. Despite its efficiency, greedy search has notable limitations.
3) Separate Encoding Strategy: The separate encoding By focusing only on immediate token probabilities, it fails
strategy in Text-to-SQL refers to independently encoding to account for long-term dependencies or global sequence
different parts of the input (typically NL and the DB schema) coherence. As a result, errors made early in the decoding
rather than combining them into a single sequence. This process may propagate and lead to suboptimal or incorrect
strategy has evolved significantly over time and can be broadly SQL queries, particularly for complex or multi-step questions.
categorized into traditional and modern forms. 2) Beam Search-based Decoding Strategy: Beam search is
In traditional separate encoding, early models like SQL- a widely used decoding strategy that explores a broader search
Net [109] and Seq2SQL [110], processed the NL and database space compared to greedy decoding, often leading to higher-
schema separately due to format mismatches. However, this quality results. Instead of selecting only the top token at each
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 11

step, it retains a fixed number of top-ranked partial sequences reasoning process, improving both the accuracy and inter-
(known as beams) and expands each by considering the top-k pretability of the generated results. In Text-to-SQL, CoT en-
most probable next tokens, as illustrated in Figure 8(b). hances model performance and ensures that the generated SQL
Given its advantages, several Text-to-SQL models employ statements are more aligned with human expectations [113].
beam search [7], [71], [82]. RAT-SQL [80] combines relation- CHESS [59] transforms NL into SQL statements through
aware graph structure encoding with beam search to generate a streamlined pipeline that utilizes LLMs and CoT. This
multiple SQL candidates, reranking them based on graph process includes entity and context retrieval, schema selection,
structure information. Unlike RAT-SQL, EditSQL [99] uses SQL generation, and revision. In addition, the integration
beam search alongside dialogue history to generate and re- of CoT with other techniques can enhance the performance
fine candidate SQL queries. SmBoP [81] employs a semi- of Text-to-SQL models. These techniques include in-context
autoregressive bottom-up decoding approach, improving ef- learning [63], [69], logical synthesis [61], calibration with
ficiency by parallelizing sub-tree construction and scoring, hints [67], [73] and multi-agent system [54]. Specifically, in-
with logarithmic time complexity. ZeroNL2SQL [44] retains context learning and logical synthesis enrich CoT by em-
the top-k hypotheses during the SQL sketch generation stage, bedding a deeper linguistic understanding, enabling precise
which are then refined for query and predicate calibration. semantic mapping to SQL constructs [63], [69]. Calibration
Compared to greedy decoding, beam search improves the with hints fine-tunes model responses, aligning them closely
ability to generate syntactically and semantically valid SQL with NL nuances for accurate intent translation [67], [73].
queries, especially in complex scenarios, by considering mul- Furthermore, integrating the multi-agent framework with CoT
tiple hypotheses at each step. However, this benefit comes at fosters a collaborative approach, with specialized agents han-
the cost of increased computational complexity and memory dling tasks like schema linking and SQL generation, which
usage, potentially slowing down the decoding process. speeds up reasoning and enhances adaptability [54].
3) Constraint-aware Incremental Decoding Strategy: Overall, these techniques create a more robust Text-to-
Constraint-aware incremental decoding strategies aim to en- SQL framework, offering better precision and reliability in
sure the structural and syntactic validity of SQL queries by translating complex NL queries into accurate SQL statements.
applying explicit constraints during the decoding process. As However, CoT prompting may introduce longer reasoning
shown in Figure 8(c), these strategies incrementally generate chains and latency, and its effectiveness can be sensitive to
SQL while enforcing SQL grammar constraints at each step. prompt design and task complexity.
A representative implementation is PICARD [77] (Parsing 2) Decomposition Strategy: The decomposition strategy
Incrementally for Constrained Auto-Regressive Decoding), divides the Text-to-SQL task into sequential subtasks, allowing
which integrates SQL grammar constraints into the decoding each sub-module to concentrate on a specific generation step,
loop. It verifies the syntactic validity of the partially generated thereby enhancing accuracy, quality, and interpretability.
query at every step, ensuring that each token adheres to the Different approaches vary in subtask decomposition gran-
SQL grammar. This significantly reduces the generation of ularity [54], [60], [66], [73], [78]. TKK [78] applies finer-
invalid or incomplete queries. Many models [43], [74]–[78] grained decomposition by breaking down Text-to-SQL parsing
have adopted this paradigm to improve performance. into subtasks like mapping NL to SELECT, FROM, and WHERE
In addition to grammar-level constraints, some mod- clauses. This approach helps the model concentrate on each
els incorporate schema-level constraints during decoding. clause, enhancing understanding of the problem, schema,
BRIDGE [83] introduces Schema-Consistency Guided Decod- and SQL alignment. Similar strategies are used in G3 R [73]
ing, which enforces alignment between the generated SQL and DEA-SQL [66]. Moreover, decomposition also reduces
query and the underlying database schema by verifying their model complexity. For example, MAC-SQL [54] introduces a
consistency and adjusting the decoding path accordingly. Decomposer agent to split the user’s query into subproblems,
While this strategy introduces additional computational making SQL generation for each part more manageable.
overhead due to per-token constraint evaluation, it offers strong In general, the decomposition strategy divides the Text-to-
guarantees of syntactic correctness and is particularly effective SQL translation task into multiple subtasks, enabling each sub-
for generating structurally complex SQL queries. module to focus on enhancing its specific output. However,
this approach also raises computational costs, making model
training and deployment more complex and resource-intensive.
C. Task-specific Prompt Strategy
In the era of LLMs, prompt engineering has become a pow- D. Intermediate Representation for Text-to-SQL Translation
erful method for harnessing LLM capabilities across diverse The Text-to-SQL is challenging due to the complexity and
tasks. In Text-to-SQL, task-specific prompts are crafted to ambiguity of NL queries, coupled with the syntax-constrained
guide LLMs in optimizing Text-to-SQL translation, enhancing nature of SQL. To simplify this process, researchers have
the accuracy of translating complex NL queries into precise developed a grammar-free intermediate representation (IR) to
SQL queries. Broadly speaking, there are two main types of bridge the “free-form” NL question and the “constrained and
task-specific prompt strategies: 1) Chain-of-Thought prompt- formal” SQL. This IR provides a structured yet flexible format,
ing, and 2) Decomposition Strategy. capturing the essential components and relationships within
1) Chain-of-Thought (CoT) Prompting: The CoT prompt- an NL query without the strict syntax requirements of SQL.
ing [112], known for its effectiveness, showcases the LLM’s Figure 9 shows two types of IR strategies, discussed below.
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 12

Intermediate Representation: instance, RESDSQL [7] introduces a rank-enhanced encoding

SELECT [Link]

and skeleton-aware decoding framework. During the decoding

WHERE count(film_actor.*)>5

(a) SQL-like Syntax Language (e.g. NATSQL)

phase, its decoder initially generates the SQL skeleton and

then transforms it into the actual SQL query. When combined
SELECT [column]
[column] title

FROM [table]
[table] film
with NatSQL, RESDSQL demonstrates the ability further to
JOIN [table]
[table] film_actor

ON [table].[column]
[table].[column] film.film_id

enhance the quality of SQL query generation.

=[table].[column]
[table].[column] film_actor.film_id
In general, SQL-like sketch structure can be more easily
GROUP BY [column]
[column] film_id

HAVING count([column]) > n [column] * [n] 5 combined with other strategies, such as decomposition strategy
(b) SQL-like Sketch Structure (e.g. SC-Prompt) or SQL-like syntax language strategy. In addition, it can
Fig. 9: An Example of the Intermediate Representation. more fully utilize the comprehension and cloze capabilities of
existing LLMs and reduce the dependence on professionals.

1) SQL-like syntax language: As shown in Figure 9(a), VI. P OST-P ROCESSING S TRATEGIES FOR T EXT- TO -SQL
SQL-like syntax language is a simplified SQL-like structure. After the Text-to-SQL model generates the SQL, post-
Early approaches used information retrieval techniques to processing can refine it to better meet user expectations. This
map the original question and schema data into this syn- step involves leveraging additional information or models to
tax [96], [114]. Subsequent research efforts have focused enhance the SQL, with a focus on SQL correction, ensuring
on consolidating or eliminating partial clauses or operations output consistency, and execution-guided checking.
in SQL queries to simplify SQL-like syntax language [97],
[98]. SyntaxSQLNet [97] simplifies the syntax language by A. SQL Correction Strategies
removing parts of the FROM and JOIN clauses. SemQL [98]
removes the entire FROM, JOIN, ON, and GROUP BY clauses, The SQL generated by Text-to-SQL models may contain
and further merges the WHERE and HAVING conditions into a syntax errors. DIN-SQL [5] introduces a self-correction mod-
unified filtering representation. Recent research has focused ule that operates in a zero-shot setting, where the model
on simplifying syntax languages to improve parsing effi- receives only the faulty SQL and attempts to correct it. Two
ciency [115]. NatSQL [100], a widely used SQL-like syntax prompts are used: a general prompt for CodeX, which directly
language, eliminates uncommon SQL operators and keywords, asks for error identification and correction, and a mild prompt
streamlining schema linking by minimizing necessary schema for GPT-4, which seeks potential issues without presuming
items. Combined with PLMs, NatSQL has achieved strong errors. To handle errors in predicate predictions, such as
results on various benchmarks [7], [68]. incorrect columns or values, ZeroNL2SQL [44] employs a
SQL-like syntax languages have demonstrated potential in multi-level matching approach. This method incrementally
bridging user queries and databases. However, previous studies expands matching across columns, tables, and databases, al-
face challenges due to high complexity and limited coverage lowing matched values to be returned to the LLMs to generate
SQL queries consistent with the database content.
of database structures [100]. As databases grow in size and
domain specificity, maintaining the simplicity of SQL-like While these methods focus on fixing syntax errors, they
syntax languages becomes increasingly difficult. Moreover, often overlook semantic errors [33], such as incorrect table
some of these languages require manual construction and joins, misaligned conditions, or inaccurate aggregations, which
adjustments, raising deployment costs and complexity. are essential for improving accuracy.
2) SQL-like sketch structure: Leveraging the structural
characteristics of SQL, researchers have developed SQL-like B. Output Consistency
sketches that mirror SQL structure for parsing, enabling diverse To enhance output consistency, self-consistency [117] has
NL queries to be mapped into a specific sketch space, as shown been introduced, based on the idea that complex reasoning
in Figure 9(b). This approach reduces parsing complexity. tasks may have multiple valid paths to a single correct answer.
Early works applied fixed sketch rules and neural networks This approach samples various reasoning paths and selects the
to map the NL into SQL-like sketch structure [97], [116]. most consistent answer to improve output quality.
SyntaxSQLNet [97] uses a syntax tree and a corresponding DAIL-SQL [6] integrates self-consistency, achieving a 0.4%
decoder, dividing the decoding into nine sub-modules that performance improvement over configurations without it. To
separately predict operators, keywords, and entities before reduce LLM randomness, Fin-SQL [50] generates n candidate
combining them to generate the final SQL. In recent years, SQL queries in parallel, clusters them based on keyword
the development of language models has allowed researchers consistency, and selects a query from the largest cluster. The
to design more elaborate SQL-like sketch structures for pars- self-consistency strategy enhances LLM output diversity by in-
ing [1], [44], [61], [71]. CatSQL [71] constructs a more creasing the temperature and selecting the final result through
general template sketch with slots serving as initial place- majority voting. However, recent studies [118] indicate that
holders. Its base model focuses on the parsing of NL to fill relying on a single model may still yield limited diversity. To
these placeholders, consequently decreasing the computational overcome this limitation, PET-SQL [62] introduces a cross-
cost. Moreover, several recent works cover both SQL-like consistency strategy, in which multiple LLMs generate SQL at
syntax language and SQL-like sketch transition methods. For lower temperatures and vote based on execution results.
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 13

Human Annotations Rules-based Synthesis LLMs for Data Synthesis

Timeline Before 2017 2017 2018 2019 2020 2021 2022 2023 2024
Milestone
ATIS
IMDb
Advising MIMISQL
SEDE ScienceBenchmark

GeoQuery
Yelp
FIBEN

BULL

Single-domain
Restaurants
Scholar BookSQL

Academic

WikiSQL Spider SParC

SQLUALL
CHASE
PAUQ
BIRD
Archer
CoSQL
ViText2SQL
Spider-Syn
knowSQL [Link]
Spider2-lite
CSpider DuSQL MT-TEQL
AmbiQT
Cross-domain KaggleDBQA

Spider-DK

Spider-Realistic

PortugueseSpider
Multi-turn SParC
CHASE
CoSQL
Spider-Syn
[Link]
Robustness
Spider-Realistic
Knowledge KaggleDBQA
BIRD Archer
Spider-DK
Efficiency BIRD
Ambiguity AmbiQT

Fig. 10: Timeline for Text-to-SQL Benchmarks.

While these methods improve accuracy by enforcing con- distinctions between candidate queries [120]. Similarly, ReF-
sistency across multiple executions, they significantly increase SQL [121] retrieves the most relevant results from the retriever
inference cost and time. and generator modules to improve final answer quality.
While N-best reranking is widely used in PLM-based meth-
C. Execution-Guided Strategies ods to refine SQL candidates, it is less common in LLM-based
methods, which typically have stronger inference capabilities.
In Text-to-SQL tasks, the execution result of an SQL query
provides critical feedback on Text-to-SQL translation accu-
racy. For example, errors or NULL values in execution results VII. T EXT- TO -SQL B ENCHMARKS
can signal potential issues with the SQL query.
In this section, we will first elaborate on the different types
To reflect human behavior in writing complex SQL queries,
of Text-to-SQL datasets, highlighting their characteristics, as
CHESS [59] provides LLMs with the database schema,
shown in Figure 10 (Section VII-A). We will then perform an
question, candidate SQL queries, and their execution results.
in-depth analysis of existing datasets (Section VII-B).
CHESS starts with a draft query and refines it based on
execution feedback, adjusting for syntax errors as needed.
CodeS [49] on the other hand, generates a complete SQL state- A. An Overview of Text-to-SQL Benchmarks
ments through beam search, producing four SQL candidates
With advancements in Text-to-SQL, various datasets have
and selecting the first executable one as the final result.
emerged to address the evolving challenges, as shown in Fig-
Execution-Guided Strategies refine SQL based on execution
ure 10. These range from single-domain databases with simple
results, ensuring the query retrieves data correctly. However,
queries to cross-domain, multi-turn, multilingual, and domain-
this approach can significantly increase SQL generation time,
specific scenarios, reflecting the progress and the emergence
especially with large databases.
of new challenges for Text-to-SQL solutions.

D. N-best Rerankers Strategies Single-Domain Text-to-SQL Datasets. Early Text-to-SQL

datasets focused on specific domains with relatively simple
In Text-to-SQL tasks, especially in cross-domain scenarios,
SQL queries, such as ATIS [122] for flight information and
generated SQL queries can vary subtly in structure and seman-
GeoQuery [123] for U.S. geographical facts. Recently, larger
tics. N-best reranking reorders the top-N model outputs, often
single-domain datasets [50], [131], [133], [142], [147], [148]
leveraging a larger model or additional knowledge sources. For
have been introduced, featuring more complex databases and
example, fine-tuning a BERT-based reranker, as demonstrated
SQL queries tailored to specific scenarios. This shift reflects
by Bertrand-dr on the Spider dataset [119], has effectively
an increased emphasis on assessing Text-to-SQL systems’
improved the performance of several Text-to-SQL models.
performance and practical utility within particular domains.
However, the effectiveness of Bertrand-dr’s reranking can
be unstable and sensitive to threshold settings, sometimes Cross-Domain Text-to-SQL Datasets. After the development
even yielding negative effects. To address these limitations, of early single-domain datasets, the Text-to-SQL field shifted
G3 R [73] introduces a feature-enhanced reranker using PLM- toward cross-domain datasets to test systems’ generalization
based hybrid prompt tuning, which bridges domain gaps across diverse SQL queries and databases. WikiSQL [110] was
without extra parameters. Contrastive learning then sharpens the first cross-domain dataset, drawing tables from Wikipedia
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 14

TABLE II: Statistics of Text-to-SQL Benchmarks.

Redundancy Measure DB Complexity Query Complexity
Dataset #-Unique #-Questions #-Tables #-Cols #-Records Scalar Math
#-Questions #-DBs #-Tables Tables Selects Agg
SQLs / #-SQLs / DB / Table / DB Func Comp
ATIS [122] 5280 947 5.6 1 25 25 5.24 162243 8.39 1.79 0.22 0 0
GeoQuery [123] 877 246 3.6 1 7 7 4.14 937 2.22 2.19 0.92 0 0.01
Restaurants [124] 378 23 16.4 1 3 3 4.00 19295 2.43 1.17 0.35 0 0
Academic [114] 196 185 1.1 1 17 17 3.12 58249674 3.48 1.04 0.54 0 0
IMDb [125] 131 89 1.5 1 17 17 3.94 40147386 2.91 1.01 0.30 0 0
Yelp [125] 128 110 1.2 1 8 8 5 4823945 2.41 1 0.45 0 0
Scholar [126] 817 193 4.2 1 10 10 2.50 147416275 3.38 1.02 0.68 0 0.02
WikiSQL [110] 80654 80257 1 26531 26531 1 6.34 17 1 1 0.28 0 0
Advising [127] 4387 205 21.4 1 15 15 7.40 332596 3.41 1.21 0.40 0 0.11
Spider [45] 11840 6448 1.8 206 1056 5.13 5.01 8980 1.83 1.17 0.54 0 0
SParC [128] 10228 8981 1.1 166 876 5.28 5.14 9665 1.58 1.10 0.44 0 0
CoSQL [129] 8350 8007 1 166 876 5.28 5.14 9665 1.54 1.11 0.42 0 0
CSpider [130] 11840 6408 1.8 206 1056 5.13 5.01 8980 1.83 1.17 0.54 0 0
MIMICSQL [131] 20000 10000 2 - - - - - 1.74 1 0.84 0 0
SQUALL [132] 11276 8296 1.4 2108 4028 1.91 9.18 71 1.22 1.29 0.40 0.03 0.16
FIBEN [133] 300 233 1.3 1 152 152 2.46 11668125 5.59 1.56 0.97 0 0.04
ViText2SQL [134] 9693 5223 1.9 166 876 5.28 5.14 9665 1.17 1.12 0.54 0 0
DuSQL [135] 25003 20308 1.2 208 840 4.04 5.29 20 1.49 1.25 0.73 0 0.30
PortugueseSpider [136] 9693 5275 1.8 166 876 5.28 5.14 9665 1.85 1.17 0.54 0 0
CHASE [137] 15408 13900 1.1 350 1609 4.60 5.19 4594 1.81 1.16 0.31 0 0
Spider-Syn [138] 1034 550 1.9 166 876 5.28 5.14 9665 1.68 1.17 0.59 0 0
Spider-DK [139] 535 283 1.9 169 887 5.25 5.14 9494 1.71 1.16 0.54 0 0
Spider-Realistic [140] 508 290 1.8 166 876 5.28 5.14 9665 1.79 1.21 0.50 0 0
KaggleDBQA [141] 272 249 1.1 8 17 2.12 10.53 595075 1.25 1.05 0.69 0 0.04
SEDE [142] 12023 11421 1.1 1 29 29 7.28 - 1.90 1.29 0.94 0.49 0.49
MT-TEQL [143] 489076 4525 108.1 489076 3279004 6.70 5.51 - 1.69 1.15 0.53 0 0
PAUQ [144] 9876 5497 1.8 166 876 5.28 5.14 9693 1.82 1.17 0.53 0 0
knowSQL [102] 28468 - - 488 - - - - - - - - -
[Link] [145] 15269 3847 4 549 2197 4 5.54 28460 1.81 1.19 0.52 0 0
BIRD [52] 10962 10840 1 80 611 7.64 7.14 4585335 2.07 1.09 0.61 0.20 0.27
AmbiQT [146] 3046 3128 1 166 876 5.28 5.14 9665 1.85 1.17 0.51 0 0.01
ScienceBenchmark [147] 5031 3654 1.4 - - - - - 1.45 1 0.24 0 0.07
BULL [50] 7932 5864 1.4 3 78 26 14.96 85631 1.22 1 0.18 0.42 0.05
BookSQL [148] 78433 39530 2 1 7 7 8.86 1012948 1.25 1.12 0.78 0.39 0.22
Archer [149] 518 260 2 10 68 6.8 6.81 31365.3 3.89 3.07 1.77 0.1 3.55
Spider2-Lite [150] 527 527 1 264 6259 23.71 35.61 - 6.53 5.10 3.57 1.60 2.94

across various domains. Subsequently, Spider [45] was in- perform well in real-world applications. KaggleDBQA [141]
troduced, containing more complex relational databases with includes database documents, such as column and table de-
multiple tables. Recently, the BIRD [52] has further advanced scriptions. Similarly, Spider-DK [139] expands the Spider
complexity by including SQL functions and operations absent development set by adding five types of domain knowledge to
in Spider, providing a greater challenge for Text-to-SQL. NL questions, testing systems’ ability to use this information.

Multi-Turn Text-to-SQL Datasets. With advancements in Text-to-SQL Datasets with Ambiguous Questions. In real-
Text-to-SQL, multi-turn datasets have been developed to sup- world Text-to-SQL tasks, ambiguities often arise, such as
port interactive dialogues. SParC [128] is a cross-domain and semantic ambiguities in NL and overlapping database schemas,
multi-turn dataset with about 4.3K NL questions, totaling over making ambiguity-focused evaluation increasingly important.
12K (NL, SQL) pairs, each NL questions requiring contextual AmbiQT [146] is the first dataset designed to assess ambiguity
understanding across turns. CoSQL [129], collected using a coverage, comprising four ambiguity types. Each NL question
Wizard-of-Oz setup, includes over 30K turns and introduces maps to two valid SQL queries, reflecting specific ambiguities.
additional challenges like unanswerable questions, further test- Synthetic Text-to-SQL Datasets. MIMICSQL [131] em-
ing context comprehension. ploys a template-based approach to generate initial template
Text-to-SQL Datasets with Robustness Testing. In real- questions and corresponding SQL queries, though manual
world applications, Text-to-SQL systems must handle diverse refinement is required to make questions more natural. Sci-
user groups and databases, emphasizing robustness. Spider- enceBenchmark [147] also uses templates for initial SQL
Syn [138] simulates user unfamiliarity with schemas by using generation but leverages GPT-3 for SQL-to-NL translation.
synonyms in NL questions, while [Link] [145] applies 17
types of perturbations to databases, NL questions, and SQL B. In-depth Analysis of Existing Text-to-SQL Datasets
queries for comprehensive robustness evaluation. To analyze and compare Text-to-SQL datasets complexity,
we use the NL2SQL360 [46] system for statistical evaluation,
Text-to-SQL Datasets with SQL Efficiency Testing. Real-
as shown in Table II, We measure the Redundancy, including
world databases often hold vast amounts of data, and a single
the number of NL questions, SQL queries, and their ratio. DB
NL may correspond to multiple SQL queries with different
Complexity covers the total databases, total tables, average
execution efficiencies. BIRD [52] introduces a metric for
tables per database, average columns per table, and average
evaluating SQL execution efficiency called the Valid Efficiency
records per database. Query Complexity measures the average
Score (VES), which will be further discussed in Section VIII.
number of tables, SELECT keywords, aggregate functions,
Knowledge-Augmented Text-to-SQL Datasets. Domain- scalar functions, and mathematical computations in each SQL
specific knowledge is essential for Text-to-SQL systems to query. For datasets without public dev/test splits, such as
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 15

CHASE [137], only statistics for public splits are reported. For execution result sets but do not have the exact string match
datasets without publicly available data, like knowSQL [102], with the ground-truth SQL queries. It can be computed as
( Ni=1 1(Yi =Ŷi ))
P
values in Table II are marked with “–”. follows: SM = .
N
From the Redundancy Measure perspective, we observe a
Component-Match Accuracy (CM) [45]. This metric evaluates
trend from early datasets to recent ones where datasets have
the detailed performance of the Text-to-SQL system by mea-
grown in size. Specifically, MT-TEQL [143] stands out with
suring the exact matching of different SQL components such
the highest number of NL questions and the largest ratio of NL
as SELECT, WHERE and others between the ground-truth
questions to SQL queries due to its automated transformation
SQL query and the predicted SQL query. For a specific SQL
of NL questions, generating a large volume of variants.
component C. The computation can be formalized as follows:
In terms of Database Complexity, the number of databases
1(YiC = ŶiC )
PN
and tables within each dataset aligns with its intended task.
Single-domain datasets, such as BookSQL [148], generally CM C = i=1 ,
N
contain fewer databases, while those aimed at robustness where YiC is the component of SQL query Yi . To correctly de-
evaluation, like [Link] [145] and MT-TEQL [143], include termine if an SQL component matches, some SQL components
a larger number of databases. (e.g.,WHERE) do not consider order constraints.
Regarding Query Complexity, datasets like FIBEN [133] Exact-Match Accuracy (EM) [45]. This metric is based on the
and SEDE [142] feature SQL queries with multiple tables Component-Match Accuracy (CM) and measures whether all
and aggregate functions, mirroring complexities in real-world SQL components C = {Ck } of the predicted SQL query match
financial domains and Stack Exchange sites. Recent datasets the ground-truth SQL query. It can be computed as follows:
also emphasize Scalar Functions and Mathematical Computa-
i=1 1( Ck ∈C Yi
PN Ck
= ŶiCk )
V
tions, adding structural challenges.
EM = .
Discussion. Despite the increasing number of datasets pro- N
posed by the Text-to-SQL community, a gap in SQL complex- Valid Efficiency Score (VES) [52]. This metric measures the
ity remains compared to real-world scenarios. Current datasets execution efficiency of valid SQL queries. It considers both
typically feature fewer SELECT keywords, indicating a lack the accuracy and efficiency of SQL execution, which can be
of nested queries and complex set operations. Additionally, computed as follows:
challenges involving Scalar Functions and Mathematical Com-
putations require further focus. We encourage the community
1(Vi = V̂i ) · R(Yi , Ŷi )
PN s
i=1 E(Yi )
to propose datasets addressing these complexities. V ES = , R(Yi , Ŷi ) = ,
N E(Ŷi )
VIII. E VALUATION AND E RROR A NALYSIS where R(·) measures the relative execution efficiency of the
predicted SQL query compared to the ground-truth SQL query,
In this section, we introduce key evaluation metrics for Text-
eliminating uncertainties due to machine status. E(·) measures
to-SQL solutions (Section VIII-A), review toolkits for low-cost
the efficiency of specific SQL query, which can be refer to
and comprehensive evaluation (Section VIII-B), and provide
execution time, memory usage and more.
an error taxonomy for analyzing SQL errors in the Text-to-SQL
process (Section VIII-C). Query Variance Testing (QVT) [46]. This metric measures the
robustness of an Text-to-SQL system in handling variations in
NL queries. For a given SQL query Yi , there are often multiple
A. Evaluation Metrics corresponding NL queries, represented as pairs {(Q1 , Yi ), (Q2 ,
Evaluation metrics are crucial for measuring Text-to-SQL Yi ), . . . , (Qm , Yi )}. The QVT metric is calculated as:
performance. We define N as the dataset size, Qi as the NL
j=1 1 (F(Qij ) = Yi )
N Pmi !
question of the i-th example, Vi as the execution result set of 1 X
the ground-truth SQL query Yi and V̂i as the execution result QV T = ,
N i=1 mi
set of the SQL query Ŷi generated by the Text-to-SQL solution.
Execution Accuracy (EX) [45]. This metric evaluates the per- where mi is the number of different NL variations for the SQL
formance of the Text-to-SQL system by comparing whether query Yi , and F(Qij ) is the predicted SQL query for the j-th
the execution result sets of the ground-truth SQL queries and NL variation of Yi .
the predicted SQL queries are identical. It can be computed by:
1(V =V̂ )
PN
EX = i=1 N i i , where 1(·) is an indicator function that B. Text-to-SQL Evaluation Toolkits
equals 1 if the condition inside is satisfied, and 0 otherwise. Recent Text-to-SQL solutions have achieved remarkable
Note that false negatives could occur because different SQL performance on various Text-to-SQL benchmarks. However, in
queries corresponding to semantically different NL queries real-world applications, variations in NL query styles, database
may produce identical execution result sets. schemas, and SQL query characteristics across domains make
String-Match Accuracy (SM) [110]. This metric, also called it difficult to fully assess system robustness using standard
Logical Form Accuracy, simply compares whether the ground- benchmark metrics alone. To address this, recent toolkits [46],
truth SQL query and the predicted SQL query are identical as [143] have been developed to provide a more comprehensive
strings. It may penalize SQL queries that produce the correct evaluation of Text-to-SQL systems in practical scenarios.
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 16

MT-TEQL [143] is a unified framework for evaluating • Mutual Exclusivity: Each error type should be clearly
the performance of Text-to-SQL systems in handling real- distinct to avoid classification ambiguity.
world variations in NL queries and database schemas. It • Extensibility: The taxonomy should be adaptable to in-
is based on a metamorphic testing approach, implement- clude emerging error types as Text-to-SQL evolves.
ing semantic-preserving transformations of NL queries and • Practicality: It should be practical, enabling users to
database schemas to generate their variants without manual diagnose and address errors in real-world scenarios.
efforts automatically. It includes four types of transformations
for NL queries (e.g. Prefix Insertion) and eight types of Our Taxonomy for Text-to-SQL Errors Analysis. Following
transformations for database schemas: (e.g. Table Shuffle). these principles, we developed a two-level Text-to-SQL error
NL2SQL360 [46] is a multi-angle evaluation framework of- analysis taxonomy:
fering fine-grained assessments of Text-to-SQL systems across • Error Localization: The first level identifies specific SQL
diverse scenarios (Figure 1(c)). Unlike MT-TEQL, it empha- components where errors occur, such as the SELECT
sizes varied SQL query characteristics in different applications, or WHERE clause. Pinpointing error locations enables
such as aggregate functions, nested queries, or top-k queries targeted adjustments and enhances correction efficiency.
typical of the Business Intelligence scenario. Comprising six • Cause of Error: The second level focuses on the under-
core components, Dataset, Model Zoo, Metrics, Dataset Filter, lying reasons for the error. For instance, errors in the
Evaluator, and Analysis, NL2SQL360 provides a unified, WHERE clause values may indicate the model’s limita-
model-agnostic interface for systematic evaluations. Users can tions in database content retrieval or interpretation.
apply both public and private datasets, customize metrics for
specific scenarios, and analyze performance on subsets with Discuss the Application of the Two-level Error Taxonomy.
scenario-specific SQL characteristics, offering valuable insights We collected and classified errors from DIN-SQL [5] on
into Text-to-SQL system effectiveness across applications. the Spider [45] using our proposed taxonomy. As shown
in Figure 1(d), only 1.8% of the errors fall into the Others
category, suggesting our taxonomy is practical and effective.
C. A Taxonomy for Text-to-SQL Errors Analysis Nonetheless, we recognize that developing a complete
Error analysis involves examining model errors to identify and universally applicable Text-to-SQL error taxonomy is
limitations and guide corrective actions for improved perfor- inherently iterative. We encourage continued efforts from the
mance. In this section, we first review the existing Text-to- community to refine and expand this taxonomy over time.
SQL error taxonomy. We then propose design principles and
introduce a two-level Text-to-SQL errors taxonomy.
IX. P RACTICAL G UIDANCE FOR T EXT- TO -SQL
Existing Taxonomies for Text-to-SQL Errors Analysis.
Recent Text-to-SQL research [5], [33], [59], [151]–[153] has In this section, we provide practical guidance for developing
increasingly incorporated error analysis, proposing various Text-to-SQL solutions, considering key factors and scenarios.
error taxonomies. Ning et al. [152] introduced a detailed
error taxonomy based on two dimensions: (1) Syntactic di-
mension identifies specific SQL parts where errors occur,
A. Data-Driven Roadmap for Text-to-SQL
organized by keywords such as WHERE and JOIN. (2) Se-
mantic dimension indicates misinterpretations of the natural In Figure 11(a), we outline a strategic roadmap designed to
language description, such as errors in understanding table optimize LLMs for Text-to-SQL task, based on data privacy
names. SQL-PaLM [153] categorizes errors into five types: and data volume. Data privacy affects the choice of open-
(1) Schema Linking, irrelevant or missing table/column se- source and closed-source LLMs, while data volume affects
lection; (2) Database Content, misinterpreting data values; the strategies for optimization for training and inference.
(3) Knowledge Evidence, failing to utilize external hints; Condition 1: Data Privacy. For privacy-sensitive data, open-
(4) Reasoning, lacking intermediate logical steps; and (5) Syn- source LLMs are preferable, as closed-source models typi-
tax, invalid SQL format. NL2SQL-BUGs [33] focuses on cally use external APIs, potentially exposing data to external
the analysis of semantic errors, organizing them into 9 main servers. Open-source models allow full control over local train-
categories and 31 subcategories. It further proposes a new ing and inference, providing stronger data privacy protection.
benchmark for evaluating models’ error detection capabilities,
advancing automated error analysis in the Text-to-SQL. Condition 2: Data Volume. For open-source LLMs, optimiza-
tion is possible in both training and inference phases, while
Taxonomy Principles for Text-to-SQL Errors Analysis. closed-source LLMs allow only inference-stage optimization
Current error taxonomies in Text-to-SQL are often specific due to limited access. With extensive Text-to-SQL data, pre-
to particular datasets, limiting their general applicability. To training enhances performance; fine-tuning is suitable for
address these issues, a standardized and effective taxonomy is datasets with hundreds to thousands of (NL, SQL) pairs. In
essential. We propose the following principles [154] to guide low-data scenarios, few-shot learning is recommended, while
the development of an Text-to-SQL error taxonomy: zero-shot methods are essential when labeled data is unavail-
• Comprehensiveness: The taxonomy should cover all pos- able. Hardware resources and API costs are also important
sible error types in the Text-to-SQL translation process. considerations in selecting the best optimization strategy.
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 17

Massive Text-to- Pre-train Large amount of Hardware

Privacy-reserved data Open-source LLMs SQL-related data
Optimize training
Guidance for
Large amount of
Fine-tune Small amount of Hardware
Data-driven LLM labeled (NL,SQL,DB)
Optimization Privacy-free data Closed-source LLMs Optimize inference Small amount of

labeled (NL, SQL,DB) Few-shot Large amount of udget API B

Without
Zero-shot Small amount of udget
labeled (NL, SQL,DB)
API B

Conditions Suggestions Resources

a Data-Driven oadmap for Optimizin LLM for e t-to- L
e t-to- L Guidance
( ) A R g s T x SQ

Schema Lin in Time cost Noise from redundant database schema

To en cost
T x SQ

Complex database
k ↓ ↓
k g

schema with numerous Mismatch between

Database Content The accuracy of the SQL value selection
↑
↑

tables and columns database content and etrieva Time cost To en cost The semantic understanding ability
The adaptability of the system
↑ k ↑ ↑

dditional nformation
R l

NL details H ave strong domain specificity c uisition

A I
Time cost To en cost
↑

in NL and database A q

Tas -specific
↑ k ↑

The ability of model s understanding

Need to guidance model Prompt Strategy To en cost
k ' ↑

understand Text-to-SQL tas k

k ↑

The success rate of SQL execution Have effective to bridge NL and SQL The gap of NL and SQL
Time cost (To en cost ) SQL Correction
↑ IR

ntermediate
↓

Strategies The generated SQL

epresentation Time cost
↑ k ↑ I

Output consistency ↑
Output has syntax errors R
System complexity
↑

Time cost (To en cost ) Consistency The SQL generated from multiple
↑

Execution results
inference paths is inconsistent
↑ k ↑

Filter non-executable SQL Execution-Guided

Time cost Strategies
↑ can be accessed
Other scenarios to be considered... Scenarios ecommendations R dvantages Disadvantages
A

b Deci ion Flow for ecommendin e t-to- L Module

( ) s R gT x SQ s

Fig. 11: A Data-Driven Roadmap and a Decision Flow for Recommending Text-to-SQL modules.

B. Decision Flow of Selecting Text-to-SQL Modules and fix potential semantic errors. Finally, current methods
In Figure 11(b), we present recommendations for choosing show limited adaptability to new domains and rely heavily
Text-to-SQL modules based on specific scenarios, highlighting on high-quality training data; how to automatically generate
both benefits and trade-offs. Below, we outline two examples. targeted training samples based on model feedback remains an
open problem. These limitations reveal shortcomings in cross-
Scenario 1: Complex Database Schema with Numerous Tables
database scalability, inference efficiency, system reliability,
and Columns. In this case, using Schema Linking strategies is
and data adaptability, highlighting the need for more efficient,
advisable. This reduces token costs and minimizes noise from
trustworthy, and scalable Text-to-SQL solutions.
irrelevant schema elements, enhancing efficiency. However, it
also incurs additional time costs. Open-Domain Text-to-SQL Problem. In real-world scenar-
Scenario 2: Execution Results Can be Accessed. Here, ios like government open data platforms, citizens may ask
Execution-Guided Strategies are recommended, as they im- questions that require querying multiple databases and aggre-
prove system performance by filtering out non-executable SQL gating results. For example, answering “What is the average
queries. The downside is the increased time required for query processing time for tax returns in the last five years?” requires
execution, which can be substantial with large databases. retrieving tables from multiple databases (e.g., tax records,
In summary, while each module offers unique advantages processing logs, and statistical reports) and generating mul-
for specific Text-to-SQL scenarios, it is essential to balance tiple SQL queries over them. Unlike traditional Text-to-SQL,
these benefits with the potential drawbacks in system design. where a single target database is specified by the user, Open
Text-to-SQL may need to generate multiple SQL queries that
X. L IMITATIONS AND O PEN P ROBLEMS access different databases for a single NL.
We analyze the limitations of LLM-based methods and pro- Thus, the Open Text-to-SQL problem introduces unique
pose corresponding open problems, highlighting unresolved challenges, including: (1) database retrieval: accurately iden-
challenges and suggesting directions for future research. tifying and retrieving relevant databases from a vast array of
data sources; (2) handling heterogeneous schemas: integrating
Limitations of Current LLM-based Solutions. Although data with varied structures and terminologies, requiring ad-
recent LLM-based Text-to-SQL methods have made signifi- vanced schema matching and linking techniques; (3) answer
cant progress, they still face several challenges when dealing aggregation: inferring final answers from multiple SQL queries
with complex queries in real-world scenarios. First, existing across databases, which demands methods to plan query
methods are typically trained and executed on a single, fixed order, resolve conflicts, and ensure consistency; (4) domain
database, which limits their ability to handle open environ- adaptation: generalizing models across domains to address
ments that require cross-database queries and multi-source differences in terminology and structure; (5) scalability and
data aggregation. Second, although LLMs have strong natural efficiency: managing large data volumes while maintaining
language understanding capabilities, they incur high token performance; and (6) evaluating and benchmarking: devel-
consumption during inference, leading to high costs and low oping metrics and datasets that accurately reflect real-world
efficiency. In addition, most Text-to-SQL methods lack inter- complexity for Open Text-to-SQL solutions.
pretability and debugging mechanisms, making it difficult for
users to understand how the model generates SQL or to detect Develop Cost-effective Text-to-SQL Methods. LLM-based
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 18

TABLE III: Resource Consumption Statistics on the Spider. Therefore, an interesting research problem is to automatically
Method Base Model API Tokens/SQL Latency/SQL (s) and incrementally generate (NL, SQL) pairs based on the
RESDSQL [7] PLM - 1.91
RESDSQL+NatSQL [7] PLM - 1.97
model performance. Specifically, by incorporating insights
ZeroNet [44] PLM+LLM 377 3.72 from evaluation metrics and evaluation results, we can identify
DIN-SQL [5] LLM 3579 10.34 specific weaknesses of the model. Using this information, we
can synthesize training data that continually evolves with the
help of LLMs to cover a broader range of domains.
Text-to-SQL methods show great potential but are limited
by high token consumption, leading to increased costs and
XI. C ONCLUSION
slower inference times. In contrast, PLM-based Text-to-SQL
methods excel at handling complex SQL queries and accurately In this paper, we comprehensively review Text-to-SQL tech-
interpreting database schemas. A promising approach is to niques from a lifecycle perspective in the LLM era. We for-
combine the strengths of both, developing modular Text-to- mally define the Text-to-SQL task, discuss key challenges, and
SQL solutions or using a multi-agent framework to integrate propose a taxonomy based on underlying language models.
LLMs and PLMs for the Text-to-SQL task (as shown in We summarize key modules of language model-driven meth-
Table III). In parallel, efforts have aimed to improve LLM- ods, including pre-processing, translation, and post-processing
based efficiency. EllieSQL [155] employs complexity-aware strategies. Furthermore, we analyze benchmarks and evalu-
routing to enhance cost-efficiency by assigning queries to ation metrics, highlighting their characteristics and common
suitable LLM-based generators. errors. We also offer a practical roadmap for adapting LLMs
to Text-to-SQL tasks and maintain an online handbook with
Make Text-to-SQL Solutions Trustworthy. Ensuring Text- the latest advancements, discussing ongoing challenges and
to-SQL solutions are trustworthy is essential for generating open problems.
accurate and reliable SQL, mitigating risk, and reducing the
need for manual intervention. Topics include the following:
R EFERENCES
Interpreting Text-to-SQL Solutions. Understanding the rea-
[1] Z. Gu, J. Fan, N. Tang, and et al., “Few-shot text-to-sql translation
soning behind a Text-to-SQL model’s performance enhances using structure and content prompt learning,” SIGMOD, 2023.
confidence in its reliability. Explainable AI techniques [156], [2] Z. Chen, S. Chen, M. White, R. J. Mooney, and et al., “Text-to-sql
[157], such as surrogate models [158] and saliency maps [159], error correction with language models of code,” in ACL, 2023.
[3] L. Wang and et al., “Proton: Probing schema linking information from
aim to reveal model decisions. However, their effectiveness pre-trained language models for text-to-sql parsing,” in KDD, 2022.
in Text-to-SQL contexts, especially with combined LLM and [4] A. Liu, X. Hu, L. Lin, and L. Wen, “Semantic enhanced text-to-sql
PLM, remains an open question. In addition, multi-agent parsing via iteratively learning schema linking graph,” in KDD, 2022.
[5] M. Pourreza and D. Rafiei, “Din-sql: Decomposed in-context learning
LLM frameworks [160] improve reliability by splitting Text- of text-to-sql with self-correction,” NeurIPS, 2024.
to-SQL into specialized sub-tasks. Although this approach [6] D. Gao, H. Wang, Y. Li, and et al., “Text-to-sql empowered by large
improves robustness, coordinating agents to ensure consistent language models: A benchmark evaluation,” Proc. VLDB Endow., 2024.
[7] H. Li, J. Zhang, C. Li, and H. Chen, “Resdsql: Decoupling schema
and optimized performance remains a major challenge. linking and skeleton parsing for text-to-sql,” in AAAI, 2023.
Text-to-SQL Debugging Tools. Inspired by compiler design, [8] N. Tang, C. Yang, J. Fan, and et al., “Verifai: Verified generative AI,”
a debugger for Text-to-SQL could improve accuracy and relia- in CIDR. [Link], 2024.
[9] Y. Zhu, S. Du, B. Li, and et al., “Are large language models good
bility by measuring semantic and syntactic errors in generated statisticians?” CoRR, 2024.
SQL queries. Such tools would detect potential errors, enable [10] Y. Xie, Y. Luo, G. Li, and N. Tang, “Haichart: Human and AI paired
users to examine the SQL generation process and identify visualization system,” Proc. VLDB Endow., 2023.
[11] L. Shen, E. Shen, Y. Luo, and et al., “Towards natural language
mismatches [33], [34]. However, achieving this goal presents interfaces for data visualization: A survey,” IEEE TVCG., 2023.
significant challenges. Traditional code compilers primarily [12] Y. Luo, X. Qin, C. Chai, and et al., “Steerable self-driving data
capture syntactic errors, while Text-to-SQL debugging must visualization,” IEEE Trans. Knowl. Data Eng., 2022.
[13] Y. Luo, N. Tang, G. Li, and et al., “Natural language to visualization by
also address semantic errors, i.e., ensuring that the generated neural machine translation,” IEEE Trans. Vis. Comput. Graph., 2022.
SQL query accurately reflects the intent of the NL query. [14] J. Tang, Y. Luo, M. Ouzzani, and et al., “Sevi: Speech-to-visualization
Interactive Text-to-SQL Tools. These tools are essential for through neural machine translation,” in SIGMOD, 2022.
[15] Y. Luo, N. Tang, G. Li, and et al., “Synthesizing natural language to
empowering professional users (e.g., DBAs) to create complex visualization (NL2VIS) benchmarks from NL2SQL benchmarks,” in
SQL queries that span multiple databases, often exceeding 50 SIGMOD, 2021.
lines of code. A key feature is the model’s ability to decom- [16] X. Qin, Y. Luo, N. Tang, and et al., “Making data visualization more
efficient and effective: a survey,” VLDB J., 2020.
pose complex queries into manageable sub-queries, reducing [17] Y. Luo, X. Qin, N. Tang, and et al., “Deepeye: Towards automatic data
cognitive load and enabling DBAs to focus on each part visualization,” in ICDE. IEEE Computer Society, 2018.
before reassembling them. Supporting both bottom-up and top- [18] Y. Luo, X. Qin, N. Tang, G. Li, and X. Wang, “Deepeye: Creating good
data visualizations by keyword search,” in SIGMOD. ACM, 2018.
down workflows, such tools enable users to iteratively refine [19] X. Zhou, Z. Sun, and G. Li, “DB-GPT: large language model meets
outputs, align SQL generation with intent, and integrate model database,” Data Sci. Eng., 2024.
assistance with domain expertise. [20] S. Amer-Yahia and et al., “From large language models to databases
and back: A discussion on research and education,” SIGMOD, 2023.
Adaptive Training Data Synthesis. Learning-based Text-to- [21] W. Zhang, Y. Wang, Y. Song, and et al., “Natural language interfaces
for tabular data querying and visualization: A survey,” TKDE, 2024.
SQL models often fail to generalize to unseen domains, partly [22] G. Katsogiannis-Meimarakis and G. Koutrika, “A survey on deep
due to limited training data coverage, quality, and diversity. learning approaches for text-to-sql,” The VLDB Journal, 2023.
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 19

[23] N. Deng, Y. Chen, and Y. Zhang, “Recent advances in text-to-sql: A [58] Y. Qin, C. Chen, Z. Fu, and et al., “Route: Robust multitask tuning and
survey of what we have and what we expect,” in COLING, 2022. collaboration for text-to-sql,” arXiv preprint arXiv:2412.10138, 2024.
[24] H. Kim, B.-H. So, W.-S. Han, and et al., “Natural language to sql: [59] S. Talaei, M. Pourreza, Y.-C. Chang, and et al., “Chess: Contextual
Where are we today?” Proc. VLDB Endow., 2020. harnessing for efficient sql synthesis,” arXiv:2405.16755, 2024.
[25] L. Shi, Z. Tang, N. Zhang, X. Zhang, and Z. Yang, “A survey on [60] M. Pourreza and D. Rafiei, “Dts-sql: Decomposed text-to-sql with small
employing large language models for text-to-sql tasks,” 2024. large language models,” arXiv:2402.01117, 2024.
[26] A. Mohammadjafari, A. S. Maida, and R. Gottumukkala, “From natural [61] G. Qu, J. Li, B. Li, and et al., “Before generation, align it! a
language to sql: Review of llm-based text-to-sql systems,” 2024. novel and effective strategy for mitigating hallucinations in text-to-sql
[27] X. Zhu, Q. Li, L. Cui, and Y. Liu, “Large language model enhanced generation,” arXiv:2405.15307, 2024.
text-to-sql generation: A survey,” 2024. [62] Z. Li and et al., “Pet-sql: A prompt-enhanced two-stage text-to-sql
[28] Z. Hong, Z. Yuan, Q. Zhang, and et al., “Next-generation database framework with cross-consistency,” arXiv:2403.09732, 2024.
interfaces: A survey of llm-based text-to-sql,” arXiv:2406.08426, 2024. [63] H. Zhang, R. Cao, H. Xu, and et al., “Coe-sql: In-context learning for
[29] F. Őzcan, A. Quamar, J. Sen, and et al., “State of the art and open multi-turn text-to-sql with chain-of-editions,” arXiv:2405.02712, 2024.
challenges in natural language interfaces to data,” in SIGMOD, 2020. [64] T. Ren, Y. Fan, Z. He, , and et al., “Purple: Making a large language
[30] Y. Li and D. Rafiei, “Natural language data management and interfaces: model a better sql writer,” arXiv:2403.20014, 2024.
Recent development and open challenges,” in SIGMOD, 2017. [65] Y. Fan, Z. He, T. Ren, and et al., “Metasql: A generate-then-rank
framework for natural language to sql translation,” 2024.
[31] G. Katsogiannis-Meimarakis and et al., “Natural language interfaces
[66] Y. Xie, X. Jin, T. Xie, and et al., “Decomposition for enhancing at-
for databases with deep learning,” Proc. VLDB Endow., 2023.
tention: Improving llm-based text-to-sql through workflow paradigm,”
[32] G. Katsogiannis-Meimarakis and G. Koutrika, “A deep dive into deep
arXiv:2402.10671, 2024.
learning approaches for text-to-sql systems,” in SIGMOD, 2021.
[67] X. Dong, C. Zhang, Y. Ge, and et al., “C3: Zero-shot text-to-sql with
[33] X. Liu, S. Shen, B. Li, and et al., “Nl2sql-bugs: A benchmark for chatgpt,” arXiv:2307.07306, 2023.
detecting semantic errors in nl2sql translation,” arXiv, 2025. [68] D. Rai, B. Wang, Y. Zhou, and et al., “Improving generalization
[34] Y. Yang, Z. Wang, Y. Xia, Wei, and et al., “Automated validation and in language model-based text-to-sql semantic parsing: Two simple
fixing of text-to-sql translation with execution consistency,” 2025. semantic boundary-based techniques,” in ACL, 2023.
[35] N. Rajkumar and et al., “Evaluating the text-to-sql capabilities of large [69] H. Zhang, R. Cao, L. Chen, and et al., “Act-sql: In-context learning for
language models,” CoRR, 2022. text-to-sql with automatically-generated chain-of-thought,” in Findings
[36] F. Li and H. V. Jagadish, “Nalir: an interactive natural language of EMNLP, 2023.
interface for querying relational databases,” in ACM SIGMOD, 2014. [70] S. Chang and E. Fosler-Lussier, “Selective demonstrations for cross-
[37] T. Yu, C.-S. Wu, X. V. Lin, and et al., “Grappa: Grammar-augmented domain text-to-sql,” in Findings of EMNLP, 2023.
pre-training for table semantic parsing,” in ICLR, 2021. [71] H. Fu, C. Liu, B. Wu, and et al., “Catsql: Towards real world natural
[38] C. Xiao, M. Dymetman, and C. Gardent, “Sequence-based structured language to sql applications,” Proc. VLDB Endow., 2023.
prediction for semantic parsing,” in ACL, 2016. [72] A. Bazaga, P. Liò, and et al., “Sqlformer: Deep auto-regressive query
[39] K. Lin, B. Bogin, M. Neumann, and et al., “Grammar-based neural graph generation for text-to-sql translation,” arXiv:2310.18376, 2023.
text-to-sql generation,” arXiv:1905.13326, 2019. [73] Y. Xiang, Q.-W. Zhang, X. Zhang, and et al., “G3r: A graph-guided
[40] B. Bogin, M. Gardner, and et al., “Representing schema structure with generate-and-rerank framework for complex and cross-domain text-to-
graph neural networks for text-to-sql parsing,” arXiv:1905.06241, 2019. sql generation,” in Findings of ACL, 2023.
[41] J. Devlin and et al., “Bert: Pre-training of deep bidirectional transform- [74] Y. Hu, Y. Zhao, J. Jiang, and et al., “Importance of synthesizing high-
ers for language understanding,” in NAACL, 2019. quality data for text-to-sql parsing,” in Findings of ACL. Association
[42] C. Raffel, N. Shazeer, A. Roberts, and et al., “Exploring the limits of for Computational Linguistics, 2023.
transfer learning with a unified text-to-text transformer,” JMLR, 2020. [75] L. Zeng, S. H. K. Parthasarathi, and D. Hakkani-Tur, “N-best hypothe-
[43] J. Li and et al., “Graphix-t5: Mixing pre-trained transformers with ses reranking for text-to-sql systems,” in SLT, 2023.
graph-aware layers for text-to-sql parsing,” arXiv:2301.07507, 2023. [76] J. Qi, J. Tang, Z. He, and et al., “Rasat: Integrating relational structures
[44] Z. Gu, J. Fan, N. Tang, and et al., “Interleaving pre-trained language into pretrained seq2seq model for text-to-sql,” arXiv:2205.06983, 2022.
models and large language models for zero-shot nl2sql generation.” [77] T. Scholak and et al., “Picard: Parsing incrementally for constrained
arXiv:2306.08891, 2023. auto-regressive decoding from language models,” 2021.
[45] T. Yu, R. Zhang, K. Yang, and et al., “Spider: A large-scale human- [78] C. Gao, B. Li, W. Zhang, and et al., “Towards generalizable and robust
labeled dataset for complex and cross-domain semantic parsing and text-to-sql parsing,” arXiv:2210.12674, 2022.
text-to-sql task,” in EMNLP 2018, 2018. [79] B. Hui and et al., “S2sql: Injecting syntax to question-schema interac-
[46] B. Li, Y. Luo, C. Chai, and et al., “The dawn of natural language to tion graph encoder for text-to-sql parsers,” in Findings of ACL, 2022.
sql: Are we fully ready?” Proc. VLDB Endow., 2024. [80] B. Wang, R. Shin, X. Liu, and et al., “RAT-SQL: relation-aware schema
[47] W. X. Zhao, K. Zhou, J. Li, and et al., “A survey of large language encoding and linking for text-to-sql parsers,” in ACL, 2020.
models,” arXiv:2303.18223, 2023. [81] O. Rubin and J. Berant, “Smbop: Semi-autoregressive bottom-up
semantic parsing,” arXiv:2010.12412, 2020.
[48] S. Minaee, T. Mikolov, N. Nikzad, and et al., “Large language models:
[82] J. Huang, Y. Wang, Y. Wang, and et al., “Relation aware semi-
A survey,” arXiv:2402.06196, 2024.
autoregressive semantic parsing for nl2sql,” arXiv:2108.00804, 2021.
[49] H. Li, J. Zhang, H. Liu, and et al., “Codes: Towards building open-
[83] X. V. Lin and et al., “Bridging textual and tabular data for cross-domain
source language models for text-to-sql,” SIGMOD, 2024.
text-to-sql semantic parsing,” arXiv:2012.12627, 2020.
[50] C. Zhang and et al., “Finsql: Model-agnostic llms-based text-to-sql [84] J. Guo, Z. Zhan, Y. Gao, and et al., “Towards complex text-to-sql in
framework for financial analysis,” in SIGMOD, 2024. cross-domain database with intermediate representation,” in ACL, 2019.
[51] R. Li, L. B. Allal, Y. Zi, and et al., “Starcoder: may the source be with [85] T. Yu, Z. Li, Z. Zhang, and et al., “Typesql: Knowledge-based type-
you!” arXiv:2305.06161, 2023. aware neural text-to-sql generation,” arXiv:1804.09769, 2018.
[52] J. Li, B. Hui, G. Qu, and et al., “Can LLM already serve as A database [86] U. Brunner and K. Stockinger, “Valuenet: A natural language-to-sql
interface? A big bench for large-scale database grounded text-to-sqls,” system that learns from database information,” in ICDE. IEEE, 2021.
in NeurIPS, 2023. [87] F. J. Damerau, “A technique for computer detection and correction of
[53] W. Lei, W. Wang, Z. Ma, and et al., “Re-examining the role of schema spelling errors,” Communications of the ACM, 1964.
linking in text-to-sql,” in EMNLP, 2020. [88] Z. Dong, S. Sun, H. Liu, and et al., “Data-anonymous encoding for
[54] B. Wang, C. Ren, J. Yang, and et al., “Mac-sql: A multi-agent text-to-sql generation,” in EMNLP-IJCNLP, 2019.
collaborative framework for text-to-sql.” CoRR, 2023. [89] T. Brown, B. Mann, N. Ryder, and et al., “Language models are few-
[55] M. Pourreza, H. Li, R. Sun, and et al., “Chase-sql: Multi-path reasoning shot learners,” NeurIPS, 2020.
and preference optimized candidate selection in text-to-sql,” 2024. [90] D. Lee, C. Park, J. Kim, and et al., “Mcs-sql: Leveraging multi-
[56] B. Li, J. Zhang, J. Fan, Y. Xu, C. Chen, N. Tang, and Y. Luo, “Alpha- ple prompts and multiple-choice selection for text-to-sql generation,”
sql: Zero-shot text-to-sql using monte carlo tree search,” in Forty- arXiv:2405.07467, 2024.
Second International Conference on Machine Learning, ICML 2025, [91] P. Yin, G. Neubig, W.-t. Yih, and et al., “Tabert: Pretraining for joint
Vancouver, Canada, July 13-19, 2025, 2025. understanding of textual and tabular data,” arXiv:2005.08314, 2020.
[57] X. Xie, G. Xu, L. Zhao, and R. Guo, “Opensearch-sql: Enhancing [92] R. Speer, C. Havasi et al., “Representing general relational knowledge
text-to-sql with dynamic few-shot and consistency alignment,” 2025. in conceptnet 5.” in LREC, 2012.
JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 20

[93] P. Indyk and R. Motwani, “Approximate nearest neighbors: towards [130] Q. Min, Y. Shi, and Y. Zhang, “A pilot study for chinese SQL semantic
removing the curse of dimensionality,” in STOC, 1998. parsing,” in EMNLP-IJCNLP, 2019.
[94] S. Robertson, H. Zaragoza et al., “The probabilistic relevance frame- [131] P. Wang, T. Shi, and C. K. Reddy, “Text-to-sql generation for question
work: Bm25 and beyond,” FOUND TRENDS INF RET, 2009. answering on electronic medical records,” in WWW, 2020.
[95] A. V. Aho and M. J. Corasick, “Efficient string matching: an aid to [132] T. Shi and et al., “On the potential of lexico-logical alignments for
bibliographic search,” Communications of the ACM, 1975. semantic parsing to SQL queries,” in Findings of EMNLP, 2020.
[96] F. Li and et al., “Schema-free sql,” in SIGMOD, 2014. [133] J. Sen, C. Lei, A. Quamar, and et al., “Athena++ natural language
[97] T. Yu and et al., “Syntaxsqlnet: Syntax tree networks for complex and querying for complex nested sql queries,” Proc. VLDB Endow., 2020.
cross-domaintext-to-sql task,” arXiv:1810.05237, 2018. [134] A. T. Nguyen, M. H. Dao, and D. Q. Nguyen, “A pilot study of text-
[98] J.-O. Lee and D.-K. Baik, “Semql: a semantic query language for to-sql semantic parsing for vietnamese,” in Findings of EMNLP, 2020.
multidatabase systems,” in CIKM, 1999. [135] L. Wang, A. Zhang, K. Wu, and et al., “Dusql: A large-scale and
[99] R. Zhang and et al., “Editing-based sql query generation for cross- pragmatic chinese text-to-sql dataset,” in EMNLP, 2020.
domain context-dependent questions,” arXiv:1909.00786, 2019. [136] M. A. José and F. G. Cozman, “mrat-sql+ gap: a portuguese text-to-sql
[100] Y. Gan and et al., “Natural SQL: Making SQL easier to infer from transformer,” in BRACIS, 2021.
natural language specifications,” in Findings of EMNLP, 2021. [137] J. Guo, Z. Si, Y. Wang, and et al., “Chase: A large-scale and pragmatic
[101] T. Wolfson, M. Geva, A. Gupta, and et al., “Break it down: A question chinese dataset for cross-database context-dependent text-to-sql,” in
understanding benchmark,” TACL, 2020. ACL/IJCNLP, 2021.
[102] L. Dou, Y. Gao, X. Liu, and et al., “Towards knowledge-intensive text- [138] Y. Gan, X. Chen, Q. Huang, and et al., “Towards robustness of text-
to-sql semantic parsing with formulaic knowledge,” in EMNLP, 2022. to-sql models against synonym substitution,” in ACL/IJCNLP, 2021.
[103] V. Karpukhin, B. Oğuz, S. Min, and et al., “Dense passage retrieval [139] Y. Gan, X. Chen, and M. Purver, “Exploring underexplored limitations
for open-domain question answering,” arXiv:2004.04906, 2020. of cross-domain text-to-sql generalization,” in EMNLP, 2021.
[104] Q. Liu, D. Yang, J. Zhang, J. Guo, B. Zhou, and J. Lou, “Awakening la- [140] X. Deng, A. H. Awadallah, C. Meek, and et al., “Structure-grounded
tent grounding from pretrained language models for semantic parsing,” pretraining for text-to-sql,” in NAACL-HLT, 2021.
in ACL/IJCNLP (Findings), 2021. [141] C. Lee, O. Polozov, and M. Richardson, “Kaggledbqa: Realistic eval-
[105] G. Sui, Z. Li, Z. Li, and et al., “Reboost large language model-based uation of text-to-sql parsers,” in ACL/IJCNLP, 2021.
text-to-sql, text-to-python, and text-to-function–with real applications [142] M. Hazoom and et al., “Text-to-sql in the wild: A naturally-occurring
in traffic domain,” arXiv:2310.18752, 2023. dataset based on stack exchange data,” arXiv:2106.05006, 2021.
[106] K. Clark, “Electra: Pre-training text encoders as discriminators rather [143] P. Ma and S. Wang, “Mt-teql: Evaluating and augmenting neural
than generators,” arXiv:2003.10555, 2020. NLIDB on real-world linguistic and schema variations,” VLDB, 2021.
[107] R. Cao and et al., “Lgesql: line graph enhanced text-to-sql model with [144] D. Bakshandaeva, O. Somov, E. Dmitrieva, V. Davydova, and E. Tu-
mixed local and non-local relations,” arXiv:2106.01093, 2021. tubalina, “PAUQ: text-to-sql in russian,” in Findings of EMNLP, 2022.
[108] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and [145] S. Chang, J. Wang, M. Dong, and et al., “[Link]: A diagnostic
Y. Bengio, “Graph attention networks,” arXiv:1710.10903, 2017. evaluation benchmark towards text-to-sql robustness,” in ICLR, 2023.
[109] X. Xu and et al., “Sqlnet: Generating structured queries from natural [146] A. Bhaskar, T. Tomar, A. Sathe, and S. Sarawagi, “Benchmarking and
language without reinforcement learning,” arXiv:1711.04436, 2017. improving text-to-sql generation under ambiguity,” in EMNLP, 2023.
[110] V. Zhong and et al., “Seq2sql: Generating structured queries from [147] Y. Zhang, J. Deriu, G. Katsogiannis-Meimarakis, and et al., “Sci-
natural language using reinforcement learning,” CoRR, 2017. encebenchmark: A complex real-world benchmark for evaluating nat-
[111] X. Bi, D. Chen, G. Chen, and et al., “Deepseek llm: Scaling open- ural language to SQL systems,” Proc. VLDB Endow., 2023.
source language models with longtermism,” arXiv:2401.02954, 2024. [148] R. Kumar, A. R. Dibbu, S. Harsola, and et al., “Booksql: A large scale
[112] J. Wei, X. Wang, D. Schuurmans, and et al., “Chain-of-thought text-to-sql dataset for accounting domain,” arXiv:2406.07860, 2024.
prompting elicits reasoning in large language models,” NeurIPS, 2022. [149] D. Zheng and et al., “Archer: A human-labeled text-to-sql dataset with
[113] C.-Y. Tai, Z. Chen, T. Zhang, and et al., “Exploring chain-of-thought arithmetic, commonsense and hypothetical reasoning,” in EACL, 2024.
style prompting for text-to-sql,” arXiv:2305.14215, 2023. [150] F. Lei, J. Chen, Y. Ye, and et al., “Spider 2.0: Evaluating language
[114] F. Li and H. V. Jagadish, “Constructing an interactive natural language models on real-world enterprise text-to-sql workflows,” 2024.
interface for relational databases,” Proc. VLDB Endow., 2014. [151] A. Narechania, A. Fourney, B. Lee, and et al., “Diy: Assessing the
[115] B. Eyal, A. Bachar, O. Haroche, and et al., “Semantic decomposition correctness of natural language to sql systems,” in IUI, 2021.
of question and sql for text-to-sql parsing,” arXiv:2310.13575, 2023. [152] Z. Ning, Z. Zhang, T. Sun, and et al., “An empiricmacsqlal study of
[116] D. Lee, “Clause-wise and recursive decoding for complex and cross- model errors and user error discovery and repair strategies in natural
domain text-to-sql generation,” arXiv:1904.08835, 2019. language database queries,” in IUI, 2023.
[117] X. Wang and et al., “Self-consistency improves chain of thought [153] R. Sun, S. O. Arik, H. Nakhost, and et al., “Sql-palm: Improved large
reasoning in language models,” arXiv:2203.11171, 2022. language modeladaptation for text-to-sql,” arXiv:2306.00739, 2023.
[118] M. Renze and E. Guven, “The effect of sampling temperature on [154] T. Taipalus and H. Grahn, “Framework for sql error message design:
problem solving in large language models,” arXiv:2402.05201, 2024. A data-driven approach,” TOSEM, 2023.
[119] A. Kelkar, R. Relan, V. Bhardwaj, and et al., “Bertrand-dr: Improving [155] Y. Zhu, R. Jiang, B. Li, , and et al., “Elliesql: Cost-efficient text-to-sql
text-to-sql using a discriminative re-ranker,” arXiv:2002.00557, 2020. with complexity-aware routing,” 2025.
[120] Y. Luo, Y. Zhou, N. Tang, and et al., “Learned data-aware image [156] S. Ali, T. Abuhmed, S. H. A. El-Sappagh, and et al., “Explainable
representations of line charts for similarity search,” SIGMOD, 2023. artificial intelligence (XAI): what we know and what is left to attain
[121] K. Zhang and et al., “Refsql: A retrieval-augmentation framework for trustworthy artificial intelligence,” Inf. Fusion, 2023.
text-to-sql generation,” in Findings of EMNLP, 2023. [157] H. Zhao, H. Chen, F. Yang, and et al., “Explainability for large language
[122] D. A. Dahl, M. Bates, M. K. Brown, and et al., “Expanding the scope models: A survey,” ACM Trans. Intell. Syst. Technol., 2024.
of the atis task: The atis-3 corpus,” in Workshop on HLT, 1994. [158] Y. Chen, R. Zhong, N. Ri, and et al., “Do models explain them-
[123] J. M. Zelle and R. J. Mooney, “Learning to parse database queries selves? counterfactual simulatability of natural language explanations,”
using inductive logic programming,” in AAAI, 1996. arXiv:2307.08678, 2023.
[124] L. R. Tang and R. J. Mooney, “Automated construction of database [159] A. Shrikumar, P. Greenside, and A. Kundaje, “Learning important
interfaces: Intergrating statistical and relational learning for semantic features through propagating activation differences,” in ICML, 2017.
parsing,” in EMNLP, 2000. [160] J. Zhang, J. Xiang, Z. Yu, and et al., “Aflow: Automating agentic
[125] N. Yaghmazadeh, Y. Wang, I. Dillig, and T. Dillig, “Sqlizer: query workflow generation,” 2024.
synthesis from natural language,” Proc. ACM Program. Lang., 2017.
[126] S. Iyer, I. Konstas, A. Cheung, J. Krishnamurthy, and L. Zettlemoyer,
“Learning a neural semantic parser from user feedback,” in ACL, 2017.
[127] C. Finegan-Dollak, J. K. Kummerfeld, L. Zhang, and et al., “Improving
text-to-sql evaluation methodology,” in ACL, 2018.
[128] T. Yu, R. Zhang, M. Yasunaga, and et al., “Sparc: Cross-domain
semantic parsing in context,” in ACL, 2019.
[129] T. Yu, R. Zhang, H. Er, and et al., “Cosql: A conversational text-
to-sql challenge towards cross-domain natural language interfaces to
databases,” in EMNLP-IJCNLP, 2019.