SQLStorm: Taking Database Benchmarking into the LLM Era

Tobias Schmidt Viktor Leis

Technische Universität München Technische Universität München
[Link]@[Link] leis@[Link]

Peter Boncz Thomas Neumann

CWI Technische Universität München
boncz@[Link] neumann@[Link]
ABSTRACT Templates

Prior Benchmarks
SQL

3 30 Plans
44 K Optimize
TPC-H: 22 < Instantiate Ð
In this paper, we introduce a new methodology for constructing Queries Run Ù
206 Traces
database benchmarks using Large Language Models (LLMs), as well SQL

256 Plans
as SQLStorm v1.0, a concrete benchmark on a real-world dataset 51.5 K mize 3
Opti
TPC-DS: 99 < Instantiate Ð
Queries Run Ù
of three sizes (1 GB, 12 GB, 220 GB) consisting of over 18 K queries. 769 Traces
This methodology of using AI to generate query workloads breaks SQL

12.8 K
new ground, not only in its ability to cheaply ($15) generate huge Prompts 9 ize 3 Plans
35 K 18 K Optim
volumes (22 MB) of realistic queries but especially because it greatly + LLM Filter s

SQLStorm
Queries Queries Run Ù
Schema 
expands the amount of SQL functionality and query constructions 10.7 K
Traces
that is covered, compared to human-written SQL benchmarks such
7 Prompts (1) Rewrite queries with syntax errors using a LLM.
as TPC-H, TPC-DS, and JOB. The use cases of SQLStorm that we × (2) Remove queries not parsable by at least 2 databases.
think will advance data systems most are: (i) improving SQL compat- 5 K requests (3) Remove queries not executable by at least 1 database.

ibility between systems, (ii) increasing system quality by identifying

crashes/errors and fixing those, (iii) improving cardinality estima- Figure 1: Comparison of query generation pipelines in
tors and query optimizers, by identifying trends and opportunities template-based benchmarks (TPC-H/DS) and our LLM-based
(queries where other systems do much better), as well as (iv) overall SQLStorm methodology. The generated queries are optimized
system performance, both in terms of speed and robustness. and executed in Umbra and we report the number of distinct
query plans and execution traces. The LLM-based approach
PVLDB Reference Format: yields a more diverse query set than templated benchmarks.
Tobias Schmidt, Viktor Leis, Peter Boncz, and Thomas Neumann.
SQLStorm: Taking Database Benchmarking into the LLM Era. PVLDB,
18(11): 4144 - 4157, 2025.
doi:10.14778/3749646.3749683 capture the complexities of production workloads [49, 56, 59, 60].
However, given the privacy concerns surrounding real-world query
PVLDB Artifact Availability: logs and database contents, synthetic OLAP benchmarks have re-
The source code, data, and/or other artifacts have been made available at mained the primary – if not the only – viable alternative.
[Link] Generative AI, powered by Large Language Models (LLMs),
presents a novel opportunity to rethink SQL benchmarking. LLMs
1 INTRODUCTION from major AI vendors have been trained on vast open-source SQL
Benchmarks crystallize performance, providing an objective foun- corpora, implicitly capturing real-world SQL usage patterns. With
dation for comparing database systems. Beyond mere measurement appropriate prompting, these models can generate realistic SQL
tools, they shape the evolutionary trajectory of database technolo- queries that better reflect practical database workloads. Moreover,
gies by influencing which workloads vendors and researchers pri- the stochastic nature of LLMs enables random sampling from a
oritize. In the OLAP space, TPC-H and TPC-DS have emerged as distribution of real-world SQL queries, creating large, diverse, and
dominant benchmarks in industry and academia. representative query sets.
Both benchmarks rely on a limited set of handcrafted queries We propose a new benchmarking methodology called SQLStorm
and synthetic data generators. While this controlled design ensures that leverages LLMs for SQL generation. By providing a database
repeatability and comparability, it also comes with drawbacks. TPC schema and carefully designed prompts, LLM APIs can generate
benchmarks, for instance, have been criticized for their limited hundreds of thousands of complex queries that more accurately re-
coverage of key real-world functionalities, such as string process- flect real-world SQL workloads than traditional handcrafted bench-
ing, and their reliance on unrealistic data distributions that fail to marks. Figure 1 illustrates SQLStorm’s generation pipeline for
rewriting and filtering LLM-generated SQL queries. It removes er-
This work is licensed under the Creative Commons BY-NC-ND 4.0 International
License. Visit [Link] to view a copy of
roneous queries and ensures compatibility with multiple systems
this license. For any use beyond those covered by this license, obtain permission by by testing them in PostgreSQL, Umbra, and DuckDB. Our new
emailing info@[Link]. Copyright is held by the owner/author(s). Publication rights approach facilitates automated large-scale benchmark generation.
licensed to the VLDB Endowment.
Proceedings of the VLDB Endowment, Vol. 18, No. 11 ISSN 2150-8097. Conventional OLAP benchmarks, like TPC-H and TPC-DS, rely
doi:10.14778/3749646.3749683 on a small set of expert-curated query templates that, while useful
for comparability, are highly sanitized and often fail to capture analyze the SQLStorm v1.0 dataset and queries. Following that,
the diversity and unpredictability of real-world workloads. Even Section 4 explores practical use cases for SQLStorm. Finally, we
when instantiated thousands of times, template-based benchmarks conclude Section 5 and discuss future directions.
yield only a limited variety of query plans, and executions of the
same template are similar. In contrast, LLM-generated benchmarks 2 RELATED WORK
exhibit a much broader range of complexity, sometimes incorpo-
There is a large body of work on human-expert crafted database
rating unexpected constructs observed in open-source SQL cor-
benchmarks. On the industrial side, the most important have been
pora, resulting in thousands of distinct query plans and execution
the efforts of TPC [20]. Good performance on TPC-H/-DS and
traces. Furthermore, the stochastic behavior of LLMs generates a
TPC-C are now primary targets to show that a new analytical re-
long-tailed distribution of queries, with some containing up to 38
spective transactional database system masters the basic techniques
operators – a characteristic common in production databases but
required to enter the (mature) database market. While these well-
rarely seen in traditional benchmarks.
understood [10, 16, 53] benchmarks exercise execution engines
Using our methodology, we also contribute SQLStorm v1.0, a
reasonably well, they fail to address crucial optimizer problems
concrete benchmark designed to be both comprehensive and chal-
like join ordering due to their small schemas, limited queries, and
lenging for modern database systems, built on the StackOverflow
uniform data distributions. JCC-H [9] and DSB [15] extend exist-
dataset, available in three sizes (1 GB, 12 GB, and 220 GB). For this
ing benchmarks by incorporating complex correlations and skew
first version, we employ OpenAI’s GPT-4o-mini model to generate
to stress systems. From the academic side, JOB [30] and its vari-
a diverse set of SQL queries tailored to the StackOverflow schema.
ants [25, 63] introduced complex join query patterns on the real-life
This constitutes a rich, realistic dataset alongside an extensive query
IMDB dataset with skewed and correlated data distributions, help-
set that spans a wide array of SQL constructs. The sheer volume
ing spur recent efforts to improve query optimization. However,
of queries (over 22 MB) not only ensure broad SQL pattern cover-
the decision to keep the query sets in these benchmarks small and
age but also makes it nearly impossible to optimize for individual
hence understandable for humans naturally limits the coverage of
queries. TPC-H, in particular, allows for per-query chokepoint anal-
complex structures.
ysis and hand-optimized implementations, as it contains only 22
Previous benchmarks with a large set of realistic queries, most
query templates [10, 14, 16].
prominently, did so by distilling real workloads – consisting of data
As the name suggests, SQLStorm is designed to stress-test data-
schemas and query logs – into synthetic equivalents [13, 26, 41, 57].
base engines across multiple dimensions, including functionality,
These methodologies, while valuable, only work for database teams
error handling, optimization, and execution. The benchmark sup-
that already have access to such customer-donated workloads. Also,
ports several key use cases:
real-life workloads [56, 60] are typically dominated by many simpler
Optimizer Stress Testing: The extensive query set exposes queries. Hence, a sample from those may not adequately cover the
weaknesses in query optimizers by incorporating a broad space of complex, large, or even outlandish queries that mature
mix of simple, complex, and edge-case queries. database systems are expected to handle robustly.
Cardinality Estimation: SQLStorm challenges cardinality es- SQLStorm uses the Stack Overflow datasets – a decision also
timation techniques on real-world data with complex queries followed by STATS-CEB [18] and Stack [36], whose query set con-
and a large set of relational operators. sists of 100 human-generated queries. These queries are relatively
Performance Regression Testing: SQLStorm’s extensive and simple and cover only a small fraction of database functionality
stable query set enables developers to track performance exercised by SQLStorm. The only existing database benchmark,
across versions, ensuring reliability as systems evolve. to our knowledge, that exploits LLMs to generate queries, is the
Scalability Analysis: With datasets spanning three orders of Surprise Benchmark [7] – however, it generates batches of only 5
magnitude in size, the benchmark enables systematic evalu- queries incrementally. A final relevant field is database testing and
ation of database scalability. database fuzzing [44]. We tried using SQLSmith [45], and while it
Comparative Benchmarking: SQLStorm facilitates head-to- can generate many queries quickly, these lack credible semantics
head comparisons of database systems. and, consequently, relevant operator and expression patterns.
Query Optimization Mining: Large relative performance dif- Large language models have been investigated for a variety
ferences between systems often highlight missing query of database-related tasks [31, 70]. Given their capabilities as cod-
optimization rules in the slower system. ing assistants, they are naturally suited for text-to-SQL transla-
Bring your own data: SQLStorm can be applied to any dataset, tion [19, 32, 54, 68]. However, researchers are also exploring deeper
allowing users to generate custom benchmarks. integration of LLMs into data systems, including applications such
as database tuning [12, 27, 33, 55], query optimization [5, 50, 64],
By leveraging LLMs for benchmark generation, SQLStorm repre-
query rewriting [34, 35, 48], data cleaning [40, 58], and diagnostic
sents a paradigm shift in SQL benchmarking – moving beyond static,
tools [69].
expert-designed queries toward a scalable, adaptable methodology
that better reflects real-world workloads.
The rest of this paper is organized as follows: We begin by re- 3 SQLSTORM
viewing related work in Section 2. Next, Section 3 introduces the Existing benchmarks like TPC-H and TPC-DS are widely used
SQLStorm methodology and details how LLMs can be used to ef- to evaluate the performance of database systems. However, these
ficiently generate large-scale benchmarks. We then present and benchmarks have several limitations: (a) the query sets are relatively
small and contain at most a hundred query templates, (b) the queries Table 1: The number of rows and (uncompressed) binary
only use a fraction of the SQL standard and do not include more size of the Stack Overflow dataset. The last part of the table
complex constructs like recursion, arrays, or JSON, and (c) the reports how many columns with the data type exists and
benchmarks use synthetic data that fail to represent real-world the fraction of space they occupy on SQLStorm-220. Text
scenarios. Our novel SQLStorm methodology uses a real-world columns consume more than 200 GB.
dataset and large-scale query sets to address these limitations.
This section describes how SQLStorm generates database bench- Table #Rows Size Integer Date Text
marks using LLMs. The process can be divided into four steps: Users 26.1 M 2.2 GB 6 (30.3 %) 2 (17.3 %) 5 (52.3 %)
Badges 52.9 M 1.5 GB 4 (37.1 %) 1 (27.0 %) 1 (36.0 %)
Posts 60.2 M 71.6 GB 11 (3.3 %) 5 (3.1 %) 6 (93.6 %)
(1) Query Generation: We use OpenAI’s GPT-4o-mini model Comments 91.0 M 16.5 GB 4 (8.2 %) 1 (4.1 %) 3 (87.7 %)
to generate SQL queries based on user-defined prompts. PostHistory 161.9 M 122.7 GB 4 (1.7 %) 1 (1.0 %) 5 (97.3 %)
(2) Query Rewriting: We rewrite the generated queries to in- PostLinks 6.5 M 0.2 GB 4 (69.2 %) 1 (30.8 %) 0 (0.0 %)
crease compatibility and remove duplicates. Tags 66.0 K 1.9 MB 6 (61.0 %) 0 (0.0 %) 1 (39.0 %)
Votes 242.4 M 5.9 GB 5 (69.2 %) 1 (30.8 %) 0 (0.0 %)
(3) Query Selection: SQLStorm relies on PostgreSQL, Umbra,
and DuckDB to identify parsable and executable queries. Total 641.1 M 220.5 GB 44 (5.1 %) 12 (3.1 %) 21 (91.8 %)
(4) Complexity Classification: We classify the generated SQL
queries based on their complexity and the features they use.
enum values. The database schema contains 19 foreign key rela-
tionships, including self-references, allowing complex join graphs.
3.1 The StackOverflow Dataset Every table has a 32-bit integer as primary key.
The Stack Exchange network is a valuable knowledge base for tech- We chose the three datasets as their size is in different orders of
nical topics with millions of questions, answers, and comments. magnitude. SQLStorm-12 is roughly 10× larger than SQLStorm-1,
The StackOverflow website, in particular, is a popular platform for while SQLStorm-220 is almost 20 times larger than the Math dataset.
asking and answering programming-related questions [4]. Stack Note that the datasets do not scale perfectly linearly; for example,
Exchange makes data dumps of their network available under the the number of users grows only by 5× between the smallest scale
Creative Commons BY-SA 4.0 license, allowing us to use the data factor and SQLStorm-12, while the number of posts and comments
for research purposes. Compared to other real-world datasets, like is 20× larger. This characteristic makes cardinality estimation and
IMDB [30], the Stack Exchange data dumps contain more than query optimization more interesting as the data’s scale and distri-
100 GB of data, and different sizes are available for download. This bution change. Alternatively, filtering records by creation date can
allows for a scalable and challenging benchmark for modern ana- reduce the 220 GB StackOverflow dataset to smaller scale factors.
lytical database systems. Over the last 17 years, the dataset has grown by 5 GB to 20 GB per
We downloaded three datasets in different sizes from the Stack- year, scaling smoothly from a few gigabytes to two hundred while
Overflow website [23], the Mathematics Stack Exchange (Math) [22], retaining real-world characteristics.
and the Database Architects Stack Exchange (DBA) [21]. The data Next, we investigate which data types are used and their storage
dumps contain all the questions, comments, votes, and users up to size. Table 1 reports the number of integer, date, and text columns,
the 3rd of October 2024. The extracted CSV from the StackOver- including their size on SQLStorm-220. 44 of the 77 columns are
flow website is the largest, with 240 GB; the other two datasets are numerical, mostly 32-bit and 16-bit integers for encoding primary
smaller and contain only 13.8 GB and 1.26 GB, respectively. Instruc- and foreign key relationships or counters. However, these columns
tions for downloading a newer version of the dataset can be found account only for 5.1 % of the total data size. Most of the data is
here [3]; we provide scripts to convert the dumps to CSV files. In stored in text columns, which make up for almost 92 % of the con-
addition, the database schema and documentation are also avail- sumed space, even though only 27 % of columns use a character
able online at [1]. We make the extracted datasets and the schemas data type. The longest text contains 116,293 characters, while the
available for download along with the SQLStorm benchmark. average string length is 338 characters. This distribution matches
The dataset consists of 13 tables, 5 of which encode enum types the observations from real-world data where variable-sized strings
and are the same size for all datasets. The remaining 8 relations are dominant [39, 49, 56, 59]. Furthermore, the strings also contain
contain the actual data and scale with the dataset size. Table 1 lists Unicode characters; synthetic benchmarks like TPC-H and TPC-DS
their size and the number of records for SQLStorm-220 (StackOver- lack these characteristics.
flow) the largest of the three datasets with 220 GB. The two smaller
datasets SQLStorm-1 (DBA) and SQLStorm-12 (Math) consist of 3.2 Query Generation
3.1 M rows with 1.2 GB and 39.6 M rows with 12.6 GB, respectively. Large language models are trained on a vast amount of data to pro-
The Users and Badges tables model the users and the badges they duce human-like output and solve complex tasks. This includes pub-
have earned. The Posts and Comments tables contain the website’s lic source code or questions from programming-related platforms
actual content, while the PostHistory table stores the edit history. like StackOverflow [8, 67]. The 2024 StackOverflow Developer Sur-
The PostLinks, Tags, and Votes relations store further metadata for vey reports that SQL is the second most popular programming
the posts. In addition, the five tables not listed — PostTypes, PostHis- language among professional developers [2]. As a result, LLMs
toryTypes, CloseReasonTypes, VoteTypes, and LinkTypes — model have seen many programming problems and solutions, including
Table 2: The prompts used to generate the query set on the StackOverflow dataset. Besides the prompt, we also include the
database schema in the request. The table also shows the number of queries that pass each step of the selection process; queries
from simple prompts are cheaper and more likely to be selected.

Prompt (GPT-4o-mini) Parse 1 Parse 2 Exec. Yield Cost

P1 Generate an interesting and elaborate SQL query for performance benchmarking, potentially including 1927 2690 2533 50.7 % $2.38
constructs such as outer joins, (correlated) subqueries, CTEs, window functions, set operators,
complicated predicates/expressions/calculations, string expressions, and NULL logic.
P2 Generate an interesting and elaborate SQL query for performance benchmarking. 2472 3461 3196 63.9 % $2.19
P3 Generate a SQL query for performance benchmarking. 3188 4205 3791 75.8 % $1.83
P4 Generate a simple SQL query for benchmarking. 3343 3694 3690 73.8 % $1.54
P5 Generate an interesting and elaborate SQL query for performance benchmarking, potentially including 1430 2060 1848 37.0 % $2.72
constructs such as outer joins, (correlated) subqueries, CTEs, window functions, set operators,
complicated predicates/expressions/calculations, string expressions, and NULL logic. Incorporate
obscure semantical corner cases and unusual or even bizarre SQL semantics.
P6 Generate an interesting and elaborate SQL query for benchmarking string processing. 1895 2676 1987 39.7 % $2.41
P7 Generate an interesting and elaborate SQL query for performance benchmarking, potentially including 951 1432 1206 24.1 % $2.66
constructs such as outer joins, (correlated) subqueries, (recursive) CTEs, window functions, set
operators, complicated predicates/expressions/calculations, string expressions, and NULL logic.
Total 15206 20218 18251 52.1 % $15.73

SQL. Therefore, asking an LLM for random queries will sample the While P1 generates a diverse set of queries comparable to TPC-
real-world distribution of queries it was trained on and produce a DS in terms of complexity, it has two significant drawbacks: (1) The
realistic workload. queries are more complex than the majority observed in real-world
OpenAI and Anthropic provide batch APIs, allowing to run the workloads [56]. Hence, simple statements using basic SQL features
models in an asynchronous fashion and reduce cost by 50%. This al- and accessing only one or two tables should also be part of the
lows for fast, scalable, and cost-effective generation of large sets of benchmarks. (2) The prompt does not cover advanced SQL con-
SQL queries. Not all queries will be syntactically correct or semanti- structs like deeply correlated subqueries or recursive CTEs. While
cally meaningful, but the same is true for queries written by humans. these constructions are not prevalent in actual workloads, they are
Our selection process will find useful queries for benchmarking still important for evaluating the system’s robustness and testing
and remove the rest. The queries’ semantics are of secondary im- edge cases that database engineers might not have considered.
portance in an analytical benchmarking context, provided they We modified the prompt step-by-step to generate simpler and
process a sufficient volume of data and incorporate various SQL more complex queries. Table 2 lists the final seven prompts we
features, such as complex joins, window functions, recursion, and used for creating SQLStorm. P2 - P4 gradually remove hints from
other advanced operations. the prompt to make the statements simpler. P5 and P7 instruct
Since LLMs are a black box, finding the right prompt is more of the LLM to generate unusual SQL constructs to test corner cases
an art than a science. We tested several prompts and found that the and recursion. As strings are prominent in real-world datasets
following prompt provides good results and diverse SQL queries: and workloads, P6 explicitly asks for queries focused on string
P1 Generate an interesting and elaborate SQL query for performance
operations. We call OpenAI’s GPT-4o-mini model for each prompt
benchmarking, potentially including constructs such as outer with a batch of 5,000 requests. A batch is processed within one day
joins, (correlated) subqueries, CTEs, window functions, set (typically much faster) and costs $2 on average. The cost between
operators, complicated predicates/expressions/calculations,
string expressions, and NULL logic. batches differs, as simple prompts produce shorter queries and
require fewer input and output tokens than complex prompts.
From our experience, adding hints to the prompt can help the
Next, let us discuss two example queries generated by the LLM
model explore a broader range of possibilities. However, sometimes
using the prompts P1 and P4. The first query in Figure 2 finds all
hints may limit the search space, causing the LLM to over-commit
users with posts from the last year and outputs statistics about
on the explicitly enumerated features. It requires careful consid-
their badges and closed posts. The second query retrieves the top
eration of what information is given to the model. However, we
10 highest-scoring posts, along with the author and comments. We
must provide the LLM with enough details to generate correct and
can observe a significant difference in the complexity: while the first
meaningful queries. To achieve this, we also included the database
query uses window functions, filtered aggregations, and multiple
schema as CREATE TABLE statements as a suffix to the main prompt:
CTEs, the second only uses simple SQL constructs.
PS Do not explain the query, only output one SQL query. Use
the following StackOverflow schema: CREATE TABLE ...

The schema includes foreign key relationships and primary keys, 3.3 Query Cleaning & Rewriting
allowing the LLM to generate queries that join multiple tables and Using our generation process, we obtained 35,000 queries on the
find the correct join conditions. Additionally, we added some com- StackOverflow dataset. However, these queries include duplicates,
ments to the schema, clarifying the structure of the database and references to the current time, or comments where the LLM tries
the five enum tables. to explain the query. Sometimes, the output also contains multiple
 1 WITH RankedPosts AS ( Table 3: The number of queries that are parseable and exe-
2 SELECT [Link] AS PostId, [Link], [Link], [Link], cutable by PostgreSQL , Umbra , and DuckDB . Almost
3 [Link], [Link], [Link],
4 ROW_NUMBER() OVER (PARTITION BY [Link] half of the initial queries (Parse 1) are compatible with two
5 ORDER BY [Link] DESC) AS rn systems. After rewriting the queries (Parse 2), even more
6 FROM Posts p JOIN Users u ON [Link] = [Link]
7 WHERE [Link] >=
queries can be parsed by . The queries that pass the
8 (CAST('2024-10-01' AS DATE) - INTERVAL '1 year')), SQLStorm selection process are highlighted in green.
9 UserBadges AS (
10 SELECT [Link],
11 COUNT(*) FILTER (WHERE [Link] = 1) AS GoldBadges, ∅
12 COUNT(*) FILTER (WHERE [Link] = 2) AS SilverBadges,
13 COUNT(*) FILTER (WHERE [Link] = 3) AS BronzeBadges Parse 1 9427 5644 32 103 166 381 1875 16085
14 FROM Badges b GROUP BY [Link]), Parse 2 17016 2157 39 137 144 379 1138 12703
15 ClosedPostCount AS ( Exec. 10910 1164 24 2747 63 3312 31 1967
16 SELECT [Link], COUNT(*) AS ClosedPosts
17 FROM PostHistory ph
18 WHERE [Link] = 10 GROUP BY [Link])
19 SELECT [Link], [Link], [Link], [Link], are parsable, while for complex prompts like P5 and P7, only less
20 [Link], [Link],
21 COALESCE([Link], 0) AS GoldBadges,
than 30 % qualify. On average, 43 % of the queries can be parsed.
22 COALESCE([Link], 0) AS SilverBadges, We noticed that GPT-4o-mini primarily generates queries in the
23 COALESCE([Link], 0) AS BronzeBadges, PostgreSQL dialect. This may be explained by the fact that Post-
24 COALESCE([Link], 0) AS ClosedPosts,
25 CASE WHEN [Link] > 100 THEN 'High Score' greSQL is widely used in open-source projects and its dialect follows
26 WHEN [Link] BETWEEN 50 AND 100 THEN 'Medium Score' the SQL standard more closely than the dialects of some commer-
27 ELSE 'Low Score' END AS ScoreCategory
28 FROM RankedPosts rp LEFT JOIN UserBadges ub
cial systems. Table 3 indicates that all three systems together can
29 ON [Link] = [Link] parse 9,427 queries. However, several queries are incompatible with
30 LEFT JOIN ClosedPostCount cpc DuckDB: 5644 queries only run in PostgreSQL and Umbra. The
31 ON [Link] = [Link]
32 WHERE [Link] = 1 two systems implement a relatively obscure feature from the SQL
33 ORDER BY [Link] DESC, [Link] DESC; standard that allows them to omit columns from the group by
clause if they are functionally dependent on another column in the
Figure 2: Complex query generated by prompt P1. As in- clause, which is used by the LLM. DuckDB, in contrast, can parse
structed, the prompt constructs outer joins, window func- 1875 queries that neither PostgreSQL nor Umbra can execute. For
tions, and complicated expressions, like filtered aggregates. instance, DuckDB supports SUM aggregations on boolean values.
Ideally, we want to have queries that run in all three systems and
1 SELECT [Link] AS UserName, [Link] AS PostTitle, avoid dialect-specific constructs. In order to make queries compati-
2 [Link] AS PostScore, [Link] AS CommentText, ble between systems, we use the LLM again to rewrite the query
3 [Link] AS CommentDate
4 FROM Posts P JOIN Users U ON [Link] = [Link]
text with the following prompt:
5 LEFT JOIN Comments C ON [Link] = [Link] PF Make the following PostgreSQL query more compatible with
6 WHERE [Link] = 1 different SQL dialects. The query might contain ’::’ casts,
7 ORDER BY [Link] DESC LIMIT 10; rewrite them to standard SQL. Remember to put all ungrouped
columns and columns that appear in window functions into
the group by clause. Do not explain the query, only output
Figure 3: Simple query generated by prompt P4. Without the converted query.
additional instructions, GPT-4o-mini generates queries with
few joins and no complex SQL constructs. The prompt instructs the LLM to make the query compatible with
standard SQL. We specifically address PostgreSQL’s shorthand casts
and missing attributes in GROUP BY clauses, as these are common
queries or DML statements instead of the expected SELECT state- patterns. After the prompt, the query text is appended.
ments. In order to make the queries suitable for performance bench- In the second pass, GPT-4o-mini rewrites 24,692 queries; these
marking, our next step is to clean and rewrite the LLM’s output. queries could not be parsed by all three systems or contain a short-
We start by deleting duplicated queries; on the StackOverflow hand cast. It fixes several incompatibilities: adding missing columns
dataset, we found 1287 duplicates, all generated by the prompt into the group by clause, rewriting short hand casts to function-
P4. Next, we remove comments in the query text and extract the style casts, and converting boolean expressions in SUM aggregations
first SELECT statement if multiple are present. Furthermore, we re- to integer values using a case expression. In addition, the LLM
place current_time and similar functions with a fixed timestamp was also able to fix several problems we did not mention in the
to ensure the queries are reproducible. For example, the query in prompt. As a result, after the second pass, 20,218 queries qualify, as
Figure 2 originally used current_date in line 7; we replaced it with shown in Table 2 (Parse 2), an increase of more than 5,000 queries.
a fixed date, 2024-10-01 to prevent empty scans in the future. More importantly, the number of queries that all three systems can
After rewriting the queries, we test how many can be parsed by parse almost doubles to 17,016 queries (cf. Table 3), and queries
PostgreSQL, Umbra, and DuckDB. We consider a query parsable not compatible with any system are reduced by 20 %. However, 869
if at least two of the three systems can execute the query on an queries became incompatible after the LLM attempted to fix them;
empty database. Table 2 reports the number of parsable queries for we revert these queries to the original version.
each prompt under Parse 1. Unsurprisingly, the simple prompts P3 Cleaning and rewriting makes the queries suitable for a bench-
and P4 yield more correct queries. More than 60 % of the queries mark. Rewriting queries with the same LLM significantly improves
the compatibility with different systems and fixes some errors in Table 4: Number of queries with identical results in at least
the queries. The cost of this step is similar to generating the queries: two systems. In some cases, one system differs from the other
the first LLM pass costs $10, and the second pass $6. two; for example, DuckDB returns different results than Post-
greSQL and Umbra for 1,932 queries ( - ). Additionally,
5,664 queries yield inconsistent outputs across all systems.
3.4 Query Selection
After rewriting, we select the final set of queries that will be used match in ≥ 2 systems one system differs from the other
for the benchmark. First, we remove all queries that cannot be - - - ∅
parsed by at least two systems, then test if each query runs on the 9519 86 685 1932 257 108 5664
SQLStorm-1 dataset. The dataset contains roughly 1 GB of data,
and a query is executable if one of the three systems finishes it in
under 1 second. We require two systems to be able to parse the the sampled queries revealed the following distribution: 53% of the
query, as we want to avoid system-specific language constructs. For queries are correct, 34% almost correct, and 13% incorrect.
execution, one system is sufficient: if a query runs in one system but Manually classifying the queries is tedious; therefore, we use a
times out in another, it indicates that the system does not optimize heuristic to check the queries automatically. We inspect the query
the query well and is interesting for performance evaluation. The plan generated by Umbra and test if all joins use one of the 19 for-
timeout eliminates queries that are too expensive to execute, such eign key relationships or reasonable attributes such as user names.
as queries with huge intermediate results due to cross-products. 69 % of the queries join only along foreign keys, and 11 % con-
Table 2 lists the results of this step: 18,251 of the 20,218 queries tain reasonable joins. The remaining 20 % of the queries contain
from the parsing step qualify and remain in the benchmark. Yield incorrect or missing join conditions. However, this automatic clas-
rates in the different batches differ significantly, while from our sification reports some false positives as queries are semantically
first prompt, P1, 50 % of the queries remain, only 24 % of the queries meaningful, even though they use an incorrect join attribute or per-
from P7 are parsable and executable. P3 and P4 have the highest form a cross-join. For the manually reviewed sample, eight queries
yield rates; more than 70 % of the queries qualify. Recall that P4 were falsely classified as incorrect by the heuristic.
generated 1287 duplicated queries, which we eliminated during the Combining the two analyses, we conclude that the majority of
cleaning step. If we do not consider these duplicates, the prompt the queries are correct or almost correct; however, a notable portion
yields more than 99 % of the queries. contains minor or, occasionally, significant errors. Nevertheless,
Of the 17,016 queries that were parsable by all three systems, we choose to retain these queries, as errors occur in real-world
10,910 queries run in all three systems, see Table 3. PostgreSQL’s scenarios as well: Users interactively write and refine SQL queries
query engine is slower than Umbra’s and DuckDB’s, and more step-by-step, leading to underspecified queries, e.g., missing join
queries time out. Consequently, 2,747 queries can only be executed or filter conditions [59]. Moreover, AI assistants are becoming in-
by Umbra and DuckDB, and 3,312 queries only by Umbra. creasingly common in the industry, helping users generate SQL
Admittedly, developing this benchmark had a major impact on snippets, optimize existing queries, and automatically correct er-
Umbra’s compatibility, prompting us to implement missing features rors [6, 46, 51]. Recent research indicates that while human experts
and resolve several bugs. Section 4.3 analyzes our changes to Umbra outperform LLMs in SQL query correctness, they still make mis-
caused by SQLStorm in more detail. To summarize, the SQLStorm takes on more than 7 % of tasks on the BIRD dataset [29, 32]. This
methodology generates a large-scale analytical benchmark at a error rate rises for less experienced users [56], suggesting that a
negligible cost. We spend less than $16 and obtain over 18,000 query set containing mistakes more accurately reflects reality than
queries compatible with PostgreSQL, Umbra, or DuckDB. a perfectly error-free set. Machine-generated SQL and non-expert
written queries are common in practice and lead to erroneous or
underspecified queries [56, 59].
3.5 Query Validation & Correctness A manual review of the queries shows that they cover various
In this chapter, we evaluate the correctness of the generated queries realistic analytical tasks. Common patterns include (a) ranking
and assess whether they are suitable for result validation. We fo- users, posts, or comments based on various scoring metrics, often
cus on two main aspects: whether the generated queries address requiring complex computations and multi-table joins; (b) retriev-
real-world scenarios and whether they are semantically correct. In ing recent records from different tables; (c) enriching tables with
addition, we offer a validation set to verify the correctness of the additional statistics or detailed information; and (d) aggregating
results, similar to benchmarks like TPC-H and -DS. data across multiple tables to produce summaries or reports. In
To evaluate the correctness, we randomly sampled 100 queries our analysis, we found that many queries answer similar questions.
from the final set and manually categorized them into three groups: However, they differ in part greatly in their implementation, the
(1) correct queries accurately join and filter the data, producing se- used attributes and tables, and the SQL constructs.
mantically meaningful and logically sound results, (2) almost correct Lastly, we assess the suitability of the generated queries for
queries are accurate in terms of join and filter conditions but contain result validation. We computed the result sets for all queries on
minor errors, e.g., using a count where a distinct count would be the three systems: 12,587 produce identical results in at least two
semantically more meaningful, and (3) incorrect queries are seman- systems, while the remaining 5,664 exhibit inconsistent or non-
tically flawed due to incorrect or missing join conditions, empty deterministic behavior. Table 4 shows the number of queries with
filter criteria, or erroneous computations. A manual inspection of matching outputs for different combinations of the three systems.
We observe that DuckDB differs from PostgreSQL and Umbra more Table 6: Average number of operators, expressions, and query
often because Umbra closely follows PostgreSQL’s semantics [52]. text size. Basic relational operators are present in all bench-
This highlights subtle differences in SQL interpretation and imple- marks, whereas advanced features like recursion (Iteration),
mentation across systems; such queries are particularly valuable to arrays, regexes, and JSON are unique to SQLStorm.
database engineers as they expose bugs and incompatibilities.
SQLStorm generates semantically meaningful queries that are SQLStorm TPC Redshift [56]
well-suited for result validation. Yet, it includes erroneous and low med. high -H -DS [60s,∞) Total
underspecified queries, reflecting the reality of machine-generated Query Text 334 1168 1347 472 1370 n.a. 4396
and non-expert written SQL commonly found in practice [56, 59]. Operators 6.42 14.41 22.08 9.09 19.27 10.74 3.50
TableScan 2.87 4.94 5.75 3.68 7.22 6.51 2.03
3.6 Complexity Classification Join 1.87 4.17 6.20 2.82 6.30 2.28 0.53
Sort 0.99 1.03 1.10 0.82 0.83 0.34 0.12
SQLStorm has diverse queries, including simple ones with only a GroupBy 0.68 2.33 4.01 1.32 2.08 0.72 0.52
few joins and complex queries with advanced SQL constructs. For Select 0.004 0.97 1.78 0.09 1.17 n.a. n.a.
developing new database systems or research prototypes, support- Window 0 0.64 1.01 0 0.20 0.2 0.04
ing all queries upfront will be challenging. We therefore classify SetOperation 0 0.004 0.003 0 0.28 n.a. n.a.
the queries into three complexity classes: low, medium, and high. ArrayUnnest 0 0 0.49 0 0 n.a. n.a.
Iteration 0 0 0.05 0 0 n.a. n.a.
Low complexity queries can be supported by implementing basic RegexSplit 0 0 0.001 0 0 n.a. n.a.
operations and require only a small subset of the SQL standard. Re-
Expressions 6.42 29.12 35.61 12.86 37.51 n.a. n.a.
searchers can use them to evaluate prototype systems with limited string modif. 0 0.13 0.77 0.23 0.18 n.a. n.a.
functionality. The medium complexity queries require additional string match. 0 0.07 0.18 0.32 0.01 n.a. n.a.
operators such as window functions, set operations, and outer joins. regex 0 0 0.004 0 0 n.a. n.a.
The high complexity class include the remaining queries, demand- json 0 0 0.003 0 0 n.a. n.a.
ing advanced features such as recursion, arrays, json, and complex
query unnesting (e.g., mark joins [43]).
We use the matrix in Table 5 to determine the complexity of a query and SQL text length in the different complexity classes. As
query based on the execution traces collected by Umbra. For in- query complexity increases, both the average query text length and
stance, medium complexity queries only use operators from the low the number of operators and expressions grow accordingly. Low
complexity class in addition to Window and SetOperation. Simi- complexity queries are roughly 3 times smaller regarding code and
larly, the expressions must not use complex regex or json functions. plan size. The table also includes statistics for the TPC benchmarks:
We also limit the number of joins and aggregations, as deeper query TPC-H is, on average, more complex than the low class and simpler
trees require sophisticated join ordering and cardinality estimation than the medium class. TPC-DS is similar to the high complexity
for efficient execution. If a query does not fulfill one condition, it class, except that it uses more expressions per query.
falls into a higher class. The queries from the low complexity class use fewer operators
We developed the assignment based on how many queries qual- as we limit the number of joins to 3. Sorts often appear as the last
ify in each category and our experience developing the required operator in the query tree to order the output. However, in more
features. The medium complexity class is the largest with 10,195 complex queries they are also used to select the top-k results on
queries, followed by 4,596 simple queries and 3,460 high complex- intermediate computations. The Iteration operator implements re-
ity queries. Table 6 shows the average number of operators per cursive CTEs in Umbra; 157 queries from P7 in SQLStorm use this
feature. ArrayUnnest and RegexSplit iterate through arrays/regex
matches; besides set operations, these operators are the least com-
Table 5: Matrix to determine the complexity of a query. mon. SQLStorm also features regex and JSON expressions.
van Renen et al. also report numbers on the query complexity in
Complexity low medium high their analysis of the Amazon Redshift fleet [56]. We extracted their
TableScan, Join, Iteration, results from the paper to compare with SQLStorm. The queries
Window, from the entire Redshift fleet are, on average, less complex than
Operators GroupBy, Select, RegexSplit,
SetOperation
Map ArrayUnnest those in the low complexity class. However, Redshift is closer to
comparison, cast, nulls, strings, date, the medium and high class if we only consider the long-running
Expression queries. The average number of table scans in long-running queries
case, arithmetic array, arithmetic regex, json
Categories exceeds the high complexity class.
(simple) (complex)
bool, int, text,
record, json, 3.7 Query Diversity
Types numeric, float, arrays
timestamptz
date, timestamp SQLStorm contains a large number of queries, raising the important
Join Types inner, outer semi, anti, single mark question: Is the query set genuinely diverse or minor variations
#Joins <= 3 <= 8 of a few templates, like TPC-H and -DS? To assess the workload’s
diversity, we compute the number of unique query plans and ex-
#GroupBy <= 1 <= 3
ecution traces produced in Umbra. A query plan is unique if its
Table 7: Number of unique query plans and execution traces StackOverflow
Redset

Density
recorded in Umbra on SQLStorm and TPC-H/DS. Snowset
TPC-H (220GB)
TPC-DS (220GB)
Unique / Total SQLStorm TPC-H TPC-DS
Plans 12,384 / 18,251 30 / 44,000 256 / 51,500 1 ms 10 ms 100 ms 1s 10 s 1 min 10 min
→ Trees 9,869 / 18,251 24 / 44,000 154 / 51,500 Runtime (log-scaled)
→ Operators 26,667 / 267,401 153 / 416,000 1,028 / 1,317,151
Traces 10,692 / 18,251 206 / 44,000 769 / 51,500
Figure 4: Runtime distribution of SQLStorm, compared to
real-world workloads (Redset & Snowset) and artificial bench-
marks (TPC-H & -DS).
tree structure differs or contains a new instance of an operator. The
tree’s shape is defined by the exact layout of the operators and their
operator type (e.g., join, map, groupby, etc.). We consider additional Many companies offer closed and open-source models: OpenAI
details for operator instances, such as the number of attributes used, (GPT), Anthropic (Claude), Meta, and Google (Gemini), to name
the specific expressions evaluated, and the join or aggregation type. only a few. The models differ in their training data, number of
Umbra’s optimizer produces 18,233 plans with 267,401 operators on parameters, and capabilities and are optimized for different tasks
SQLStorm. Among these are 9,869 distinct query trees and 26,667 and price points. When we generated the queries (October 2024
unique operator instances, resulting in 12,384 unique queries. Com- through February 2025), OpenAI’s GPT-4o-mini model was the
pared to TPC-H and TPC-DS, SQLStorm offers greater query plan most cost-effective LLM in terms of query yield. Table 8 compares
diversity. The template-based benchmark corpora cannot match LLMs for generating SQL benchmark queries.
the textual and structural diversity of LLM-generated queries. The cheaper models, GPT-4o-mini, Claude 3.0 Haiku, and Claude
For execution traces, the trend is similar: Umbra records 10,692 3.5 Haiku, produce query sets with similar complexity, i.e., the av-
distinct traces, while TPC-H and TPC-DS have 10× fewer despite erage number of operators is almost identical. The latest Haiku
having almost 3× more queries (44 K and 51.5 K, respectively). The model achieves the highest query yield but is three times more ex-
traces capture dimensions such as runtime, memory usage, number pensive than GPT-4o-mini. We also include the older GPT-3.5-turbo
of scanned rows, allocated memory, and operator counts. model in the comparison. While many of the model’s generated
Snowflake and Redshift published execution traces for their cloud queries qualify, the queries are also significantly shorter and less
data warehouses [56, 60]. Figure 4 compares the runtime distri- complex; the results are comparable to P3 or P4 on GPT-4o-mini.
bution of SQLStorm to these traces. Unlike artificial benchmarks, Even though Anthropic’s models are priced higher than OpenAI’s
which lack a broad distribution and exhibit more pronounced spikes, GPT-4o-mini, the overall costs are comparable, as Claude supports
SQLStorm’s runtime distribution more closely resembles real-world prompt caching, reducing the price by roughly two-thirds.
patterns. Snowflake and Redshift report queries running for over On average, the GPT-4o queries use two more operators than the
10 hours; for practicality, we limited execution time to 10 minutes other models. Claude 3.5 Sonnet generates the highest number of
in our measurements. Nevertheless, some SQLStorm queries hit operators yet. However, as the two models generate more complex
this limit, so the workload also includes long-running queries. queries, they are also more expensive. GPT-4o and Claude 3.5 Son-
The SQLStorm methodology generates queries with diverse net cost 10 times more than GPT-4o-mini. Therefore, we decided to
query plans and execution traces. We observe complex queries use GPT-4o-mini since it is the most cost-effective model.
with up to 61 operators and 72 expressions for the StackOverflow To our surprise, Anthropic’s Claude 3.5 Sonnet model performs
dataset. The queries cover a wider range of features than standard worst of all LLMs regarding the number of qualifying queries. Only
benchmarks like TPC-H or TPC-DS and stress query engines in 15.4 % of the 1,000 generated queries pass the selection process and
multiple dimensions, resulting in more than 10,000 unique query can be executed by one of three database systems. We found that
plans and execution traces. The runtimes range from milliseconds the queries generated by the model often contain the same error:
to several minutes, similar to Redshift and Snowflake. Sonnet combines aggregations and window functions but “forgets”
to add the columns used by the window to the GROUP BY clause.
3.8 Exploring Other Large Language Models
The state of the art for LLMs is changing rapidly: new models are Table 8: Comparison of OpenAI’s GPT and Anthropic’s
released monthly, and funding has increased eightfold between 2022 Claude models for generating benchmark queries using P1
and 2023, reaching $2.52 billion [37]. Training data sets, the number (cf. Table 2). While GPT-4o-mini is the most cost-effective,
of parameters, the context window, and underlying hardware all newer and larger models generate more complex queries.
grow rapidly. We anticipate significant advancements over the next
decade, and the models’ capabilities will continue to improve. The Model Yield Cost Avg length Avg ops.
concrete benchmark we present in this paper is just the first version, GPT-3.5-turbo (2024-01-25) 66.7 % $3.44 620 B 8.7
and we plan to extend it in future releases. Upcoming models and GPT-4o-mini (2024-07-18) 49.7 % $2.38 1221 B 14.1
prompt tuning will yield new queries with features not yet covered GPT-4o (2024-07-18) 52.3 % $26.27 1342 B 16.2
and increase the query text size. The SQLStorm methodology is Claude 3.0 Haiku (2024-03-07) 38.1 % $3.29 1300 B 13.4
the first step into systematically exploiting LLMs and Foundation Claude 3.5 Haiku (2024-10-22) 57.9 % $6.73 1110 B 13.6
Claude 3.5 Sonnet (2024-10-22) 15.4 % $30.48 1575 B 20.9
Models for database benchmarking.
Prompt PF attempts to resolve this issue in a second pass through Table 9: Database systems used for evaluating different use
GPT-4o-mini. However, the queries are too complex for the small cases for SQLStorm. The table also reports the number of
model to fix the mistakes. Rewriting the queries with the Claude 3.5 queries each system can execute on SQLStorm-1.
Sonnet might be more effective but increases the cost significantly.
SQLStorm v1.0 solely relies on GPT-4o-mini for query generation. System (Version) Success
Syntax Exec. Timeout
Crash
Differ.
Error Error + OOM Result
However, we observe a trend that newer and larger models are
capable of generating more complex queries. For example, GPT-4o Umbra (25.01) 18,165 18 34 34 0 2.0 %
→ before SQLStorm 16,068 2,053 9 5 116 3.2 %
generates twice as many operators as GPT-3.5-turbo and doubles PostgreSQL (17.0) 15,731 87 13 2,420 0 0.9 %
the code size. Claude 3.5 Sonnet gives an idea of the potential of the DuckDB (1.2.0) 15,208 2,164 168 711 0 15.5 %
Hyper (0.0.21200) 13,316 4,727 37 123 48 1.7 %
next generation of LLMs. We will explore these models in future
DBMS X 2,938 14,566 307 440 0 8.3 %
releases of SQLStorm and optimize the prompts further. → rewritten 12,451 3,646 1,093 1,061 0 10.3 %
DBMS Y 8,744 9,200 0 307 0 15.1 %
→ rewritten 13,138 4,647 20 442 4 25.2 %
3.9 Availability and Future Releases
SQLStorm serves as both a methodology and a new database bench-
mark. With this paper, we release version 1.0 of SQLStorm, includ- (OOM), or fatal crashes. We consider a query fatal if it crashes the
ing all associated artifacts – the query sets, benchmark results, and database system or the system is not responsible within 100 seconds
recorded traces – and provide scripts to run the generated queries after the query exceeds its time limit. Additionally, we report the
in more than six database systems. Additionally, we also release percentage of queries that return incorrect results, considering only
the source code and prompts used to generate, rewrite, and select those with deterministic outputs (see Table 4).
queries compatible with PostgreSQL, Umbra, and DuckDB. The en- None of the six systems can execute all queries successfully.
tire process is fully automated, enabling the creation of large-scale Umbra has the highest success rate with 18,165 queries, followed
benchmarks without human intervention. by PostgreSQL with 15,731 queries. However, both systems still
For future releases of SQLStorm, we plan to incorporate more experience syntax errors as they do not support some features other
datasets, prompts, and new models to increase the workload’s di- systems do. Timeouts are common in PostgreSQL due to its slower
versity and complexity. New LLMs and prompts are selected based performance on analytical [Link] have already improved the
on their cost, query quality, and whether they can generate queries current version of Umbra with the SQLStorm queries. Before these
with new features, e.g., query tree shapes and operator instances. changes, Umbra failed for over 2,000 queries and crashed for 116.
We encourage other researchers and database developers to propose For the other systems, compatibility is a bigger issue. Hyper, for
datasets, prompts, or query patterns to better reflect real-world instance, fails for 15.9 % of the queries as it lacks the string_agg
workloads. For instance, SQLStorm can be extended to support and string_to_array functions. More interestingly, 48 queries
graph workloads, streaming data analysis, and other domains. crashed Hyper, requiring a system restart to continue the bench-
mark. For the large SQLStorm-220 dataset, we also observe database
crashes in DuckDB and Umbra. Only the commercial DBMS X and
4 EXAMPLE USE CASES
PostgreSQL are stable and do not experience any fatal queries.
The SQLStorm benchmark is designed to evaluate the performance While the first four systems can parse most queries, the commer-
of database systems using a real-world dataset and a realistic work- cial systems struggle with the SQLStorm. Their SQL differs from
load. In this section, we will explore various use cases for SQLStorm. the PostgreSQL dialect. As a result, DBMS X has syntax errors on
We conduct all benchmarks on a server with an AMD EPYC more than 14,500 queries. To evaluate these systems on a large set
9454P CPU (48 cores, 96 threads) and 384 GB of RAM. We evalu- of queries, we rewrite the benchmark using GPT-4o-mini to the
ate four database systems implementing the PostgreSQL dialect: system-specific dialect using the following simple prompt:
Umbra, PostgreSQL, DuckDB, and Tableau Hyper. In addition, we
PR Convert the following PostgreSQL query to <DIALECT> syntax.
also test two commercial systems, DBMS X and Y, which adopt a Remember to put all ungrouped columns and columns that
different SQL dialect. We list the different database systems and appear in window functions into the group by clause. Do
their versions in Table 9. The database systems run in privileged not explain the query, only output the converted query.
Docker containers to improve reproducibility. A Python script sends The prompt works well for both commercial systems; we replace
queries to the database and measures the end-to-end latency. Ex- <DIALECT> with the system dialect name and let the LLM translate
cept for PostgreSQL, all systems use a columnar storage format and the queries. With the rewritten queries, every system can execute
implement vectorized or compilation-based query execution. at least two-thirds of SQLStorm successfully. Of course, the LLM
might accidentally change the semantics of the query; for DBMS X,
4.1 Improving Compatibility and Robustness the number of incorrect queries increases by 2 %, while DBMS Y
First, we investigate the systems’ compatibility and robustness sees an increase of 10 %. Consequently, the rewritten queries are
when executing the queries. We run all queries on SQLStorm-1 not guaranteed to perform the same computations as the original
and report the number of successful and failed queries in Table 9. queries, and comparing the runtimes requires some caution.
Queries might fail for multiple reasons, such as syntax errors, execu-
tion failures (e.g., arithmetic overflows, scalar subqueries with mul- 4.2 Evaluating Cardinality Estimation
tiple results, recursion depth exceeded, etc.), timeouts (we cancel Cardinality estimation is a crucial part of query optimization. Ac-
a query if it does not finish within 10 seconds), out-of-memory curate estimates are decisive for finding a cheap and robust query
 Umbra Umbra (TPC-H) DuckDB DuckDB (TPC-H)
Crash Error Timeout + OOM Performance

Runtime [geomean, ms]

200 3K 120 30

#Queries
150 2K 80 20
100
50 SQLStorm 1K 40 10
0 0 0 0
300 3K 1K 150

#Queries
2K 750
200 500 100
v1.0.0 1K
100 250 50
0 0 0 0
Jun. 23 Oct. 24 Jun. 23 Oct. 24 Jun. 23 Oct. 24 Jun. 23 Oct. 24

Figure 6: SQLStorm on Umbra and DuckDB over the last 2

years. In Oct. 24 we created SQLStorm and started fixing bugs
in Umbra (dotted line). The dashed line in the last plot shows
the geometric mean runtime on TPC-H 10 GB. SQLStorm
allowed Umbra to catch up with DuckDB in terms of crashes
Figure 5: Estimation error in SQLStorm-1 and JOB. Each box and it reduced the number of errors significantly.
plot shows the error distribution for operators with the given
number of downstream operators. While underest. are more
common on JOB, overest. dominate in SQLStorm as it com-
bines different operators. Accurately estimating joins alone PostgreSQL towards overestimations. JOB lacks this characteristic;
is insufficient for precise cardinality estimations. it only includes inner joins and aggregations at the end of long join
pipelines, missing the complexity of real-world workloads [28].
For robust query optimization, more accurate cardinality esti-
mate algorithms are needed. SQLStorm is a good starting point for
plan, as mistakes during query optimization can accumulate [11,
investigating these aspects further. Large-scale benchmarks help to
30, 38, 42]. In this section, we evaluate the accuracy of Umbra’s,
identify symptomatic weaknesses and edge cases in query optimiz-
DuckDB’s, and PostgreSQL’s query optimizer by comparing the
ers and cardinality estimators. Furthermore, database developers
estimated cardinalities from the query plans with the actual car-
can use SQLStorm to identify the impact of changes made in the
dinalities. Figure 5 shows the estimation error for different levels
query optimizer on the plan quality. For instance, several lines of
in the query tree for the three systems. We compute the factor by
recent work focus on making cardinality estimation and query pro-
which the estimate and the exact cardinalities differ and distinguish
cessing more robust [24, 47, 62, 65, 66]; SQLStorm can be used to
between under- and overestimation. The number of nodes in all
evaluate the effectiveness of these approaches.
subtrees of an operator determines the number of downstream op-
erators. For instance, while table scans have no children and their
number of downstream operators is always 0, a join operator has 4.3 Regression Testing
two subtrees, and we sum up the number of operators in both trees. Given the complexity of modern database systems, regression test-
All three systems perform reasonably well for up to three down- ing plays a crucial role in maintaining the integrity and performance
stream operators, maintaining a median estimation error close to 1. of database applications [17, 61]. For example, Umbra and DuckDB
However, as the number of downstream operators increases, the use TPC-H and TPC-DS to ensure reliability, efficiency, and func-
error grows as well (note the logarithmic scale). Misestimations tionality across versions. However, these benchmarks have limited
propagate through the tree, making estimating the cardinality for scope, and the small query set may not reveal all issues. SQLStorm
operators higher up more challenging. For eight or more operators, tests a broader range of features and unveils performance gains
DuckDB and PostgreSQL misestimate the cardinality by at least and losses on a large workload.
two orders of magnitude for the majority of queries. Umbra, in Figure 6 investigates how the robustness and compatibility of
contrast, is more robust and keeps the median error below 3 for Umbra and DuckDB changed over time. In January 2023, Umbra
all depths. Nevertheless, a significant fraction of the queries are could execute only 15,102 queries on SQLStorm-1. Until Septem-
over- and underestimated by more than four orders of magnitude, ber 2024, the number of successful queries increased to 16,067 as
even for a few downstream operators. DuckDB experiences errors we implemented more features and fixed bugs in Umbra. To our
as severe as 14 orders of magnitude. surprise, the number of fatal queries that cause system crashes tem-
We included the estimation errors on the Join Order Benchmark porarily increased to 232 at the start of 2024. We fixed these issues;
for comparison. A different pattern emerges in JOB: underesti- however, it took several months to discover and resolve them. After
mations are more common than overestimations, and misestima- introducing SQLStorm, we could fix 90 % of the crashes within one
tions exceeding four orders of magnitude are less frequent. The month, and until January 2025, we resolved all fatal queries and
combination of different operators, in particular, aggregations and most of the timeouts and exceptions on SQLStorm-1. At the time
joins, makes cardinality estimation more difficult on SQLStorm of writing this paper, only 86 queries remain that Umbra cannot
and results to more extreme errors. In addition, outer joins and execute. SQLStorm greatly improves development speed and helps
incorrect join conditions skew the estimation error in Umbra and to identify and fix issues early.
 SQLStorm-1 SQLStorm-12 SQLStorm-222 SQLStorm-1 SQLStorm-222
1e4 ↑ Umbra faster ↑ Umbra faster
Umbra DuckDB 3 3

B uts
B s

B s
kD or

kD or
1

kD eo
1

uc rr

uc rr
1e2

SQLStorm

D Tim
+ OOM

+ OOM
Timeout

Timeout

D E

D E
7.93x

Speedup
10x 13x 4.11x
Density

uc
4 5 4
1
1e2
2 2
1e4 DuckDB faster ↓ DuckDB faster ↓
0 18,204 0 10,983 15,350
+ OOM

+ OOM
Timeout

Timeout

TPC-DS
Density

Queries Queries

Figure 8: Umbra vs. DuckDB. We report per-query speedups

1 100 10000 1 100 10000 of Umbra over DuckDB in ascending order. Umbra succedds
Runtime [ms] Runtime [ms] on more queries and executes the majority of queries faster.
TPC-DS 1 TPC-DS 10 TPC-DS 100

Figure 7: Scalability of Umbra and DuckDB on SQLStorm and less than 100 ms, and timeouts are rare. As the scale factor increases,
TPC-DS. The dashed lines show the median runtime. Queries the queries take longer to finish, and timeouts become more fre-
in TPC-DS are closer to the median and timeouts are almost quent. Nevertheless, Umbra and DuckDB scale well, and the median
non-existent. Scaling on SQLstorm is more challenging. runtime only increases by 8× and 10×, respectively, when moving
from SQLStorm-12 to SQLStorm-220. On SQLStorm-220, DuckDB
times out on 4,938 queries, while Umbra exceeds the time limit of
DuckDB started with 361 crashes in January 2023, and the de- 10 s for 1,807 queries and the memory limit for 1,166 queries.
velopers constantly reduced this number as they prepared for the We also included the runtime distribution of TPC-DS, which
major 1.0.0 release in June 2024. At that point in time, some occa- consists of 100 query templates and is considered one of the most
sional crashes still occurred, and the last ones were addressed in extensive and challenging analytical benchmarks. TPC-DS lacks
the latest release, 1.2.0. DuckDB is supported by a large community outliers, and the runtimes are spread closer to the median. Like real-
that played a crucial role in identifying the bugs, even without world workloads [56, 60], SQLStorm covers a broader spectrum and
SQLStorm. However, queries that return with error in DuckDB experiences more timeouts. Furthermore, the median runtime of
remain almost unchanged over time. Syntax errors dominate this TPC-DS increases by only 3× from TPC-DS-10 to TPC-DS-100 even
category: there is no immediate need for DuckDB to implement though the data grows by a factor of 10. In SQLStorm, 20× more
missing PostgreSQL construct as users can adapt their queries. The data leads to 10× longer runtimes in DuckDB. Therefore, scaling
number of timeouts in DuckDB fluctuates between 550 and 700 up on SQLStorm is more challenging than on TPC-DS. We believe
queries. SQLStorm aids in pinpointing regressions and addressing that solving these scalability challenges involves not just better
them before releasing a new version. engineering but also innovations in query optimization and query
While Umbra does not see a significant change in runtime, the processing. For example, one of the queries that succeeds on the
geometric mean of the end-to-end time improve by almost 2× in small dataset but fails on the larger datasets performs a join with
DuckDB of the last two years. For Umbra, we observe a perfor- substring search as the join predicate of the form R.a LIKE ’%’
mance regression of 11.4 % around January 2024 that was resolved || S.b || ’%’. In existing systems this semantically meaningful
in August of the same year. This regression was not evident in the query results in quadratic runtime even though the query could be
TPC-H 10 GB benchmark, which explains why we did not notice efficiently processed using a suffix array.
it earlier. In DuckDB, the latest release improved the performance
on SQLStorm considerably. Conversely, the same release caused 4.5 Comparing Systems
a performance regression of 22 % on TPC-H. This highlights the Existing database systems can only execute a subset of SQLStorm.
limitations of TPC-H as a sole benchmark, as it may overlook perfor- Because this subset differs between systems, statistics such as aver-
mance regressions or improvements that become evident in more age or geometric mean can be somewhat misleading, e.g., a system
diverse, real-world workloads like SQLStorm. that fails on difficult queries might have good average performance.
Users would probably care at least as much about queries that fail
4.4 Scaling with the Dataset Size or take a very long time.
In contrast to other real-world datasets, such as IMDB [30], the To illustrate this behavior, we compare the runtimes and fail-
StackOverflow dataset and therefore SQLStorm is available in dif- ures for Umbra and DuckDB, shown in Figure 8. We consider only
ferent sizes: 1 GB (SQLStorm-1), 12 GB (SQLStorm-12), and 220 GB queries that run in one of the evaluated systems and calculate the
(SQLStorm-220). This substantially broadens the applicability of speedup for every query. If a query timeouts or fails in one sys-
SQLStorm, e.g., making it possible to run the benchmark on both tem but succeeds in the other, we set the speedup to 1e4 in the
small machines and larger, e.g., distributed, deployments. Increasing figure. The speedups are arranged in ascending order from left to
the dataset size and comparing the performance for the same query right: first, the queries that fail or timeout in Umbra but succeed
furthermore provides insights into a query engine’s scalability. in DuckDB; next, the queries that run in both systems; and finally,
Figure 7 shows the runtime distribution of the queries on the the queries that fail or timeout in DuckDB but succeed in Umbra.
three datasets for Umbra and DuckDB. The two systems behave This visualization provides a comprehensive but succinct overview
broadly similarly. On the small dataset, most queries complete in between two systems, even for thousands of queries.
 Umbra vs. PostgreSQL Umbra vs. DuckDB Umbra vs. Hyper Umbra vs. DBMS X Umbra vs. DBMS Y
1e4 ↑ Umbra ↑ Umbra faster ↑ Umbra faster ↑ Umbra faster ↑ Umbra faster
84.10x

er rs

S ors

S ors
s
27.65x

yp ro
ut

rs

rs

BM rr

BM rr
1e2

H Er
ro

ro
11.26x

Y
eo

D E

D E
6.79x
Speedup

Er

Er
m
2.22x

Ti
1
1e2
1e4 PostgreSQL faster ↓ DuckDB faster ↓ Hyper faster ↓ DBMS X faster ↓ DBMS Y faster ↓
0 10,829 17,531 0 17,511 0 12,608 17,559 0 11,339 17,518 0 10,956 17,512
Queries Queries Queries Queries Queries

Figure 9: System comparison on SQLStorm-10. SQLStorm uncovers for every system promising queries for further optimizations.

Table 10: The query generation process, the number of exe- (real-world or synthetic) datasets as well. We tested this using
cutable queries per system, and the number of operators on the well-known analytical benchmarks: TPC-H/-DS, and JOB [30].
the StackOverflow, the TPC-H and the JOB dataset. The methodology remains the same: GPT-4o-mini generates 35,000
queries using the seven prompts from Table 2 for each dataset; only
Dataset Query Selection Systems avg. the schema is adjusted to match the dataset.
Parse 1 Parse 2 Exec. Ops. Table 10 shows the number of queries for all three datasets, how
StackOverflow 15206 20218 18251 12161 18133 13712 13.9 many are executable in PostgreSQL, Umbra, and DuckDB, and the
TPC-H 16964 21074 17036 6964 16777 15301 16.0 average number of operators per query. Although the specific num-
TPC-DS 13361 16037 15242 12309 15053 14461 12.9 bers differ, the broad picture is similar to the original SQLStorm.
JOB 14749 17425 11714 2720 10948 9495 16.8
After pre-processing and filtering, more than 10,000 queries remain,
enough to stress-test any database system and provide a compre-
hensive evaluation. On TPC-H and JOB queries have two operators
The plot reveals some interesting findings: 1 is the medium more as they perform more joins than the queries from the other
speedup of both systems’ executable queries (success or timeout). two datasets. Aside from this difference, GPT-4o-mini generates a
Umbra is 4.11× faster than DuckDB on SQLStorm-1 and 7.93× on similar distribution of operators across all datasets.
SQLStorm-220. While at 2 , DuckDB performs significantly better SQLStorm is not limited to the StackOverflow data and can be
than Umbra, at 3 , Umbra outperforms DuckDB by up to 1,000×. applied other datasets as well. The technique is a simple and cost-
These queries are particularly interesting for database developers as efficient way to generate large-scale benchmarks for database sys-
the other systems perform multiple orders of magnitude better. Such tems. We make all the queries publicly available, and database devel-
large performance differences are usually caused by differences in opers can use the SQLStorm prompts to generate new benchmarks
query optimizations such as missing optimization rules or large on custom datasets.
cost model errors.
The red area at 4 shows the queries that fail in DuckDB but 5 SUMMARY AND FUTURE WORK
succeed in Umbra. In 5 , DuckDB encounter timeouts while Um- SQLStorm shifts database benchmarking from handcrafted queries
bra finishes the queries in under 10 seconds. Even on the small to large-scale LLM-generated workloads. We demonstrate that large
SQLStorm-1 dataset, the number of unsuccessful queries in DuckDB language models generate diverse, complex, and unpredictable
is considerable. Umbra also has queries that time out or fail but queries that encompasses a broad range of SQL constructs. The
succeed in DuckDB. However, the number of such instances is small initial SQLStorm release includes 18,251 queries on a real-world
and almost invisible in the figures. dataset. These challenging queries reveal weaknesses and opti-
In Figure 9, we repeat the analysis between Umbra and all other mization potential in database system, fostering new research di-
systems on SQLStorm-12. Interestingly, each system outperforms rections in query optimization, cardinality estimation and robust
Umbra on certain queries. For instance, PostgreSQL, which is typ- data processing. SQLStorm complements existing benchmarks like
ically 84× times slower than Umbra due to its row-based storage TPC-H and TPC-DS, which are limited by a few query templates
and volcano-style query execution, still surpasses Umbra in per- and small feature spaces. While these benchmarks contain high-
formance on 108 out of 17,531 queries. These queries will help quality expert-written queries, our generated queries occasionally
pinpoint weaknesses in Umbra and guide further performance im- include rare SQL constructs, semantic mistakes, and edge cases
provements. Database developers can use SQLStorm to explore capable of stressing database systems.
optimization opportunities in their systems and uncover missing SQLStorm v1.0 leverages OpenAI’s GPT-4o-mini model. Future
features. While the median runtime indicates the performance ben- releases of this benchmark will incorporate next-generation LLMs,
efits of different architectures, such as columnar versus row-based further enhancing complexity and diversity. Database engineers
storage, the long tails reveal isolated performance bugs. can refine the prompts and datasets to tailor the benchmark to their
needs and generate queries to new SQL features such as property
4.6 Generating New Benchmarks graph queries or ASOF joins. This paper takes the first step towards
We selected the StackOverflow dataset because it contains real- automated large-scale database benchmark generation in the LLM
world data and features more and longer strings than synthetic era; SQLStorm offers a scalable real-world dataset and a replicable,
datasets. Yet, the SQLStorm methodology can be applied to other cost-effective query generation pipeline.
REFERENCES [25] Andreas Kipf, Thomas Kipf, Bernhard Radke, Viktor Leis, Peter A. Boncz, and
[1] 2022. Database schema documentation for the public data dump and SEDE. Re- Alfons Kemper. 2019. Learned Cardinalities: Estimating Correlated Joins with
trieved February 1, 2025 from [Link] Deep Learning. In CIDR. [Link].
database-schema-documentation-for-the-public-data-dump-and-sede [26] Skander Krid, Mihail Stoian, and Andreas Kipf. 2025. Redbench: A Benchmark
[2] 2024. 2024 Developer Survey. Retrieved February 1, 2025 from [Link] Reflecting Real Workloads. In aiDM@SIGMOD. ACM.
[Link]/2024/ [27] Jiale Lao, Yibo Wang, Yufei Li, Jianping Wang, Yunjia Zhang, Zhiyuan Cheng,
[3] 2025. How do I access a data dump for a Stack Exchange site? Retrieved February Wanghu Chen, Mingjie Tang, and Jianguo Wang. 2024. GPTuner: A Manual-
1, 2025 from [Link] Reading Database Tuning System via GPT-Guided Bayesian Optimization. Proc.
[4] 2025. Stack Exchange. Retrieved February 1, 2025 from [Link] VLDB Endow. 17, 8 (2024), 1939–1952.
com/about [28] Kukjin Lee, Anshuman Dutt, Vivek R. Narasayya, and Surajit Chaudhuri. 2023.
[5] Peter Akioyamen, Zixuan Yi, and Ryan Marcus. 2024. The Unreasonable Effec- Analyzing the Impact of Cardinality Estimation on Execution Plans in Microsoft
tiveness of LLMs for Query Optimization. CoRR abs/2411.02862 (2024). SQL Server. Proc. VLDB Endow. 16, 11 (2023), 2871–2883.
[6] Amazon. 2023. Amazon Q generative SQL. Retrieved May 1, 2025 [29] Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin
from [Link] Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong,
generative-sql-query-editor-preview/ Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2025. Spider 2.0:
[7] Lawrence Benson, Carsten Binnig, Jan-Micha Bodensohn, Federico Lorenzi, Jigao Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows.
Luo, Danica Porobic, Tilmann Rabl, Anupam Sanghi, Russell Sears, Pinar Tözün, In ICLR. [Link].
and Tobias Ziegler. 2024. Surprise Benchmarking: The Why, What, and How. In [30] Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter A. Boncz, Alfons Kemper,
DBTest@SIGMOD. ACM, 1–8. and Thomas Neumann. 2015. How Good Are Query Optimizers, Really? Proc.
[8] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, Simran Arora, VLDB Endow. 9, 3 (2015), 204–215.
Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma [31] Guoliang Li, Xuanhe Zhou, and Xinyang Zhao. 2024. LLM for Data Management.
Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Proc. VLDB Endow. 17, 12 (2024), 4213–4216.
Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya [32] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang,
Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li,
John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Kevin Chen-Chuan Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023.
Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale
Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Database Grounded Text-to-SQLs. In NeurIPS.
Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha [33] Yiyan Li, Haoyang Li, Pu Zhao, Jing Zhang, Xinyi Zhang, Tao Ji, Luming Sun,
Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Cuiping Li, and Hong Chen. 2024. Is Large Language Model Good at Database
Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. 2021. Knob Tuning? A Comprehensive Experimental Evaluation. CoRR abs/2408.02213
On the Opportunities and Risks of Foundation Models. CoRR abs/2108.07258 (2024).
(2021). [34] Zhaodonghui Li, Haitao Yuan, Huiming Wang, Gao Cong, and Lidong Bing. 2024.
[9] Peter A. Boncz, Angelos-Christos G. Anadiotis, and Steffen Kläbe. 2017. JCC-H: LLM-R2: A Large Language Model Enhanced Rule-based Rewrite System for
Adding Join Crossing Correlations with Skew to TPC-H. In TPCTC (Lecture Notes Boosting Query Efficiency. Proc. VLDB Endow. 18, 1 (2024), 53–65.
in Computer Science, Vol. 10661). Springer, 103–119. [35] Jie Liu and Barzan Mozafari. 2024. Query Rewriting via Large Language Models.
[10] Peter A. Boncz, Thomas Neumann, and Orri Erling. 2013. TPC-H Analyzed: CoRR abs/2403.09060 (2024).
Hidden Messages and Lessons Learned from an Influential Benchmark. In TPCTC [36] Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Al-
(Lecture Notes in Computer Science, Vol. 8391). Springer, 61–76. izadeh, and Tim Kraska. 2021. Bao: Making Learned Query Optimization Practical.
[11] Surajit Chaudhuri, Vivek R. Narasayya, and Ravishankar Ramamurthy. 2009. In SIGMOD Conference. ACM, 1275–1288.
Exact Cardinality Query Optimization for Optimizer Testing. Proc. VLDB Endow. [37] Nestor Maslej, Loredana Fattorini, C. Raymond Perrault, Vanessa Parli, Anka
2, 1 (2009), 994–1005. Reuel, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James
[12] Sibei Chen, Ju Fan, Bin Wu, Nan Tang, Chao Deng, Pengyi Wang, Ye Li, Jian Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, and Jack Clark. 2024.
Tan, Feifei Li, Jingren Zhou, and Xiaoyong Du. 2024. Automatic Database Artificial Intelligence Index Report 2024. CoRR abs/2405.19522 (2024).
Configuration Debugging using Retrieval-Augmented Language Models. CoRR [38] Guido Moerkotte, Thomas Neumann, and Gabriele Steidl. 2009. Preventing Bad
abs/2412.07548 (2024). Plans by Bounding the Impact of Cardinality Estimation Errors. Proc. VLDB
[13] Shaleen Deep, Anja Gruenheid, Kruthi Nagaraj, Hiro Naito, Jeffrey F. Naughton, Endow. 2, 1 (2009), 982–993.
and Stratis Viglas. 2020. DIAMetrics: Benchmarking Query Engines at Scale. [39] Ingo Müller, Cornelius Ratsch, and Franz Färber. 2014. Adaptive String Dic-
Proc. VLDB Endow. 13, 12 (2020), 3285–3298. tionary Compression in In-Memory Column-Store Database Systems. In EDBT.
[14] Jonathan Dees and Peter Sanders. 2013. Efficient many-core query execution in [Link], 283–294.
main memory column-stores. In ICDE. IEEE Computer Society, 350–361. [40] Avanika Narayan, Ines Chami, Laurel J. Orr, and Christopher Ré. 2022. Can
[15] Bailu Ding, Surajit Chaudhuri, Johannes Gehrke, and Vivek R. Narasayya. 2021. Foundation Models Wrangle Your Data? Proc. VLDB Endow. 16, 4 (2022), 738–
DSB: A Decision Support Benchmark for Workload-Driven and Traditional 746.
Database Systems. Proc. VLDB Endow. 14, 13 (2021), 3376–3388. [41] Parimarjan Negi, Laurent Bindschaedler, Mohammad Alizadeh, Tim Kraska,
[16] Markus Dreseler, Martin Boissier, Tilmann Rabl, and Matthias Uflacker. 2020. Jyoti Leeka, Anja Gruenheid, and Matteo Interlandi. 2023. Unshackling Database
Quantifying TPC-H Choke Points and Their Optimizations. Proc. VLDB Endow. Benchmarking from Synthetic Workloads. In ICDE. IEEE, 3659–3662.
13, 8 (2020), 1206–1220. [42] Thomas Neumann and César A. Galindo-Legaria. 2013. Taking the Edge off
[17] Florian Haftmann, Donald Kossmann, and Eric Lo. 2007. A framework for Cardinality Estimation Errors using Incremental Execution. In BTW (LNI, Vol. P-
efficient regression tests on database applications. VLDB J. 16, 1 (2007), 145–164. 214). GI, 73–92.
[18] Yuxing Han, Ziniu Wu, Peizhi Wu, Rong Zhu, Jingyi Yang, Liang Wei Tan, [43] Thomas Neumann and Alfons Kemper. 2015. Unnesting Arbitrary Queries. In
Kai Zeng, Gao Cong, Yanzhao Qin, Andreas Pfadler, Zhengping Qian, Jingren BTW (LNI, Vol. P-241). GI, 383–402.
Zhou, Jiangneng Li, and Bin Cui. 2021. Cardinality Estimation in DBMS: A [44] Manuel Rigger and Zhendong Su. 2020. Testing Database Engines via Pivoted
Comprehensive Benchmark Evaluation. Proc. VLDB Endow. 15, 4 (2021), 752– Query Synthesis. In OSDI. USENIX Association, 667–682.
765. [45] Andreas Seltenreich, Bo Tang, and Sjoerd Mullender. 2024. SQLsmith: A random
[19] Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran SQL query generator. Retrieved February 1, 2025 from [Link]
Huang, and Xiao Huang. 2024. Next-Generation Database Interfaces: A Survey sqlsmith
of LLM-based Text-to-SQL. CoRR abs/2406.08426 (2024). [46] Snowflake. 2025. Snowflake Copilot. Retrieved May 1, 2025 from [Link]
[20] Karl Huppler. 2009. The Art of Building a Good Benchmark. In TPCTC (Lecture [Link]/en/user-guide/snowflake-copilot
Notes in Computer Science, Vol. 5895). Springer, 18–30. [47] Mihail Stoian, Andreas Zimmerer, Skander Krid, Amadou Latyr Ngom, Jialin
[21] Stack Exchange Inc. 2024. Database Administrators StackExchange. Retrieved Ding, Tim Kraska, and Andreas Kipf. 2025. Parachute: Single-Pass Bi-Directional
October 1, 2025 from [Link] Information Passing. Proc. VLDB Endow. 18, 10 (2025).
[22] Stack Exchange Inc. 2024. Mathematics StackExchange. Retrieved October 1, [48] Zhaoyan Sun, Xuanhe Zhou, and Guoliang Li. 2024. R-Bot: An LLM-based Query
2025 from [Link] Rewrite System. CoRR abs/2412.01661 (2024).
[23] Stack Exchange Inc. 2024. StackOverflow. Retrieved October 1, 2025 from [49] Jan Vincent Szlang, Sebastian Bress, Sebastian Cattes, Jonathan Dees, Florian
[Link] Funke, Max Heimel, Michel Oleynik, Ismail Oukid, and Tobias Maltenberger.
[24] Mahmoud Abo Khamis, Vasileios Nakos, Dan Olteanu, and Dan Suciu. 2024. Join 2025. Workload Insights From The Snowflake Data Cloud: What Do Production
Size Bounds using lp -Norms on Degree Sequences. Proc. ACM Manag. Data 2, 2, Analytic Queries Really Look Like? Proc. VLDB Endow. 18, 11 (2025).
Article 96 (2024). [50] Jie Tan, Kangfei Zhao, Rui Li, Jeffrey Xu Yu, Chengzhi Piao, Hong Cheng, Helen
Meng, Deli Zhao, and Yu Rong. 2025. Can Large Language Models Be Query
 Optimizer for Relational Databases? CoRR abs/2502.05562 (2025). [61] Florian M. Waas, Leo Giakoumakis, and Shin Zhang. 2011. Plan space analysis:
[51] BIRD Team and Google Cloud. 2025. BIRD-CRITIC: Can LLMs Fix User Issues an early warning system to detect plan regressions in cost-based optimizers. In
in Real-World Database Applications? Retrieved May 1, 2025 from [Link] DBTest. ACM, 2.
[Link]/ [62] Qichen Wang, Bingnan Chen, Binyang Dai, Ke Yi, Feifei Li, and Liang Lin. 2025.
[52] DuckDB Team. 2025. DuckDB Documentation: PostgreSQL Compatibility. Re- Yannakakis+: Practical Acyclic Query Evaluation with Theoretical Guarantees.
trieved May 1, 2025 from [Link] Proc. ACM Manag. Data 3, 3, Article 235 (2025).
[Link] [63] Johannes Wehrstein, Timo Eckmann, Roman Heinrich, and Carsten Binnig.
[53] Pinar Tözün, Ippokratis Pandis, Cansu Kaynak, Djordje Jevdjic, and Anastasia 2025. JOB-Complex: A Challenging Benchmark for Traditional & Learned Query
Ailamaki. 2013. From A to E: analyzing TPC’s OLTP benchmarks: the obsolete, Optimization. In AIDB@VLDB.
the ubiquitous, the unexplored. In EDBT. ACM, 17–28. [64] Zhiming Yao, Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2025. A Query
[54] Immanuel Trummer. 2022. CodexDB: Synthesizing Code for Query Processing Optimization Method Utilizing Large Language Models. CoRR abs/2503.06902
from Natural Language Instructions using GPT-3 Codex. Proc. VLDB Endow. 15, (2025).
11 (2022), 2921–2928. [65] Zixuan Yi, Yao Tian, Zachary G. Ives, and Ryan Marcus. 2025. Low Rank Learning
[55] Immanuel Trummer. 2022. DB-BERT: A Database Tuning Tool that "Reads the for Offline Query Optimization. Proc. ACM Manag. Data 3, 3, Article 183 (2025).
Manual". In SIGMOD Conference. ACM, 190–203. [66] Junyi Zhao, Kai Su, Yifei Yang, Xiangyao Yu, Paraschos Koutris, and Huanchen
[56] Alexander van Renen, Dominik Horn, Pascal Pfeil, Kapil Vaidya, Wenjian Dong, Zhang. 2025. Debunking the Myth of Join Ordering: Toward Robust SQL Analyt-
Murali Narayanaswamy, Zhengchun Liu, Gaurav Saxena, Andreas Kipf, and Tim ics. Proc. ACM Manag. Data 3, 3, Article 146 (2025).
Kraska. 2024. Why TPC Is Not Enough: An Analysis of the Amazon Redshift [67] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou,
Fleet. Proc. VLDB Endow. 17, 11 (2024), 3694–3706. Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang,
[57] Alexander van Renen and Viktor Leis. 2023. Cloud Analytics Benchmark. Proc. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang,
VLDB Endow. 16, 6 (2023), 1413–1425. Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A Survey of Large
[58] Alexander van Renen, Mihail Stoian, and Andreas Kipf. 2024. DataLoom: Simpli- Language Models. CoRR abs/2303.18223 (2023).
fying Data Loading with LLMs. Proc. VLDB Endow. 17, 12 (2024), 4449–4452. [68] Xinyang Zhao, Xuanhe Zhou, and Guoliang Li. 2024. Chat2Data: An Interactive
[59] Adrian Vogelsgesang, Michael Haubenschild, Jan Finis, Alfons Kemper, Viktor Data Analysis System with RAG, Vector Databases and LLMs. Proc. VLDB Endow.
Leis, Tobias Mühlbauer, Thomas Neumann, and Manuel Then. 2018. Get Real: 17, 12 (2024), 4481–4484.
How Benchmarks Fail to Represent the Real World. In DBTest@SIGMOD. ACM, [69] Xuanhe Zhou, Guoliang Li, Zhaoyan Sun, Zhiyuan Liu, Weize Chen, Jianming
1:1–1:6. Wu, Jiesi Liu, Ruohang Feng, and Guoyang Zeng. 2024. D-Bot: Database Diagnosis
[60] Midhul Vuppalapati, Justin Miron, Rachit Agarwal, Dan Truong, Ashish Motivala, System using Large Language Models. Proc. VLDB Endow. 17, 10 (2024), 2514–
and Thierry Cruanes. 2020. Building An Elastic Query Engine on Disaggregated 2527.
Storage. In NSDI. USENIX Association, 449–462. [70] Xuanhe Zhou, Zhaoyan Sun, and Guoliang Li. 2024. DB-GPT: Large Language
Model Meets Database. Data Sci. Eng. 9, 1 (2024), 102–111.