The VLDB Journal (2025) 34:38

[Link]

REGULAR PAPER

PROMPT 4 VIS : prompting large language models with example

mining for tabular data visualization
Shuaimin Li1 · Xuanang Chen2 · Yuanfeng Song3 · Yunze Song4 · Chen Jason Zhang1 · Fei Hao1 · Lei Chen5

Received: 17 May 2024 / Revised: 7 February 2025 / Accepted: 4 March 2025 / Published online: 2 May 2025
© The Author(s) 2025

Abstract
We are currently in the epoch of Large Language Models (LLMs), which have transformed numerous technological domains
within the database community. In this paper, we examine the application of LLMs in text-to-visualization (text-to-vis).
The advancement of natural language processing technologies has made natural language interfaces more accessible and
intuitive for visualizing tabular data. However, despite utilizing advanced neural network architectures, current methods such
as Seq2Vis, ncNet, and RGVisNet for transforming natural language queries into DV commands still underperform, indicating
significant room for improvement. In this paper, we introduce Prompt4Vis, a novel framework that leverages LLMs and
In-context learning to enhance the generation of data visualizations from natural language. Given that In-context learning’s
effectiveness is highly dependent on the selection of examples, it is critical to optimize this aspect. Additionally, encoding
the full database schema of a query is not only costly but can also lead to inaccuracies. This framework includes two main
components: (1) an example mining module that identifies highly effective examples to enhance In-context learning capabilities
for text-to-vis applications, and (2) a schema filtering module designed to streamline database schemas. Comprehensive testing
on the NVBench dataset has shown that Prompt4Vis significantly outperforms the current state-of-the-art model, RGVisNet,
by approximately 35.9% on development sets and 71.3% on test sets. To the best of our knowledge, Prompt4Vis is the first
framework to incorporate In-context learning for enhancing text-to-vis, marking a pioneering step in the domain.

Keywords Text-to-vis · NLP for database · In-context learning · Large language model · Prompt engineering

1 Introduction
B Chen Jason Zhang
[Link]@[Link] We have entered the era of large language models (LLMs),
Shuaimin Li which have revolutionized nearly every field. This paper aims
[Link]@[Link] to explore the applications of large models in the field of data
Xuanang Chen visualization.
chenxuanang@[Link] As we know, big data serves as the primary driving force
Yuanfeng Song across various sectors. The analysis of big data to uncover
yfsong@[Link] underlying patterns is increasingly critical [26, 42, 53]. Data
Yunze Song visualization emerges as a powerful tool in realizing this
YunzeSong77@[Link] objective. Therefore, the topic of automatic data visualiza-
Fei Hao tion has captured growing interest within the database and
[Link]@[Link] data mining communities [25, 42, 44, 48, 51, 53, 54, 62]. For
Lei Chen example, Qin et al. [44] in VLDB journal surveyed how to
leichen@[Link] make data visualization more efficient and effective.
[Link]
1 The Hong Kong Polytechnic University, Hong Kong, China
2 4 University of Liverpool, Liverpool, UK
Institute of Software, Chinese Academy of Sciences, Beijing,
China 5 The Hong Kong University of Science and Technology, Hong
3 AI Group, WeBank Co., Ltd, Shenzhen, China Kong, China

123
38 Page 2 of 26 S. Li et al.

One essential step in conducting data visualization is already been learned during the training phase of LLMs. In
the formulation of visualization specifications using declar- contrast, text-to-vis is relatively new, so LLMs have a lim-
ative visualization languages (DVLs), i.e., Vega-Lite [47], ited understanding of it, making direct use of LLMs less
ggplot2 [61], ZQL [51], ECharts [19], and VizQL [15]. How- effective. To this end, in this paper, we propose a precisely
ever, this composing specification demands users possess a designed prompting framework called Prompt4Vis with an
thorough understanding of domain-specific data and famil- ICL paradigm to adapt LLMs to generate data visualization
iarity with the syntax of these languages, which presents a queries from NLQs. To maximize the ability of LLMs on the
significant challenge, especially for beginners. Thus, text-to- text-to-vis, Prompt4Vis introduces two key components to
visualization (text-to-vis), which aims to directly transform create clearer and more effective prompt text for LLMs.
natural language questions (NLQs) into data visualization Existing studies have demonstrated that the choice and
queries (DVQs) [26–28], has garnered more and more atten- order of examples significantly affect the final performance,
tion within the community. with impacts ranging from state-of-the-art (SOTA) levels to
An automatic text-to-vis system must first have a deep as random as guessing [23, 24]. Moreover, The prompt length
understanding of the NLQ and its corresponding database that LLMs can accept is limited, so selecting quality exam-
schema. Then, it needs to answer the given NLQ using the ples within this limited length for task-specific prompts is
correct data visualization language. To achieve this goal, an important topic of interest for many researchers [4, 65].
a series of efforts [6, 9, 26, 28, 35] have been made, e.g., Therefore, it is crucial to identify an effective set of examples
DeepEye [26], NL4DV [35], Seq2Vis [28], ncNet [29] and for the text-to-vis task within the ICL paradigm. Motivated by
RGVisNet [52]. Specifically, DeepEye and NL4DV rely this, an example mining module is designed to find out truly
on rule-based methodology or semantic parsing techniques effective demonstrations, wherein the similarity between the
in natural language processing, Seq2Vis [28] and ncNet candidate examples and the target example, the influence of
[29] attempt to build encoding-decoding frameworks using candidate examples on the target example, and the diver-
deep neural networks for text-to-vis, and RGVisNet [52] is sity among the candidate examples are all comprehensively
retrieval-and-generation combined framework for data visu- considered, which indeed helps LLMs know what and how to
alization language generation inspired by the concept of code perform well on this task. Specifically, the Euclidean distance
reuse. Although these efforts have achieved a noticeable between examples and the target input based on sentence vec-
enhancement in the performance of text-to-vis, such per- tor representation is employed to measure the similarity and
formance still falls short of expectations, especially when diversity of examples, and a contrastive learning-based influ-
the models are trained and tested in cross-domain settings. ence model is trained to bring examples related to positive
For instance, the overall accuracy of ncNet on the bench- influence closer and push away from negative examples, this
mark nvBench is above 90% [29], while on the cross-domain model can effectively measure the influence of examples. In
setting, the performance of ncNet is below 30% [52]. Specif- the process of selecting the prompt example set, we attempt
ically, cross-domain setting means that the examples in to maximize the similarity and influence scores between the
training and test sets have no overlap databases. candidate example set and the target example while ensuring
Recently, LLMs, especially the GPT series [3, 5, 38, 39], diversity within the candidate example set.
have revolutionized the field of natural language process- Moreover, given that encoding the entire schema for
ing (NLP). Leveraging their huge number of parameters and databases with numerous columns is not only expensive
training data, LLMs learn substantial world knowledge and and impractical but also introduces irrelevant information
perform in pairs with humans [2, 43]. Meanwhile, with the to increase the difficulty of selecting the correct data for
development of LLMs, In-context learning (ICL) that does LLMs, a schema filtering module is also proposed to sim-
not rely on large-scale labeled data and does not require plify the schema of the database. Specifically, considering
parameter updates [30, 36, 37, 60] attracts researchers in var- the all-round capabilities of LLMs, we prompt LLMs via
ious fields. ICL enables LLMs to make predictions about a in-context examples to help us select the necessary table in
new example by learning from only a few labeled examples. the schema of the database for the input question, eliminat-
Hence, it is feasible and promising to leverage LLMs to real- ing irrelevant and redundant schema related to the current
ize text-to-vis and also to effectively alleviate the problem of question.
insufficient generalization ability of existing methods. Extensive experiments are conducted on the widely-used
However, although directly prompting LLMs in the text- multi-domain dataset NVBench [28] to empirically verify
to-SQL task may significantly outperform smaller models. the effectiveness of our Prompt4Vis framework. Evaluation
Some preliminary experiments show that directly prompt- results demonstrate that Prompt4Vis not only outperforms
ing LLMs for text-to-vis can not yield ideal results. This is all baseline methods with obvious improvements but also
because text-to-SQL is a well-established task with a rela- shows better stability across different cross-domains.
tively rich body of knowledge and corpus, which may have In summary, our contributions are as follows:

123
Prompt4Vis: prompting large language models with example mining… Page 3 of 26 38

• We propose a novel framework, Prompt4Vis, for adapt-

ing LLMs to the tabular data visualization task. To the
best of our knowledge, we are the first to introduce LLMs
and advanced ICL to generate data visualizations for the
text-to-vis task. This novel approach brings new insights
to the text-to-vis task and will inspire further exploration
of this promising new paradigm.
• The example mining module in Prompt4Vis identifies
examples that maximize the benefits of In-context learn-
ing. We formulate the example mining problem, prove its
NP-hardness, and provide an approximation algorithm as
a solution with a theoretical guarantee.
• Our approach includes a contrastive learning-based
method to measure the influence of examples in In-
context learning and a method to collect training exam-
ples for the contrastive-based method. Fig. 1 Pipeline of tabular data visualization
• Extensive experiments demonstrate that the proposed
Prompt4Vis brings around 36% and 71% relative
improvements in overall accuracy on the development As shown in Table 1, each DVL comprises three essen-
and test sets, respectively. Ablation studies also ver- tial clauses (bold fonts) along with several auxiliary ones.
ify the effectiveness of all designed components in Notably, X and Y denote two distinct columns within the
Prompt4Vis. database table. The VISUALIZE clause specifies the type of
chart that users intend to generate. The SELECT clause is
responsible for extracting the desired columns, where X may
The rest of this paper is organized as follows: First, we
represent either the original column X or its binned equiv-
introduce the background knowledge in Sect. 2. Then, we
alent B I N (X ) (for example, binned by hour), and Y may
provide details of the proposed Prompt4Vis method in
correspond to either the original column Y or an aggregated
Sect. 3. Next, we present the experimental results and discuss
version AGG(Y ). The aggregation function AGG includes
the findings in Sect. 4. Finally, the related work is discussed
operations such as SU M, M AX , M I N , C OU N T , among
in Sect. 5, followed by limitations and future work in Sect. 6,
others. The FROM clause identifies the data source table
and the conclusions in Sect. 7.
from which the data is retrieved. The TRANSFORM clause
applies transformations such as binning or grouping to the
selected column X . The ORDER BY clause arranges X  or Y 
2 Background in either descending or ascending order along the x-axis or
y-axis of the visualization, respectively. Lastly, the WHERE
2.1 Preliminary concepts clause introduces conditions utilizing operators such as “≥”,
“=”, “=”, “≤”, and “LIKE”, facilitating refined data selec-
Natural Language Question (NLQ) is a human-understa- tion for visualization.
ndable expression used to describe the desired data visual- As depicted in Fig. 1, the DVQ corresponding to the NLQ
ization (DV), making it more user-friendly, particularly for “Bar chart x-axis date claim settled, y-axis the number of
novice users and those without a background in data science. date claim settled, ranked from high to low by the Y-axis” is
For example, in Fig. 1, a user could ask a NLQ like “Bar articulated as “Visualize BAR SELECT Date_Claim_Settled,
chart x axis date claim settled y axis the number of date COUNT(Date_Claim_Settled) FROM Settlements ORDER
claim settled, rank from high to low by the Y-axis” for DV BY COUNT(Date_Claim_Settled) DESC BIN Date_Claim_
systems. Settled BY YEAR”. This formulation explicitly instructs the
Data Visualization Query (DVQ) is proposed by Luo et generation of a bar chart, where the x-axis represents the set-
al. [26, 27], involves initially executing a query on a database tlement date, categorized annually, and the y-axis displays the
to retrieve the desired data, followed by defining the visual- aggregated count of settlements. The ordering of the data is
ization details for presenting the acquired data. Importantly, determined in descending order based on the count, prioritiz-
DVQs are not limited to a single declarative visualization lan- ing data points with higher frequencies for more immediate
guage (DVL). On the contrary, once the DVQ corresponding visual analysis.
to a specific question is obtained, it can be seamlessly trans- Visualization Specification usually follows the gram-
formed to suit any DVL. mar of a common declarative visualization language (DVL),

123
38 Page 4 of 26 S. Li et al.

Table 1 Mandatory and

optional clauses in DVQ [26] VISUALIZE TYPE ∈ {bar , pie, line}
  
SELECT X , Y , (X ∈ {X , B I N (X )}, Y ∈ Y , AGG(Y ))
FROM D
TRANSFORM X (using an operator ∈ {B I N , G R OU P})
 
ORDER BY X ,Y

WHERE X OP v
Bold fonts highlight the most critical operations in data visualization query processing

pointing out the details of visualization, e.g., data mapping, The final predicted response y p for x is the candidate answer
chart typologies, stylistic configurations, interactivity fea- exhibiting the highest probability, determined by:
tures, and layout design. Utilizing advanced DVLs, e.g.,
Vega-Lite [47], ggplot2 [61], the precise control of data y p = argmax P(y j |x) (2)
visualization is achieved. Figure 1 gives the visualization j∈Y
specification in vega-lite language.

3 P ROMPT 4V IS
2.2 Task definition

Text-to-vis aims to translate NLQs into DVQs, which is

a general step for tabular data visualization, as shown in
3.1 Overview
Fig. 1. Formally, given an NLQ q and corresponding database
In this section, we first give the ICL paradigm adapted for
schema s, the text-to-vis task aims to character the corre-
text-to-vis. Subsequently, we introduce the workflow of the
sponding DVQ v to answer q. Specifically, schema s consists
proposed Prompt4Vis framework.
of a collection of tables Ts = {t1 , t2 , ..., tn T }, where ti
ICL paradigm for text-to-vis. In the ICL framework, the
represents i th table in Ts , n T is the number of tables in
model is initially furnished with a task instruction and a set
Ts . And ti is composed of a collection of columns, i.e.,
of labeled examples as input, which predicts outputs for new
ti = {c1 , c2 , ..., cn t }, where n t is the number of columns
instances based on these labeled examples. For text-to-vis,
in ti .
each labeled example can be formalized as a triple: (q, s, v),
where q is the NLQ asked by users, s is the corresponding
2.3 In-context learning database schema for q, and v denotes the target DVQ. There-
fore, given a dataset D containing N triples (q, s, v), a target
In-context learning is a paradigm that follows language mod- question qt , and a LLM g, In-context learning to generated
els to learn tasks given only a few examples in the form DVQ vt based on database schema st can be formulated as:
of demonstration. [10]. This approach capitalizes on the
adeptness of highly-trained language models to appraise the vt = g( p pr e f i x , P, (qt , st )) (3)
likelihood of potential responses based on these demonstra-
tions [10]. where P is the set with K labeled examples for text-to-vis,
Formally, consider a query input text x accompanied by P = {(q1 , s1 , v1 ), (q2 , s2 , v2 ), ..., (q K , s K , v K )}.
a set of candidate answers Y = {y1 , y2 , ..., ym }, where Y Workflow of Prompt4vis. In this work, we aim to lever-
is a set of class labels for classification tasks or free-form age LLMs with an ICL framework with effective labeled
text for generative tasks. A proficiently trained language examples in the demonstrations for text-to-vis. Therefore, we
model g selects the candidate answer with the maximal give a brief but concise task instruction for LLMs as shown in
probability as the definitive response for x, predicated on Fig. 2, “/*Generate the DVQ for each question based on the
the demonstration M. Here, M consists of an optional database schema.*/”. Then, our Prompt4Vis finds effective
task instruction p pr e f i x and several illustrative examples examples with the example mining module, which considers
{(x1 , y1 ), (x2 , y2 ), ...(xk , yk )}, where k is the number of the the similarity, influence, and diversity of the selected labeled
examples. The probability of the i th candidate answer yi for examples in demonstrations. Since the labeled examples for
x in the In-context learning mechanism can be formalized as text-to-vis include complex database schema that may consist
follows: of a large number of tables, we further introduce a schema
filtering module for the proposed framework Prompt4Vis
P(yi |x) = g( p pr e f i x , M, x) (1) to reduce the irrelevant information in the database schema.

123
Prompt4Vis: prompting large language models with example mining… Page 5 of 26 38

Fig. 2 Workflow of Prompt4Vis, which prompts LLMs with an example mining module and a schema filtering module. The first module finds
truly effective examples for text-to-vis, and the second one simplifies the database schema

Therefore, effective examples with filtered schemas consist existing methods to directly calculate the influence scores, we
of the demonstrations in Prompt4Vis for LLMs to generate design an influence model and introduce the details about it
DVQs. in Sect. 3.2.5.
We detail example mining and schema filtering modules In particular, in ICL paradigm for text-to-vis, each prompt
of the proposed Prompt4Vis in the subsequent sections. example consists of a question, a database schema, and a data
visualization query. Since the goal of this task is to answer
3.2 Example mining a given NLQ, with the data schema serving merely as auxil-
iary information, and considering the significant differences
Previous research in ICL has demonstrated that similar exam- among database schemas of different databases, we opt to
ples of the target example can bring good performance [23, select examples based on NLQs as the unit.
65]. However, they ignored the direct task-oriented influence
of each example. To mitigate this gap, in addition to simi-
3.2.1 Methodology for calculating metrics
larity, we propose to introduce influence in the process of
example mining. At the same time, we introduce diversity
In this section, we define C(qi ) to measure the similarity of qi
to reduce the irrelevant information brought by similar can-
in D with the target question qt , I(qi ) to measure the rewards
didate examples. In summary, our example mining method
influence of qi , and V(A) to measure the diversity of an
optimizes three different dimensions: influence, similarity,
example subset A. In the following, we introduce the detailed
and diversity, to find out effective examples for text-to-vis.
definitions and explanations of C(qi ), I(qi ), and V(A).
In the following, we will first introduce the methodol-
Similarity C. The similarity metric C(qi ) is defined:
ogy for calculating the metrics of influence, similarity, and
diversity in Sect. 3.2.1. Then, we introduce the concentrate
algorithm to find effective examples based on the aforemen- ρ(qi , qt ) − ρ(q j , qt )
C(qi ) = 1 − (4)
tioned three measurements in Sect. 3.2.2. Since there are no ρ(ql , qt ) − ρ(q j , qt )

123
38 Page 6 of 26 S. Li et al.

where ρ(qi , qt ) measures the Euclidean distance of the vector qt . In other words, it aims to find a subset of the training set
representations between one single candidate NLQ qi and the that maximizes the similarity, influence, and diversity in the
ρ(q ,q )−ρ(q ,q )
target NLQ qt . ρ(qil ,qtt )−ρ(q jj ,qtt ) is the min-max normalization. meantime. Formally, it can be represented as follows:
j is the index of the NLQ that minimizes ρ: 
A∗ = arg max φ(qi ) + V(A) subject to |A| ≤ K (9)
j = arg minρ(q j , qt ) (5) A:A⊆D q ∈A
i
j∈|D|
φ(qi ) = αC(qi ) + (1 − α)I(qi ) (10)
and l is the index of the NLQ that maximizes ρ:
where φ(·) is the score function to calculate the weighted
l = arg maxρ(ql , qt ) (6) score of similarity and influence for each NLQ in the training
l∈|D| set.

Influence I. In relation to similarity measurement, we 3.2.3 Complexity analysis

define influence sore I(qi ) as below:

ω(qi , qt ) − ω(q j  , qt ) To establish the complexity of this problem, we now proceed

I(qi ) = (7) to prove that this example subset mining problem is NP-hard.
ω(ql  , qt ) − ω(q j  , qt ) We will achieve this by reducing the well-known Maximum
Coverage Problem (MCP) to our problem, thereby demon-
where ω(qi , qt ) denotes the influence score of the candidate strating that solving our problem efficiently would imply a
NLQ qi to q j . The evaluation of ω(., .) is carried out with an solution to MCP, which is a known NP-hard problem.
influence model. We define the influence of the examples as
how much reward it can bring for the final data visualization. Theorem 1 The example subset mining problem is NP-hard.
 
l is the index of the NLQ that minimizes ω(., .), and j is
the index of the NLQ that maximizes ω(., .). Moreover, We Proof We will prove that the example subset mining problem
will go through the details of the designed influence model is NP-hard by reducing the Maximum Coverage Problem
in Sect. 3.2.5. (MCP) to it.
Diversity V. The average distance between any two can- Maximum Coverage Problem (MCP): Given a collection
didate NLQs in a set measures the diversity in that set. of sets S1 , S2 , . . . , Sm and an integer k, the goal is to select
Therefore, we define the diversity measurement V(A) as: k sets such that their union covers the maximum number of
 elements.
qi ,q  ∈A ρ(qi , qi  ) Example Subset Mining Problem: Given a dataset D of
V(A) = i
(8) questions, a target question qt , and metrics C(qi ) for simi-
|A|
larity, I(qi ) for influence, and V(A) for diversity, select a
where ρ(qi , qi  ) is the Euclidean distance between one sin- subset A of size at most K that maximizes:
gle candidate NLQ qi and another candidate NLQ qi  . The 
greater the distance, the less similar the examples are to each (αC(qi ) + (1 − α)I(qi )) + V(A)
other. qi ∈A
Given that sentence embeddings generated by LLMs have
been validated as effective representations of real-world Reduction Construction: Map each element e ∈ U (the
sentence semantics, this study utilizes NLQ embeddings cal- universe of elements in MCP) to a question qi in D. Each
culated by sentence-transformers [45] in metric computation. set S j ∈ {S1 , S2 , . . . , Sm } corresponds to a combination of
In particular, sentence-transformers are specialized models similarity C(qi ) and influence I(qi ) scores for the questions
for sentence embedding, trained on extensive sentence- covering elements in S j . For each question qi corresponding
level datasets through a self-supervised contrastive learning to an element ei , define C(qi ) and I(qi ) to reflect the coverage
approach. in MCP, and set α to balance their contributions. The goal of
MCP, maximizing the number of covered elements, translates
3.2.2 Example subset mining to maximizing the sum of similarity and influence scores. The
diversity term V(A) ensures that selected questions are not
Having established the methodology for metric calculation, redundant, initially assuming V(A) = 0 and later extending
we will introduce the specific algorithm employed to identify it. The constraint |A| ≤ K maps directly to selecting k sets
effective examples in this section. in MCP.
The goal of example mining is to automatically select K Reduction Correctness: Given an instance of MCP with
effective examples from the training dataset for a target NLQ universe U = {e1 , e2 , . . . , en }, a collection of sets S =

123
Prompt4Vis: prompting large language models with example mining… Page 7 of 26 38

Algorithm 1 Example Subset Mining (ESM) increase φ(x):

Input: Training Set D, Example Number of ExampleSubset K ,
Score function φ(·). g(S ∪ {x}) − g(S) = φ(x),
Onput: Example Subset A
1: A←∅ g(T ∪ {x}) − g(T ) = φ(x).
2: while |A| < K do
3: z  ← arg maxz∈D−A (φ(z) + V ({z} ∪ A))
Since themarginal gain of adding x is the same for any set,
4: A ← A ∪ {z  };
5: return Example Subset A.
g(A) = qi ∈A φ(qi ) is submodular.
Next, we consider the term V(A), which is defined as:

qi ,q  ∈A ρ(qi , qi  )
{S1 , S2 , . . . , Sm }, and an integer k, we construct the corre-
V(A) = i
.
sponding instance of the example subset mining problem as |A|
follows. The questions D = {q1 , q2 , . . . , qn } each corre-
spond to an element ei . Define the similarity and influence Rewriting V(A) in a more convenient form:
 (C)  (I )
scores as C(qi ) = S j ei wi j and I(qi ) = S j ei wi j ,
2 
(C) (I )
where wi j and wi j reflect the importance of ei in S j . Define V(A) = ρ(qi , qi  ),
|A|
φ(qi ) = αC(qi ) + (1 − α)I(qi ), and the goal is to maximize
 {qi ,q  }∈(
i
A
2 )
qi ∈A φ(qi ). Given an MCP instance, construct the corre-
sponding similarity and influence scores and select K = k  
where A2 denotes the set of all unordered pairs of elements
in the example subset mining problem. from A.
NP-Hardness Proof: Each element ei in MCP is mapped To test submodularity, we examine the marginal gains
to a question qi with scores C(qi ) and I(qi ). The sum when adding a new element x to sets S and T , where S ⊆ T .

qi ∈A (αC(qi ) + (1 − α)I(qi )) corresponds to maximizing We need to show:
coverage in MCP. Since MCP is NP-hard, solving the exam-
ple subset mining problem in polynomial time would solve
 S (x) = V(S ∪ {x}) − V(S)
MCP in polynomial time, making the example subset mining
problem NP-hard. Incorporating diversity V(A) adds com- ≥ T (x) = V(T ∪ {x}) − V(T ).
plexity but aligns with typical NP-hard constraints, like the
maximum weighted set cover. Thus, by reducing MCP to Computing V(S ∪ {x}):
the example subset mining problem, we demonstrate its NP- 
hardness. 
 D(S) + qi ∈S ρ(qi , x)
V(S ∪ {x}) = ,
|S| + 1
3.2.4 The solution and optimality analysis 
where D(S) = qi ,q  ∈S ρ(qi , qi  ).
i
Due to the NP-hardness of this problem, we propose adopt- The marginal gain  S (x) is:
ing a greedy algorithm to efficiently approximate the optimal 
solution. The algorithm, named ESM, is illustrated in Algo- D(S) + qi ∈S ρ(qi , x) D(S)
 S (x) = − .
rithm 1. In summary, ESM first initializes an empty set and |S| + 1 |S|
then searches over questions in the training set for target ques-
tions to select the example that maximizes the combining of Similarly, for T ∪ {x}:
similarity, influence, and diversity of the example subset A. 
D(T ) + qi ∈T ρ(qi , x)
 V(T ∪ {x}) = ,
Theorem 2 The function f (A) = qi ∈A φ(qi ) + V(A) is |T | + 1
submodular. 
where D(T ) = qi ,q  ∈T ρ(qi , qi  ).
i
Proof To prove that f (A) is submodular, we need to show The marginal gain T (x) is:
that for every S ⊆ T ⊆ N and every x ∈ N \T , the following

inequality holds: D(T ) + qi ∈T ρ(qi , x) D(T )
T (x) = − .
|T | + 1 |T |
f (S ∪ {x}) − f (S) ≥ f (T ∪ {x}) − f (T ).
We need to show:

First, consider the term qi ∈A φ(qi ). This term is modular
because adding any element x to a set A results in a constant  S (x) ≥ T (x).

123
38 Page 8 of 26 S. Li et al.

Substitute the expressions for  S (x) and T (x): Contrastive learning is an apt choice because it enables
 the model to capture the relative influence of examples by
D(S) + qi ∈S ρ(qi , x) D(S) observing their interactions with the target. It reinforces
−
|S| + 1 |S| closer ties with examples exerting a stronger influence and
distances those with lesser influence. This approach is espe-
 cially effective in scenarios where explicit labeling is not
D(T ) + qi ∈T ρ(qi , x) D(T ) feasible, allowing us to deduce a latent influence score.
≥ − .
|T | + 1 |T | Considering these points, we designed a contrastive
learning-based influence learning model. In this section, we
Rearrange and simplify both sides: will provide a detailed introduction to the construction of
   training data for this influence model and its training objec-
|S| D(S) + qi ∈S ρ(qi , x) − (|S| + 1)D(S) tives.
|S|(|S| + 1) Training Data Construction for Influence Model. To
train the contrastive learning-based influence model, we need
to construct the training data in the first step. In other words,
  
we need to find a positive and negative set for each exam-
|T | D(T ) + qi ∈T ρ(qi , x) − (|T | + 1)D(T )
≥ . ple qi ∈ DTI rain , where DTI rain is the training set of the
|T |(|T | + 1) influence model. Recall that ICL paradigm is defined as
vt = g( p pr e f i x , P, (qt , st )) in Eq. 3. For each example, given
 Since S ⊆ T  , we know that D(S) ≤ D(T ) and
the NLQ qt and the schema st , we predict vt = g((qm , sm ,
qi ∈S ρ(q , x) ≤ qi ∈T ρ(qi , x). However, |S| < |T |, so
i
vm ), (qt , st )) where g is the LLM and qm is the NLQ in the
the denominator for the marginal gain in  S (x) is smaller
prompt example, sm is the corresponding database schema
compared to T (x), leading to a larger value for  S (x) under
for qm , noted that m! = t. In particular, qm is extracted from
the assumption of diminishing returns.
a set that includes the top-L similar NLQs to qt . Since we
Therefore, the marginal gain  S (x) when adding x to a
have the ground truth vˆt for (qt , st ), we can measure the per-
smaller set S is greater than or equal to the marginal gain
formance by comparing the predicted vt and ground truth vˆt .
T (x) when adding x to a larger set T , which confirms that
Then the performance will be set as the influence score of qm
V(A) is a submodular function.
on qt in the ICL paradigm for text-to-vis.
Since the sum of a submodular function and a modular
Concretely, the influence score consists of two aspects,
function is submodular, the function
i.e., the average accuracy of the predicted vˆt , and the semantic
 similarity between the predicted vt and the ground truth vˆt .
f (A) = φ(qi ) + V(A)
Formally, it can be formalized as:
qi ∈A

ω̂(qi , qt ) = λSim(vˆt , vt ) + (1 − λ)Acc(vˆt , vt ) (11)

is submodular. 

vˆt = g((qi , si , vi ), (qt , st )) (12)
As a result, we show that the greedy approach achieves a
(1 − 1/e) approximation ratio given the submodularity of the where Sim(vˆn , vn ) is measured by Euclidean distance func-
objective function. Therefore, the greedy algorithm provides tion −ρ. Acc(vˆn , vn ) is the average score of the four metrics
a guaranteed approximation bound for this problem. described in Sect. 4.1.2.
For each qn , we extract the top-M NLQs with the high-
3.2.5 Contrastive learning-based influence model est influence score and their schema to form the positive
examples, while the lowest M are the negative examples.
As mentioned previously, similarity and diversity are achieved The workflow of training data construction for the influence
through sentence vector representation and Euclidean dis- model is shown in Fig. 3.
tance calculation. However, existing methods cannot deter- Training Objective. In the influence model, we first
mine the influence of each example, which is crucial for final encode the input NLQs with a pre-trained language model
data visualization. The influence score, calculated based on BERT [7]: h = f θ (q), and then fine-tune the parameters
the ground truth, measures the reward a labeled example using the contrastive learning objective. Inspired by [13], we
brings to the visualization. Since the ground truth of the test train the influence model to follow the contrastive framework
set is unknown, we cannot compute the influence score for with a cross-entropy objective. Let h(qi ) denote the represen-
the test set using the same method as for the training sam- tation of target NLQ, qi+ and qi− denote the representations
ples. To address this, we need a model to predict the influence of positive and negative NLQs for qi , then the training objec-
scores. tive for (qi , qi+ , qi− ) within a mini-batch of M pairs is (Fig.

123
Prompt4Vis: prompting large language models with example mining… Page 9 of 26 38

Fig. 3 Training data construction for the influence model, which first ence scores of the DVQs generated by the LLMs. Finally, the positive
finds similar questions with LLMs and takes each one of them as a and negative sets are established based on influence scores
prompt example. Then, these questions are sorted based on the influ-

4): To implement the schema filtering module, we first pro-

cess the training examples, using the labeled DVQ to identify
+
ecos( f (qi ), f (qi ))/τ the gold tables used to answer the given question. Next, we
− log  (13)
cos( f (qi ), f (q +
j ))/τ
− use these preprocessed examples to construct the prompt con-
N
j=1 (e ) + ecos( f (qi ), f (q j ))/τ
)
tent and feed it into GPT−3.5 for prompting. Specifically, the
prompt content includes the task description and the prompt
where τ is a temperature hyper-parameter and cos( f (q1 ),
) f (q2 ) examples. The prompt example for schema filter consists of a
f (q2 )) denotes the cosine similarity || f f(q(q11)||·|| f (q2 )|| . question, the corresponding database schema, and the name
of the target tables. During the inference process, we con-
3.3 Schema filtering struct the prompt content for each case with 10 randomly
sampled labeled examples from the preprocessed examples.
Apart from the NLQ qt , the input in the prompt examples Specifically, an example of the prompt content is shown in
includes a database schema st consisting of a set of tables Table 2.
T , with each table tb ∈ T comprising of a set of columns c.
The gold DVQ vt for the target question mentions a subset
R(vt ) of schema elements from st . Schema elements can be
either tables or columns. In the schema filtering process, we 4 Experiments
propose to filter a subset R̂(vt ) from st covering R(vt ), i.e.,
R(vt ) ⊆ R̂(vt ) and | R̂(vt )|  |st |. In this work, we adopt a 4.1 Experimental setup
table as the basic unit in the schema filtering for two reasons.
First, extracting precise column information is challenging 4.1.1 Dataset
because sometimes there is no direct semantic connection
between the natural language description and the database We utilize NVBench [28], a public text-to-vis dataset, to
column names. Additionally, in some situations, the DVQ conduct experiments. It is initially created to evaluate text-
operations are performed across multiple column names in to-vis systems. Furthermore, to thoroughly validate the
various tables. Providing relatively coarse-grained base units effectiveness of the experiments and mitigate the impact of
can help prevent the omission of information. different database partitions, we partition the data accord-

123
38 Page 10 of 26 S. Li et al.

Table 2 Prompt content of schema filtering module

Select the related tables for generating SQL queries for each question based on the database schema.

Question: Give me a bar chart showing the total number of each minister, I want to list in asc by the total number.
Schema: Table region, columns = [*,Region_ID, Region_name, Date, Label, Format, Catalogue]
Table party, columns = [*,Party_ID, Minister, Took_office, Left_office, Region_ID, Party_name]
Table member, columns = [*,Member_ID, Member_Name, Party_ID, In_office]
Table party_events, columns = [*,Event_ID, Event_Name, Party_ID, Member_in_charge_ID]
Selected Table: Table party
...
Question: Which catalog contents has price above 700 dollars ? Show their catalog entry names and capacities, list by the
X in ascending.
Schema: Table Attribute_Definitions, columns = [*,attribute_id, attribute_name, attribute_data_type]
Table Catalog_Structure, columns = [*,catalog_level_number, catalog_id, catalog_level_name]
Table Catalog_Contents, columns = [*,catalog_entry_id, catalog_level_number, parent_entry_id, previous_entry_id,
next_entry_id, catalog_entry_name, product_stock_number, price_in_dollars, price_in_euros, price_in_pounds, capacity,
length, height, width]
Table Catalog_Contents_Additional_Attributes, columns = [*,catalog_entry_id, catalog_level_number, attribute_id,
attribute_value]
Selected Table:

Table 3 The number of NLQ and DVQ pairs under different data par- where Nvis is the number of visualization-type components
titions in the generated queries that match the ground truth queries.
Data partitions Train Dev Test Total Data Accuracy reflects the degree of match for the data
transformation components between the generated DVQ and
1 12,923 1694 1172 15,789
the ground truth query. It is formally defined as:
2 11,386 2000 2403 15,789
3 12,948 1864 977 15,789
Data Acc = Ndata /N (15)
4 8537 3676 3576 15,789
5 10,799 4130 860 15,789 where Ndata represents the number of matching data trans-
formation components between the generated DVQ and the
ground truth query.
ing to databases, performing this split five times randomly. Axis Accuracy measures the match for the x, y, and z
NVBench contains 7247 DVQs, each corresponding to sev- coordinate axis components between the generated DVQ and
eral NLQs and a specific database schema. Following the the ground truth query. The accuracy is defined as:
previous work [52], we partitioned the dataset with the train-
ing set, development set, and test set containing 98, 29, and 14 Axis Acc = Naxis /N (16)
databases, respectively. Specifically, the five data partitions
utilized in this study are detailed in Table 3. where Naxis is the number of matching x/y/z axis components
between the generated DVQ and the ground truth query.
Overall Accuracy measures the extent of exact match
4.1.2 Evaluation metrics between the generated DVQ and the ground truth query. It is
formally defined as:
Following the prior works [28, 29, 52], we use four popular
metrics to evaluate the models in the experiments including Overall Acc = Nexact /N (17)
Vis Accuracy, Axis Accuracy, Data Accuracy, and Overall
Accuracy. In the following, we introduce the details of the where Nexact represents the number of queries in the set that
evaluation metrics. completely match the ground truth query.
Vis Accuracy reflects the match between the visualization
types of the generated DVQ and the ground truth query. It is 4.1.3 Baselines
formally defined as:
We implement the proposed Prompt4Vis with three differ-
V is Acc = Nvis /N (14) ent LLM-based backbones. Then, we compare four widely

123
Prompt4Vis: prompting large language models with example mining… Page 11 of 26 38

recognized baselines. We introduce the details of the baseline sentence-transformer [45], which maps sentences and para-
models and our models in the following. graphs into a 768-dimensional dense vector and can be used
for semantic search. The version of the pre-trained sentence
• Seq2Vis [28] frames the text-to-vis task as a form transformer used in this paper is all-mpnet-base-v2.4 For the
of machine translation and employs the widely-used schema filtering module, the example number in the prompt-
sequence-to-sequence framework [1] with LSTM net- ing of schema filtering is 10.
work [16], augmented with an attention mechanism, to In the data construction process of the influence model, we
generate data visualization queries. first split NVBench into two sets B and C in terms of database,
• Transformer [55] is a more effective neural network where B and C contain 10001 and 884 pairs of natural lan-
compared to LSTM. Its core component, the self- guage questions and data visualization queries, respectively.
attention mechanism, allows it to capture the global Then we calculate the influence scores of examples in B for
context of the input sequence. The effectiveness of Trans- examples in C. The number of positive and negative exam-
former has been demonstrated in various NLP tasks, ples M is set to 5, and the number of questions in the candidate
including machine translation [59] and text summariza- set for similar questions L is set to 50. λ is determined as 0.3
tion [40]. Consequently, we have adopted Transformer by grid search. Following [13], we set τ = 0.05 in Eq. 13.
as a baseline model in our work. In particular, the influence model is fine-tuned based on the
• ncNet [29] is a neural network-based text-to-vis model version of “unsup-simcse-bert-base-uncased”.5
implemented using the Transformer architecture [55]. For fair comparisons, we use GPT−3.5-Turbo in OpenAI
Furthermore, to adapt Transformer better for the text- API 6 released in December 2023 for few-shot prompting in
to-vis task, several visualization-aware optimizations the experiments. For all the ICL-based methods, the maxi-
have been incorporated, such as attention forcing and mum output length is set to 200, while the maximum input
visualization-aware rendering. length is 3896 tokens. All baselines are tuned until their best
• RGVisNet [52] is a state-of-the-art framework for text- performance. Seq2Vis and Transformer baselines are imple-
to-vis, which first retrieves the most relevant DVQ as the mented by OpenNMT .7
prototype and then revises it to generate the desired DVQ.
• Prompt4Vis is the In-context learning-based framework 4.2 Experimental results
proposed in this work, which finds the most effective
examples for text-to-vis with an example mining mod- 4.2.1 Main results
ule and a schema filtering module. Specifically, we use
three different LLMs as the backbones for Prompt4Vis: In this study, we initially assess the proposed method against
Llama−3.1-8B-Instruct ,1 Mistral-7B-Instruct-v0.3 ,2 established baseline models utilizing specified metrics across
and GPT−3.5-Turbo .3 Accordingly, the variants of the five distinct data partitions. The aggregated scores derived
Prompt4Vis method are named Prompt4Vis-LLaMA, from these assessments are documented in Table 4. Accord-
Prompt4Vis-Mistral, and Prompt4Vis-GPT, respec- ing to the experimental results in Table 4, we have the
tively. By default, we choose GPT−3.5-Turbo as our following findings.
backbone model. Prompt4Vis outperforms all baselines on different
data partitions. Existing text-to-vis models i.e., Seq2vis
4.1.4 Implementation details [28], Transformer [55], and ncNet [29], tailored specifically
for the text-to-vis task, demonstrate robust performance in
For the example mining module, the default number K domain-specific settings, achieving up to 90% overall accu-
of labeled examples is 5. α is determined as 0.9 by grid racy as verified in previous research [28, 29]. However, our
search. In the implementation, we sort the examples with experimental data reveal a significant decline in their effec-
the score function in the training set and select the top-X tiveness within cross-domain scenarios. Despite the superior
samples as a replacement of D for reducing the searching performance of custom models like ncNet [29] and RGVis-
space in the next step. In particular, X is set to 10. What’s Net [52] over simpler seq-to-seq models, i.e., Seq2vis [28]
more, to mitigate the impact of different domains on sam- and Transformer [55], they only manage to attain an over-
ple selection in the example mining module, we masked the all accuracy between 23% and 45% as indicated in Table 4.
column names in the questions. The vector representations
for calculating the Euclidean distance are obtained through 4 [Link]
5 [Link]
1 [Link] uncased
2 [Link] 6 [Link]
3 [Link] 7 [Link]

123
38 Page 12 of 26 S. Li et al.

Table 4 Average results of the baseline models and our Prompt4Vis on the NVBench dataset with five data partitions
Method Test set Dev set
Vis Acc Axis Acc Data Acc Overall Acc Vis Acc Axis Acc Data Acc Overall Acc

Seq2Vis [28] 86.97% 0.02% 11.88% 0.01% 84.10% 0.81% 11.31% 0.32%
Transformer [55] 98.82% 0.58% 12.16% 0.42% 98.31% 2.54% 11.11% 1.71%
ncNet [29] 98.86% 41.34% 40.62% 23.61% 98.27% 37.44% 45.79% 23.97%
RGVisNet [52] 95.46% 44.80% 37.35% 30.75% 95.38% 60.06% 52.37% 44.44%
Prompt4Vis-(LLaMA) 79.37% 73.63% 53.99% 43.30% 90.09% 76.74% 62.97% 55.61%
Prompt4Vis (Mistral) 93.31% 66.82% 48.31% 40.93% 95.29% 73.68% 58.92% 53.25%
Prompt4Vis (GPT) 98.37% 79.23% 58.64% 52.69% 97.77% 79.23% 65.68% 60.39%

Table 5 Performance comparison of different models with different hardness levels

Hardness Method Test set Dev set
Vis Acc Axis Acc Data Acc Overall Acc Vis Acc Axis Acc Data Acc Overall Acc

Extra hard Seq2Vis 94.58% 0.00% 0.00% 0.00% 81.95% 0.00% 0.00% 0.00%
Transformer 100.00% 0.00% 0.00% 0.00% 99.84% 0.00% 0.00% 0.00%
ncNet 99.47% 23.49% 9.16% 1.05% 100.00% 26.49% 26.20 0.00%
RGVisNet 72.26% 43.90% 1.99% 1.66% 93.85% 35.29% 3.23% 3.07%
Prompt4Vis 98.90% 72.62% 32.67% 31.29% 91.97% 77.22% 38.31% 36.06%
Hard Seq2Vis 87.92% 0.00% 0.00% 0.00% 92.76% 0.95% 0.00% 0.00
Transformer 99.85% 0.18% 0.00% 0.00% 99.22% 2.47% 0.23% 0.14%
ncNet 99.85% 16.38% 24.21% 0.00% 98.94% 14.12% 11.09% 0.00%
RGVisNet 96.70% 30.99% 6.56% 5.06% 94.29% 46.76% 10.10% 8.19%
Prompt4Vis 95.78% 69.78% 46.46% 39.74% 94.41% 72.79% 50.25% 44.65%
Medium Seq2Vis 91.98% 0.04% 8.89% 0.04% 93.93% 1.70% 10.34% 0.97%
Transformer 99.31% 0.63% 11.77% 0.38% 98.90% 3.01% 9.62% 2.06%
ncNet 96.99% 51.27% 46.76% 41.75% 97.25% 69.25% 62.09% 57.06%
RGVisNet 96.99% 51.27% 46.76% 41.75% 97.25% 69.25% 62.09% 57.06%
Prompt4Vis 99.17% 77.89% 57.86% 51.70% 98.44% 78.65% 64.78% 58.05%
Easy Seq2Vis 80.27% 0.04% 25.01% 0.00% 82.15% 1.30% 23.77% 0.74
Transformer 96.93% 0.66% 21.23% 0.66% 97.07% 2.19% 17.04% 1.76%
ncNet 95.87% 44.61% 65.41% 38.60% 97.03% 30.44% 54.39% 27.05%
RGVisNet 91.41% 47.40% 46.48% 36.04% 93.06% 50.57% 51.26% 40.82%
Prompt4Vis 96.99% 78.38% 70.37% 61.61% 96.96% 81.63% 73.23% 68.54%

Specifically, the best results for each metric are indicated by demonstrates superior stability in cross-domain settings, as
bolded values in Table 4. evidenced by the consistently shorter box heights compared
However, as shown in Table 4, Prompt4Vis marks a to other baseline models like ncNet [29] and RGVisNet [52],
significant advancement in handling cross-domain settings, indicating minimal fluctuation across various accuracy met-
when using GPT−3.5-Turbo as the backbone, outstripping rics. Each subplot consistently positions our model in the
the contemporary state-of-the-art model RGVisNet [52] by top-right corner, underscoring its superior performance rela-
substantial margins of 71.4% and 35.9% in overall accuracy tive to baseline models.
on the test and development sets, respectively. Moreover, Prompt4Vis outperforms all baselines across different
when using different backbones, Prompt4Vis always out- hardness levels. Due to the varying hardness levels of differ-
performs the current SOTA model, RGVisNet [52], in the ent queries, we further investigate the performance of various
cross-domain setting. The results underscore the effective- text-to-vis methods across these hardness levels. Experimen-
ness of Prompt4Vis and signal a shift from conventional tal results are shown in Table 5. The results indicate that as
complex neural network designs to a more adaptive text-to- the hardness of the queries increases, the performance of all
vis approach. In addition, as shown in Figure 4, the model text-to-vis models generally declines. However, our model

123
Prompt4Vis: prompting large language models with example mining… Page 13 of 26 38

Table 6 Performance comparison of different models on weighted samples with different hardness levels
Method Test Set Dev Set
Vis Acc Axis Acc Data Acc Overall Acc Vis Acc Axis Acc Data Acc Overall Acc

Seq2Vis [28] 90.63% 0.01% 4.28% 0.01% 87.61% 0.76% 4.45% 0.27%
Transformer [55] 99.51% 0.25% 4.48% 0.14% 99.19% 1.56% 3.70% 0.63%
ncNet [29] 98.73% 29.03% 26.82% 12.63% 98.83% 31.73% 31.66% 14.12%
RGVisNet [52] 86.45% 41.85% 16.77% 14.14% 94.58% 47.05% 21.87% 19.18%
Prompt4Vis 97.83% 73.40% 45.61% 40.94% 94.50% 76.62% 50.68% 46.28%

demonstrates superior performance across samples of vary- random variant, with margins of 57.2% and 52.3% on the test
ing hardness levels. Furthermore, we assign weights of 0.4, and dev sets, respectively, underscoring the method’s effec-
0.3, 0.2, and 0.1 to samples with hardness levels of extra- tiveness beyond the mere capability of LLMs. Moreover,
hard, hard, medium, and easy, respectively, to compute each Prompt4Vis also shows superior performance compared to
model’s weighted scores based on different evaluation met- the w. sim approach, indicating that our method’s enhance-
rics. The related results are shown in Table 6. Bold fonts ments are not just due to example similarity. Notably, LLMs
indicate the best-performing results for their respective eval- prompted with random examples even underperform com-
uation metrics. pared to RGVisNet on the development set, highlighting the
importance of strategic example selection.
4.2.2 Ablation studies Additionally, experiments are conducted to verify the
efficacy of individual elements within the example mining
To gain a deeper understanding of the mechanisms underly- module, we conducted tests by removing the similarity (w/o
ing Prompt4Vis, we conducted several ablation studies to sim), influence (w/o influence), and diversity (w/o diver-
address specific research questions (RQs): sity) metrics one at a time. The results, as depicted in Table
7, reveal that omitting any of these elements leads to a decline
• RQ1: Are the improvements attributable solely to the in performance, confirming their collective importance.
capabilities of LLMs? Overall, both the capabilities of the LLM-based backbone
• RQ2: Does the schema filtering module contribute pos- and the proposed example mining strategy contribute to the
itively? outstanding performance of Prompt4Vis.
• RQ3: Does Prompt4Vis sensitive to the early errors and For RQ2: Evaluations of Prompt4Vis with different
the manual parameters? schema linking methods are conducted in this section.
• RQ4: Can the performance of Prompt4Vis be further Furthermore, we conduct case studies of the outputs of
improved by integrating text-to-SQL capabilities? Prompt4Vis with different schema linking methods.
First, we experiment by providing the complete schema
To facilitate these investigations, we employed the full for each example, denoted as w. all schemas. Furthermore,
Prompt4Vis configuration, modifying it by either omitting we replace the ICL-based schema filtering mechanism of
or substituting its components to create various variants. The Prompt4Vis with RAT-SQL [56]. It is a representative
experimental results in the ablation studies are presented in schema linking method based on a relation-aware self-
Table 7. Specifically, we set Prompt4Vis with all designed attention mechanism. Specifically, we implement it in two
components as the baseline. And the default backbone in the variants: w. rat-table provides tables extracted by RAT-SQL
ablation studies is GPT−3.5-Turbo. Then we create variants [56], and w. rat-column provides schema information down
of Prompt4Vis by removing or replacing the designed com- to the column level with RAT-SQL [56].
ponents. We introduce the details of the answers to the RQs Experimental results show that w. all schemas leads to
in the following section. a decrease in performance, with a relative drop of 4.3% on
For RQ1: We evaluate whether improvements are due the test set and 6.6% on the development set compared to
primarily to the inherent capabilities of LLMs by introduc- the full Prompt4Vis setup. It suggests that an overload of
ing a variant called w. random, which uses GPT−3.5-Turbo schema information can confuse the LLMs, detracting from
prompted with randomly selected examples from the training their ability to focus on relevant schema elements for the
set. Contrasting this, based on prior research suggesting that DVQ generation. What’s more, the performance of w. rat-
selecting similar examples enhances In-context learning [23, table and w. rat-column declines by approximately 4.2% to
65], we introduce another variant, w. sim. As illustrated in 6.1%, highlighting the effectiveness of the schema filtering
Table 7, The Prompt4Vis significantly outperforms the w. module employed in Prompt4Vis.

123
38 Page 14 of 26 S. Li et al.

Fig. 4 Performance range of different models on evaluation metrics

Table 7 Ablation study results

Method Test set Dev set
Vis Acc Axis Acc Data Acc Overall Acc Vis Acc Axis Acc Data Acc Overall Acc

Prompt4Vis (ours) 98.37% 79.23% 58.64% 52.69% 97.77% 79.23% 65.68% 60.39%
w. random 90.68% 58.67% 45.09% 33.51% 89.97% 59.67% 53.72% 39.65%
w. sim 96.96% 76.57% 57.33% 50.15% 96.48% 76.58% 63.35% 56.75%
w/o inf 97.72% 77.01% 57.64% 50.72% 97.12% 77.73% 64.33% 58.75%
w/o div 97.22% 78.61% 58.25% 51.65% 97.11% 78.36% 64.18% 58.32%
w/o sim 98.29% 79.79% 58.59% 52.16% 97.74% 80.65% 64.98% 59.40%
w. all schemas 98.20% 76.10% 57.02% 50.42% 97.60% 74.53% 63.81% 56.40%
w. rat-table 97.92% 76.77% 57.11% 50.46% 97.61% 74.99% 63.72% 56.72%
w. rat-column 98.14% 58.14% 48.86% 38.33% 97.23% 60.90% 57.65% 46.83%
w/o parameter tuning 98.22% 76.71% 57.17% 50.50% 97.83% 78.20% 62.94% 57.47%
w. Qwen+SQL 97.73% 81.10% 61.97% 56.08% 97.39% 80.47% 69.55% 64.21%
w. CoT+SQL 88.33% 68.85% 49.19% 43.37% 87.43% 69.88% 55.68% 50.92%

Table 8 The precision of the target schemas for different schema link- the LLMs often generate outputs influenced by irrelevant
ing methods tables, resulting in inaccurate or overly complex DVQs. RAT-
Schema filter method Dataset based Linking methods narrow down the schema components
Train Dev Test but often include too many irrelevant tables or columns,
leading to unnecessary operations in the generated DVQ.
rat-table 0.39 0.43 0.44
In contrast, the schema filtering module in Prompt4Vis
rat-column 0.23 0.22 0.25
effectively identifies the most pertinent tables or columns,
Our method 0.65 0.66 0.70
thereby streamlining the information provided to the LLMs
and enhancing the accuracy and relevance of the output.
Overall, the experimental results and analysis presented
Second, to have a deep understanding of the effectiveness above indicate that the schema filtering module makes a pos-
of the schema filtering module, we calculate the proportion itive contribution to Prompt4Vis.
of the target schemas in the selected schemas of different For RQ3: First, Table 8 shows that the precision of the
schema linking methods. Experimental results are shown in rat-column method is relatively low. However, as shown in
Table 8. It demonstrates that our method achieves the best Tables 4 and 7, even under these conditions, the Prompt4Vis
performance among the various schema-linking approaches, with rat-column method achieves an overall accuracy of
which we believe is a crucial reason for the effectiveness of 38.33% and 46.83% on the test and dev sets, respectively.
our method in the generation process of text-to-vis. Although the scores are lower than that achieved with
Finally, we present a comparative example in Table 9 to our schema filter method, it still outperforms RGVisNet.
give the intuitive results of different schema linking methods. Next, for the robustness of the example selection method,
The example illustrates the pitfalls of overloading LLMs with Table 7 demonstrates that using several suboptimal example
unnecessary schema details: When all schemas are provided,

123
 Table 9 An example of generated DVQ of Prompt4Vis with different schema representation methods
Schema linking method Question: Compute the total the total number across party as a pie Generated DVQ of Prompt4Vis
chart

All schemas Table election, columns = [Election_ID, Representative_ID, Date, VISUALIZE PIE SELECT Party, COUNT(*) FROM election e JOIN
Votes, Vote_Percent, Seats, Place] representative r ON e.Representative_ID = r.Representative_ID
GROUP BY Party
Table representative, columns = [Representative_ID, Name, State,
Prompt4Vis: prompting large language models with example mining…

Party, Lifespan]
Schema filtering Table representative, columns = [Representative_ID, Name, State, VISUALIZE PIE SELECT Party, COUNT(*) FROM representative
Party, Lifespan] GROUP BY Party
rat-table Table election, columns = [*,Election_ID, Representative_ID, Date, VISUALIZE PIE SELECT Party, COUNT(*) FROM election e JOIN
Votes, Vote_Percent, Seats, Place] representative r ON e.Representative_ID = r.Representative_ID
GROUP BY Party
Table representative, columns = [*,Representative_ID, Name, State,
Party, Lifespan]
rat-column [Link] VISUALIZE PIE SELECT party, COUNT(*) FROM
[Link] GROUP BY party
Page 15 of 26

123
38
38 Page 16 of 26 S. Li et al.

selection methods, such as removing any one of our three

optimization objectives still yields relatively good results.
Second, we also consider the robustness of the manual
tuning of weights and parameters (similarity, influence, and
diversity scores) during example mining. Therefore, we set
all parameters in the example mining stage to 1 and named
this variant as w/o parameter tuning. The experimental
results are shown in Table 7. It indicates that our example
mining method still performs well with default parameters
and weights set to 1 and exhibits a certain level of robustness
regarding manually configured parameters.
Overall, the above analysis indicates that our method
exhibits a certain level of robustness against early-stage error
accumulation and manual tuning of weights and parameters.
For RQ4: Although Prompt4Vis performs the best
among several models, its data accuracy is relatively low,
which impacts the overall model performance. The target
output of Prompt4Vis includes visualization type, group-
ing, and data queries, with the latter expressed via SQL. As
the text-to-SQL task is a long-established task with superior
performance, it outperforms text-to-vis. Based on this, we
explore transferring text-to-SQL capabilities to Prompt4Vis
using two methods: fine-tuning and CoT.
In the fine-tuning approach, we use Qwen2.5-Coder-14B-
Instruct [64], which performs well on Text-to-SQL, as the
base model. We fine-tune it with data formatted as <NLQ,
database schema, SQL part>, then predict the SQL parts in
the Nvbench test set and use these as additional inputs in
Prompt4Vis to enhance its performance. In the CoT-based
approach, we select the best prompts and design specific
instructions for generating the Vis Type, SQL Part, and Bin-
ning Part to ultimately generate the final DVQ. Experimental
results in Table 7 show that fine-tuning to generate SQL parts
before using Prompt4Vis improves its performance. How-
ever, a simple CoT-based approach does not significantly Fig. 5 Performance of GPT−3.5-Turbo prompted by random example
enhance the results. selection method and example mining with the different number of
In conclusion, generating high-quality SQL parts using a prompt examples.
strong model as input improves Prompt4Vis’s performance
in Text-to-Vis tasks, while simple chain-of-thought prompt-
ing is insufficient. Future work will focus on designing more of examples reaches 45, the prompt length approaches the
refined chain-of-thought methods for further enhancement. maximum limit that LLMs can accept.
Overall, Prompt4Vis is flexible and can benefit from Experimental results shown in Fig. 5 help us draw the
transferring capabilities from text-to-SQL models. following conclusions: first, the performance of ICL-based
text-to-vis methods shows a certain improvement with an
increased number of prompt examples. However, the perfor-
4.2.3 Few-shot number study mance improvements are limited, the model cannot always
achieve significant gains as the number of prompt exam-
In this section, we investigate the performance of example ples increases. Moreover, the experimental results indicate
mining in Prompt4Vis with larger context lengths. Specif- that even when the input length reaches the upper limit
ically, we conduct experiments with varying numbers of accepted by LLMs, the visualization performance achieved
prompt examples, i.e., 5, 15, 25, 35, and 45, using both the by random example selection remains significantly inferior
random example selection strategy and the example mining to our method. Additionally, the improvements of the ran-
method proposed in our work. Notably, when the number dom example selection method become slow as the number

123
Prompt4Vis: prompting large language models with example mining… Page 17 of 26 38

of prompt examples increases, suggesting that with longer on columns. For instance, in case 3 of Table 13, the question
input sequences, random example selection will still fail to requests an ascending sort based on the count of “Job_ID”.
yield ideal results. These findings further demonstrate the However, the model misinterprets this requirement and fails
importance of an effective example selection strategy in text- to apply the count function to “Job_ID”, resulting in an incor-
to-vis tasks and validate the efficacy of our proposed example rect visualization.
mining approach. Some use cases are shown in Table 10. In summary, the results of case studies indicate that our
model performs less effectively on higher-hardness exam-
4.2.4 Case studies ples that require extensive data manipulation. For samples
that require data operations, the model primarily struggles
In this section, we conduct case studies to give intuitive with mapping axis labels to the correct column names and
results of Prompt4Vis from different perspectives. applying the appropriate functions to these columns.
(1) Case studies of predictions for P ROMPT 4V IS and (3) Error case studies for Prompt4Vis. Although several
baselines. Tables 11 and 12 present two cases of the DVQs vivid cases are displayed to demonstrate further the superior-
generated by Prompt4Vis and the baselines. The corre- ity of the proposed Prompt4Vis. However, some limitations
sponding charts generated by these models are also displayed still exist in Prompt4Vis. LLMs play the backbone of this
in them. We take Table 11 as an example to show the effec- work and provides strong world knowledge. But LLMs them-
tiveness of the proposed method. As shown in Table 11, selves face the challenge of hallucination [17], which is also
Seq2Vis [28] and Transformer [55] produce wrong table reflected in Prompt4Vis. For instance, three wrong cases of
names, which leads to no image generated in Table 11a. ncNet Prompt4Vis are shown in Table 14. In case 1 and case 2,
[29] produces the DVQ with correct table “country”, how- Prompt4Vis generates wrong DVQs with extra “WHERE”
ever, it selects “COUNT(*)” from “country”, which results in clause, which are not asked in the corresponding NLQs. In
a wrong image in Table 11b. The first half of the DVQ gener- case 3, Prompt4Vis even creates the illegal grammar “CASE
ated by RGVisNet [52] is completely correct, as it can learn WHEN... THEN... ELSE... END”. These errors can be seen as
the prototype close to the gold query by retrieval and obtain hallucinations resulting from LLMs. While the researchers in
the DVQ for the current NLQ through revision. However, the community demonstrate that hallucination is inevitable
it lacks “ORDER BY SurfaceArea DESC LIMIT 5”, which [63]. Therefore, in the future, post-hoc correction methods
may be due to the rarity of this expression in the training set, [14, 31, 50] can be incorporated into the pipeline to further
not enabling the neural network to learn this pattern. Differ- improve the data visualizations generated by LLMs.
ent from the aforementioned models, Prompt4Vis is able to
accurately produce the DVQ as the same as the target query, 4.2.5 Overhead and scalability
which results in the correct charts presented in Table 11d.
(2) Case studies for Prompt4Vis on samples with the In this section, we analyze the overhead of our example
hardness of “extra hard”. As mentioned in Sect. 4.2.1, mining framework, which can be calculated through the
Prompt4Vis has poorer performance with the samples with following steps: (1) similarity and influence score calcula-
higher hardness levels. Therefore, we conduct case analysis tion: for test example, we calculate similarity and influence
on samples with a hardness level of “extra hard” to iden- scores against all training examples, leading to a time com-
tify possible reasons for the model’s poorer performance as plexity of O(Ntrain ). The space complexity is O(Ntrain )
shown in Table 13. to store the scores for examples in the training set, i.e.,
We observe that for more challenging examples, the the example pool. Ntrain is the number of training exam-
model encounters difficulties in performing coordinate- ples. (2) selection of prompting examples: for selecting
related operations in the visualization query based on its k diverse prompting examples, we apply a greedy search
understanding of the natural language question. For exam- algorithm to the training examples, resulting in time com-
ple, in case 1 of Table 13, the question requires the model to plexity of O(Ntrain log Ntrain ) + O(k · Ntrain ), where
sort the Y-axis in ascending order. However, the model fails O(Ntrain log Ntrain ) represents the time for sorting in the
to identify which column corresponds to the Y-axis, leading greedy search, and O(k · Ntrain ) corresponds to the time for
to an incorrect ascending sort operation and, subsequently, selecting diverse examples. In addition, the space complexity
an inaccurate visualization result. In case 2 of Table 13, the in this phrase is O(Ntrain 2 ) for storing the matrix of distances
question asks for descending sorting of the column corre- between any two training samples. Overall, the method has a
sponding to the X-axis, but the model again confuses the combined time complexity of O(Ntrain log Ntrain ) + O(k ·
mapping between the axis and the column name, affecting Ntrain ) and a space complexity of O(Ntrain 2 ).
the visualization. In practice, the time required to generate prompts for
Additionally, the model sometimes struggles to map rel- each target example with Prompt4Vis from different sizes
evant descriptions in the question to the required operations of training sets is shown in Fig. 6. It indicates that as the

123
38 Page 18 of 26 S. Li et al.

Table 10 Case study of the generated DVQs of Prompt4Vis with 5 and 45 prompt examples
Case 1

Question: Return a histogram on how many matches were played in each year ?, order Y in ascending order.
Ground truth: VISUALIZE BAR SELECT year, count(*) FROM matches GROUP BY year ORDER BY count(*) ASC
Predicted DVQ of P ROMPT 4V IS-5-shot: VISUALIZE BAR SELECT year, COUNT(*) FROM matches GROUP BY year ORDER
BY COUNT(*) ASC
Predicted DVQ of P ROMPT 4V IS-45-shot: VISUALIZE BAR SELECT year, COUNT(*) FROM matches GROUP BY year ORDER
BY year ASC

Case 2

Question: Please use a bar chart to show the total amount of payment by each payment method code, could you sort by the names
from high to low ?
Ground truth: VISUALIZE BAR SELECT Payment_Method_Code, SUM(Amount_Payment) FROM Payments GROUP BY
Payment_Method_Code ORDER BY Payment_Method_Code DESC
Predicted DVQ of P ROMPT 4V IS-5-shot: VISUALIZE BAR SELECT Payment_Method_Code, sum(Amount_Payment) FROM
Payments GROUP BY Payment_Method_Code ORDER BY Payment_Method_Code DESC
Predicted DVQ of P ROMPT 4V IS-45-shot: VISUALIZE BAR SELECT Payment_Method_Code, SUM(Amount_Payment) FROM
Payments GROUP BY Payment_Method_Code ORDER BY SUM(Amount_Payment) DESC

Table 11 DVQs generated by baselines and Prompt4Vis and their corresponding charts. Case 1

NL Question Return the names and surface areas of the

5 largest countries. Visualize by a pie
chart

Target DVQ VISUALIZE PIE SELECT Name,

SurfaceArea FROM country ORDER
BY SurfaceArea DESC LIMIT 5
Seq2Vis VISUALIZE BAR SELECT Surface,
COUNT(*) FROM hiring GROUP BY
Surface ORDER BY surface ASC →
Figure (a)

Transformer VISUALIZE PIE SELECT Area,

COUNT(Area) FROM appellations
GROUP BY Area → Figure (a)

ncNet VISUALIZE PIE SELECT Name,

COUNT(*) FROM country → Figure
(b)
RGVisNet VISUALIZE PIE SELECT Name,
SurfaceArea FROM country → Figure
(c)
Prompt4Vis VISUALIZE PIE SELECT Name,
SurfaceArea FROM country ORDER
BY SurfaceArea DESC LIMIT 5 →
Figure (d)

123
Prompt4Vis: prompting large language models with example mining… Page 19 of 26 38

Table 12 DVQs generated by baselines and Prompt4Vis and their corresponding charts. Case 2

NL Question Create a bar chart showing the total

number across police force, show by the
names in desc.

Target DVQ Visualize BAR SELECT Police_force,

COUNT(*) FROM
county_public_safety GROUP BY
Police_force ORDER BY Police_force
DESC
Seq2Vis VISUALIZE BAR SELECT Name,
count(Name) FROM city GROUP BY
Name ORDER BY Name DESC →
Figure (a)

ncNet VISUALIZE PIE SELECT Name,

COUNT(*) FROM country → Figure
(b)

RGVisNet VISUALIZE BAR SELECT Name,

COUNT(Name) FROM city GROUP
BY Name ORDER BY Name DESC →
Figure (c)
Prompt4Vis Visualize BAR SELECT Police_force,
COUNT(*) FROM
county_public_safety GROUP BY
Police_force ORDER BY Police_force
DESC→ Figure (d)

training dataset grows exponentially, the time required for prompt length will offer meaningful contributions. In this
sample mining increases rapidly. It reveals the limitations of section, we try two optimization methods for prompt length:
our method’s scalability. However, we believe that when the utilizing the compression method and establishing a prompt
number of samples in the training dataset is within 100,000, length optimization target. Details of the explorations are as
the computing time remains within an acceptable range. follows.
(1) Prompt Content Compression. We apply the selec-
tive context compression method proposed by Li et al. [22]
4.2.6 Explorations of optimization for prompt length to compress the prompt content. This method evaluates the
informativeness of lexical units (i.e., tokens, phrases, or sen-
Since LLMs can only handle inputs within a limited length tences) using self-information computed by a base causal
range, and in reality, NLQs and database schemas can be language model [49]. By selectively retaining content with
quite lengthy, therefore, exploring methods to optimize the

123
38 Page 20 of 26 S. Li et al.

Table 13 Case study of predicted DVQs of Prompt4Vis on extra hard samples

Case 1

Question: For all employees who have the letters D or S in their first name, show me about the distribution of job_id and the sum of
salary, and group by attribute job_id in a bar chart, and order from low to high by the y axis please.
Database schema: Table jobs, columns = [*,Job_ID, Job_Title, Min_Salary, Max_Salary]; Table employees, columns =
[*,Employee_ID, First_Name, Last_Name, Email, Phone_Number, Hire_Date, Job_ID, Salary, Commission_PCT, Manager_ID,
Department_ID]; Table job_history, columns = [*,Employee_ID, Start_Date, End_Date, Job_ID, Department_ID]
Target DVQ: VISUALIZE BAR SELECT Job_ID, SUM(Salary) FROM employees WHERE First_Name LIKE ’%d%’ OR
First_Name LIKE ’%s%’ GROUP BY Job_ID ORDER BY SUM(Salary) ASC
Predicted DVQ: of Prompt4Vis: VISUALIZE BAR SELECT Job_ID, SUM(Salary) FROM employees WHERE First_Name LIKE
’%d%’ OR First_Name LIKE ’%s%’ GROUP BY Job_ID ORDER BY Job_ID ASC

Case 2

Question: For all employees who have the letters D or S in their first name, give me the comparison about the amount of job_id over
the job_id, and group by attribute job_id, show from high to low by the X-axis please.
Database schema: Table jobs, columns = [*,Job_ID, Job_Title, Min_Salary, Max_Salary]; Table employees, columns =
[*,Employee_ID, First_Name, Last_Name, Email, Phone_Number, Hire_Date, Job_ID, Salary, Commission_PCT, Manager_ID,
Department_ID]; Table job_history, columns = [*,Employee_ID, Start_Date, End_Date, Job_ID, Department_ID]
Target DVQ: VISUALIZE BAR SELECT Job_ID, COUNT(Job_ID) FROM employees WHERE First_Name LIKE ’%d%’ OR
First_Name LIKE ’%s%’ GROUP BY Job_ID ORDER BY Job_ID DESC
Predicted DVQ: of Prompt4Vis: VISUALIZE BAR SELECT Job_ID, COUNT(Job_ID) FROM employees WHERE First_Name
LIKE ’%d%’ OR First_Name LIKE ’%s%’ GROUP BY Job_ID ORDER BY COUNT(Job_ID) DESC

Case 3

Question: For all employees who have the letters D or S in their first name, visualize a bar chart about the distribution of job_id and
the amount of job_id, and group by attribute job_id, and display the number of job id from low to high order.
Database schema: Table jobs, columns = [*,Job_ID, Job_Title, Min_Salary, Max_Salary]; Table employees, columns =
[*,Employee_ID, First_Name, Last_Name, Email, Phone_Number, Hire_Date, Job_ID, Salary, Commission_PCT, Manager_ID,
Department_ID]; Table job_history, columns = [*,Employee_ID, Start_Date, End_Date, Job_ID, Department_ID]
Target DVQ: VISUALIZE BAR SELECT Job_ID, COUNT(Job_ID) FROM employees WHERE First_Name LIKE ’%d%’ OR
First_Name LIKE ’%s%’ GROUP BY Job_ID ORDER BY COUNT(Job_ID) ASC
Predicted DVQ: of Prompt4Vis: VISUALIZE BAR SELECT Job_ID, COUNT(Job_ID) FROM employees WHERE First_Name
LIKE ’%d%’ OR First_Name LIKE ’%s%’ GROUP BY Job_ID ORDER BY Job_ID ASC

Table 14 Error cases of Prompt4Vis

Case 1

Question: Give me a bar chart for team_id of each all neutral.

Ground truth: Visualize BAR SELECT All_Neutral, Team_ID FROM basketball_match
Predicted DVQ of P ROMPT 4V IS: Visualize BAR SELECT Team_ID FROM basketball_match WHERE All_Neutral = ’neutral’

Case 2

Question: Show the relation between school id and all_games_percent for each ACC_Road using a scatter chart
Ground truth: Visualize SCATTER SELECT School_ID, All_Games_Percent FROM basketball_match GROUP BY ACC_Road
Predicted DVQ of P ROMPT 4V IS: Visualize SCATTER SELECT School_ID, All_Games_Percent FROM basketball_match
WHERE ACC_Road = 1

Case 3

Question: Show me a bar chart for what are the different affiliations, and what is the total enrollment of schools founded after 1850
for each enrollment type ?, and sort from low to high by the Y-axis please.
Ground truth: Visualize BAR SELECT Affiliation, sum(Enrollment) FROM university WHERE founded > 1850 GROUP BY
affiliation ORDER BY sum(Enrollment) ASC
Predicted DVQ of P ROMPT 4V IS: Visualize BAR SELECT Enrollment, SUM(CASE WHEN Founded > 1850 THEN 1 ELSE 0
END) FROM university GROUP BY Enrollment ORDER BY SUM(CASE WHEN Founded > 1850 THEN 1 ELSE 0 END) ASC

123
Prompt4Vis: prompting large language models with example mining… Page 21 of 26 38

i.e., NLQs, and semi-structured text, i.e., database schemas.

Semi-structured text has specific characteristics, and when
treated as plain text for deletion or compression, it can
lead to mismatches with key query elements in NLQs. Sec-
ond, the text compression method itself may be not perfect,
and its incompleteness may hide some crucial information
within the queries. Specifically, Table 16 shows a compressed
prompt with the selective compression method at a ratio of
0.2. In this example, the red font indicates text that may be
confused due to compression. For example, in the second
prompt example in Table 16, the original natural language
query required the visualization query to “show by y axis
Fig. 6 Time of example mining versus the number of training samples
from low to high”. However, the compressed text omits the
need to sort the axis and does not clearly express the sort-
ing method. Moreover, the data transformation clauses are
higher self-information, this method generates a more com- confused due to the compression, e.g., “ORDER BY” is com-
pact text representation compared to the original. pressed as “ORDER”.
Specifically, we compress the prompt content at ratios (2) Optimization of Length in Example Mining. We also
ranging from 0.1 to 0.5. Notably, our prompt consists of attempt to incorporate input length as an additional optimiza-
three parts: (a) the prompt prefix, which defines the task; tion objective. Consequently, our optimization objective can
(b) prompt example cases, which provide sample examples; be represented as follows:
and (c) the prompt target, which outlines the target output.
During the prompt compression process using the selective
context method, we do not compress the task description or
the prompt target but only compress the prompt examples. 
There are two reasons: first, we aim to maintain a clear and A∗ = arg max φ(qi )/L(qi ) + V(A) subject to |A|
A:A⊆D q ∈A
complete description of the task and the generation target; i

second, the task description and generation target occupy a ≤K (18)

smaller proportion of the prompt compared to the prompt φ(qi ) = αC(qi ) + (1 − α)I(qi ) (19)
examples.
We generate DVQs with Prompt4Vis by feeding the
compressed prompt content. Experimental results shown in
Table 15 indicate that the overall performance of Prompt4Vis
gradually decreases as the compression ratio increases, with where L(qi ) represents the length of qi .
a noticeable decline. Specifically, values in bold in Table 15 We conduct experiments using the above optimization
indicate the best results for specific metrics. objective, and the results are shown in Table 17. It indicates
By analyzing the compressed text, we identify two possi- that incorporating input length as an additional optimization
ble reasons for the performance drop: first, the underscores objective reduces prompt length but also leads to a decline in
and commas in the database schema are split apart. This is performance. However, the decline remains within an accept-
because the compression method is designed for plain text, able range. We think that when the input length becomes too
whereas in our task, the input consists of both plain text, long, this approach can be used to optimize Prompt4Vis.

Table 15 Performance of Prompt4Vis with prompts compressed by the selective compression method at different compression ratios
Compress ratio Test Set Dev Set
Vis Acc Axis Acc Data Acc Overall Acc Vis Acc Axis Acc Data Acc Overall Acc

0 98.37% 79.23% 58.64% 52.69% 97.77% 79.23% 65.68% 60.39%

0.1 97.80% 75.37% 36.92% 32.56% 97.38% 76.58% 40.91% 37.70%
0.2 97.86% 73.68% 34.76% 30.61% 97.42% 74.55% 38.54% 35.27%
0.3 97.39% 68.12% 36.99% 31.39% 97.26% 70.83% 40.97% 36.28%
0.4 96.87% 58.42% 36.84% 29.23% 96.58% 62.71% 42.19% 35.01%
0.5 95.22% 46.35% 33.42% 22.66% 94.73% 49.24% 38.72% 27.28%

123
38 Page 22 of 26 S. Li et al.

Table 16 An example of the compressed prompt content with selective compress method [22]
Prompt prefix:

/* Generate the DVQ for each question based on the database schema. */

Prompt Examples:

/* Question Please give a bar chart to show the distribution results of all music festival, and display by the Y-axis from low. */
Database schema: Table music_festival, columns = [*,ID, Music_Festival Date_of_ceremony Category Volume Result]
Answer: Visualize BAR SELECT Result, COUNT(Result FROM music_festival GROUP BY Result ORDER COUNT(Result ASC
...
/* Question Bar graph to show how many country name from different country name, and show by from low. */
Database schema: Table countries, columns = [*,COUNTRY_ID, REGION_ID
Answer: Visualize BAR SELECT COUNTRY_NAME, COUNT(COUNTRY_NAME) FROM countries ASC
/* Question Please give a bar chart to show the distribution results of all music festival, and order Y from high. */
Database schema: Table music_festival, columns = [*,ID, Music_Festival Date_of_ceremony Category Volume Result]
Answer: Visualize BAR SELECT Result, COUNT(Result) FROM music_festival GROUP Result ORDER COUNT(Result DESC

Prompt Target:

/* Question: Tell me the distribution of the number of trains at each arrival time with a bar chart, and display by the Y-axis from high to low. */
Database schema: Table train, columns = [*,Train_ID, Train_Num, Name, From, Arrival, Railway_ID]
Answer:

Table 17 Performance of Prompt4Vis with length optimization

Method Test Set Dev Set
Vis Acc Axis Acc Data Acc Overall Acc Vis Acc Axis Acc Data Acc Overall Acc

Prompt4Vis 98.37% 79.23% 58.64% 52.69% 97.77% 79.23% 65.68% 60.39%

Prompt4Vis [Link] 97.68% 74.94% 55.75% 49.33% 97.11% 77.32% 62.51% 57.09%

5 Related work What’s more, researchers in NLP and CV have found

that the examples selected of In-context learning have a
5.1 In-context learning great influence on the performance of downstream tasks [23,
65]. They have made efforts to find good examples of In-
Recently, LLMs, represented by ChatGPT and GPT-4, have context learning. For instance, Liu et al. [23] supposed that
demonstrated remarkable ability in comprehending natural good examples should be semantically similar to the target
language and have made significant advancements in a series example. Rubin et al. [46] and Zhang et al. [65] retrieve
of NLP tasks [2, 43]. In-context learning is a novel learning- examples that can maximize the performance of downstream
free paradigm that originally emerged in LLMs. Recently, tasks for In-context learning. Different from the aforemen-
it has inspired the passion of database researchers [11, 12, tioned works, we find effective examples from three different
34, 41]. For instance, Narayan et al. [34] explore the abil- dimensions for text-to-vis and conduct schema filtering to
ity of In-context learning in data cleaning and integration give clean prompts to LLMs.
tasks including error detection, entity matching, and data
transformation. Dong et al. [11] propose a zero-shot method
for text-to-SQL, named C3, which first formulates clear task
5.2 Text-to-Vis
prompts, then devises calibration techniques to address incor-
Text-to-vis has garnered significant interest from the database
rect outputs, and finally orders the large language model
and visualization communities as it enables non-experts to
to produce consistent outputs to enhance the quality of the
interact with the visualization systems through natural lan-
generated SQL. Different from the above works, our work
guage questions. Existing typical works for text-to-vis can
mainly focuses on text-to-vis and explores how to find effec-
be divided into rule-based methods and neural network-based
tive examples for In-context learning of text-to-vis.
methods. Rule-based methods represent the main trend at the
beginning of text-to-vis. For instance, a rule-based approach

123
Prompt4Vis: prompting large language models with example mining… Page 23 of 26 38

to transform textual commands into infographics is employed 6 Limitations and future work
in text-to-vis by Cui et al. [6]. Moritz et al. [33] design a set
of constraints to model the knowledge in visualization and First, as discussed in Sect. 4.2.5, while our method achieves
then optimize these constraints. DeepEye [26] is an automatic promising accuracy, it encounters scalability challenges
visualization system that employs semantic parsing tools in when handling large training sets. In the future, we will con-
NL. NL4DV [35] is also implemented based on parsing tools tinue to address this issue. For example, if the training set
and offers a Python toolkit that supports various high-level grows to a size that results in unacceptable processing time,
operations to assist users in creating NL-based DV systems. we can implement a filtering step during the preprocessing
To further promote the field of text-to-vis, Luo et al. [28] stage of the training data. Additionally, different example
create a cross-domain text-to-vis dataset named NVBench, selection methods can be applied in practical applications
based on a popular text-to-SQL benchmark. Meanwhile, with based on the user’s requirements for efficiency and accuracy.
the development of neural networks, the Seq2Vis model [28] Second, experimental results indicate that further improve-
is proposed based on the encoder-decoder framework and ment is needed for higher-hardness samples that involve more
neural networks. Inspired by the way developers reuse pre- complex data transformations. We have explored several
viously validated code snippets from code search engines methods, and experiments have shown that the data transfor-
or large codebases during software development, Song et al. mation capability of Prompt4Vis can be further enhanced
[52] introduce a novel hybrid retrieval-generation framework by transferring high-quality text-to-SQL abilities. Inspired
called RGVisNet for text-to-vis. It retrieves the most relevant by some ICL-based methods using multi-step mechanisms
DVQ candidate as a prototype from the DVQ codebase and [21], we also attempted a CoT-based contextual approach to
then refines this prototype to generate the desired DVQ. enhance performance in data transformation. However, the
Most recently, researchers have also been attracted by experimental results are suboptimal due to inference noise
large language models, which further promote the develop- introduced by the CoT methodology. In the future, we will
ment of data visualization [8, 18, 32, 57, 57, 58]. For instance, explore more alternative approaches to enhance the data
Chat2Vis system [32] makes efforts to prompt LLMs to pro- transformation capabilities of text-to-vis models.
duce Python code for data visualization. LIDA [8] is a tool Finally, Prompt4Vis relies on LLMs that impose an upper
to visualize with the assistance of large language models and limit on input length. Although we explored prompt length
image generation models, which provides a novel Python API optimization in this work, the results are not promising. Mov-
and a user interface to interact with users. Ko et al. [18] aim at ing forward, we aim to develop methods that better balance
promoting the development of visualization by enriching the prompt length and model performance in Prompt4Vis.
datasets in this field. Therefore, they propose a novel frame-
work based on LLMs to generate natural language datasets
taking Vega-Lite specifications. What’s more, Wang et al.
[58] propose to leverage LLMs as a recommendation tool 7 Conclusion
for visualization.
Several of the aforementioned methods incorporate LLMs This work is a timely study of In-context learning for text-to-
as integral components within their frameworks for vari- vis tasks in the era of LLMs. We systematically study how
ous visualization tasks, including recommendation [58] and to perform In-context learning for text-to-vis and propose a
concept binding [57]. Additionally, some approaches uti- novel framework, Prompt4Vis, which includes an example
lize LLMs to generate programming code [8, 18, 32], such mining module and a schema filtering module. Compared
as Python, specifically tailored for data visualization. How- to the previous SOTA models and methods that randomly
ever, our approach sets itself apart by proposing the use select examples for the ICL paradigm, our model achieved
of LLMs as a comprehensive pipeline to generate queries significant results, demonstrating the immense potential of
for data visualization. This method transcends the limita- In-context learning in the field of text-to-vis even data sci-
tions of using a singular declarative visualization language, ence.
providing a more expansive approach to automatic data visu- We also discover some interesting findings in this work,
alization. Concurrently, a study [20] explores In-context such as in the process of example selection for text-to-vis,
learning for visualization. This study evaluates the capability aside from the similarity between examples, the influence and
of GPT−3.5-Turbo to generate visualization specifications diversity of examples also play important roles. Moreover,
for text-to-vis. In contrast, our prompt4vis framework is for each test case, the full database schema may introduce
designed with a more precise structure, including an exam- redundant information. Hence, a good schema filtering mod-
ple mining module and a schema filtering module, leading to ule can reduce the irrelevant schema information and enhance
more effective performance than the aforementioned study. the performance. We believe that it is valuable to attempt and
practice these approaches in other data science fields as well.

123
38 Page 24 of 26 S. Li et al.

Acknowledgements Yuanfeng Song and Chen Jason Zhang are the A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A.M.,
corresponding authors. We thank the reviewers for their constructive Pillai, T.S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polo-
feedback. This work is supported by the following grants: [P0036742, zov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O.,
P0038989, P0046701, P0046703, P0040041, P0040568, P0043864, Catasta, M., Wei, J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov,
P0045948, P0046453, P0048183, P0048191, P0048566, P0048887, S., Fiedel, N.: Palm: scaling language modeling with pathways. J.
WEB24EG01-H]. Mach. Learn. Res. 24(240), 1–240 (2023)
6. Cui, W., Zhang, X., Wang, Y., Huang, H., Chen, B., Fang, L.,
Funding Open access funding provided by The Hong Kong Polytechnic Zhang, H., Lou, J., Zhang, D.: Text-to-viz: automatic generation of
University infographics from proportion-related natural language statements.
IEEE Trans. Vis. Comput. Graph. 26(1), 906–916 (2020)
7. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training
Declarations of deep bidirectional transformers for language understanding. In:
J. Burstein, C. Doran, T. Solorio (eds.) Proceedings of the 2019
Conference of the North American Chapter of the Association
Conflict of interest The authors declare that they have no conflict of for Computational Linguistics: Human Language Technologies,
interest. NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Vol-
ume 1 (Long and Short Papers), pp. 4171–4186. Association for
Open Access This article is licensed under a Creative Commons Computational Linguistics (2019)
Attribution 4.0 International License, which permits use, sharing, adap- 8. Dibia, V.: LIDA: A tool for automatic generation of grammar-
tation, distribution and reproduction in any medium or format, as agnostic visualizations and infographics using large language
long as you give appropriate credit to the original author(s) and the models. In: D. Bollegala, R. Huang, A. Ritter (eds.) Proceedings
source, provide a link to the Creative Commons licence, and indi- of the 61st Annual Meeting of the Association for Computational
cate if changes were made. The images or other third party material Linguistics: System Demonstrations, ACL 2023, Toronto, Canada,
in this article are included in the article’s Creative Commons licence, July 10-12, 2023, pp. 113–126. Association for Computational Lin-
unless indicated otherwise in a credit line to the material. If material guistics (2023)
is not included in the article’s Creative Commons licence and your 9. Dibia, V., Demiralp, C.: Data2vis: automatic generation of data
intended use is not permitted by statutory regulation or exceeds the visualizations using sequence-to-sequence recurrent neural net-
permitted use, you will need to obtain permission directly from the copy- works. IEEE Comput. Graph. Appl. 39(5), 33–46 (2019)
right holder. To view a copy of this licence, visit [Link] 10. Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X.,
[Link]/licenses/by/4.0/. Xu, J., Sui, Z.: A survey on in-context learning. arXiv preprint
arXiv:2301.00234 (2022)
11. Dong, X., Zhang, C., Ge, Y., Mao, Y., Gao, Y., Chen, L., Lin, J., Lou,
D.: C3: zero-shot text-to-sql with chatgpt. CoRR arXiv:2307.07306
References (2023)
12. Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., Zhou, J.:
1. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by Text-to-sql empowered by large language models: A benchmark
jointly learning to align and translate. In: 3rd International Con- evaluation. CoRR arXiv:2308.15363 (2023). [Link]
ference on Learning Representations, ICLR 2015, San Diego, CA, 48550/ARXIV.2308.15363. [Link]
USA, May 7-9, 2015, Conference Track Proceedings (2015) 15363
2. Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., 13. Gao, T., Yao, X., Chen, D.: Simcse: simple contrastive learning of
Lovenia, H., Ji, Z., Yu, T., Chung, W., Do, Q.V., Xu, Y., Fung, P.: A sentence embeddings. In: M. Moens, X. Huang, L. Specia, S.W. Yih
multitask, multilingual, multimodal evaluation of chatgpt on rea- (eds.) Proceedings of the 2021 Conference on Empirical Methods in
soning, hallucination, and interactivity. CoRR arXiv:2302.04023 Natural Language Processing, EMNLP 2021, Virtual Event / Punta
(2023) Cana, Dominican Republic, 7-11 November, 2021, pp. 6894–6910.
3. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhari- Association for Computational Linguistics (2021)
wal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, 14. Gero, Z., Singh, C., Cheng, H., Naumann, T., Galley, M., Gao, J.,
S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, Poon, H.: Self-verification improves few-shot clinical information
A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., extraction. CoRR arXiv:2306.00024 (2023). [Link]
Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, 48550/ARXIV.2306.00024. [Link]
S., Radford, A., Sutskever, I., Amodei, D.: Language models are 00024
few-shot learners. In: Advances in Neural Information Processing 15. Hanrahan, P.: Vizql: a language for query, analysis and visualiza-
Systems 33: Annual Conference on Neural Information Process- tion. In: S. Chaudhuri, V. Hristidis, N. Polyzotis (eds.) Proceedings
ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual of the ACM SIGMOD International Conference on Management
(2020) of Data, Chicago, Illinois, USA, June 27–29, 2006, p. 721. ACM
4. Chen, J., Chen, L., Zhu, C., Zhou, T.: How many demonstrations do (2006)
you need for in-context learning? In: Findings of the Association 16. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural
for Computational Linguistics: EMNLP 2023, pp. 11,149–11,159. Comput. 9(8), 1735–1780 (1997)
Association for Computational Linguistics, Singapore (2023) 17. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen,
5. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Q., Peng, W., Feng, X., Qin, B., Liu, T.: A survey on hallucination in
Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, large language models: principles, taxonomy, challenges, and open
S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., questions. CoRR arXiv:2311.05232 (2023). [Link]
Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., 48550/ARXIV.2311.05232. [Link]
Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur- 05232
Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., 18. Ko, H., Jeon, H., Park, G., Kim, D.H., Kim, N.W., Kim, J., Seo,
Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., J.: Natural language dataset generation framework for visualiza-
Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, tions powered by large language models. CoRR abs/2309.10245

123
Prompt4Vis: prompting large language models with example mining… Page 25 of 26 38

(2023). [Link] [Link] 2023, NeurIPS 2023, New Orleans, LA, USA, December 10–16
org/10.48550/arXiv.2309.10245 (2023)
19. Li, D., Mei, H., Shen, Y., Su, S., Zhang, W., Wang, J., Zu, M., 32. Maddigan, P., Susnjak, T.: Chat2vis: generating data visualizations
Chen, W.: Echarts: a declarative framework for rapid construction via natural language using chatgpt, codex and GPT-3 large language
of web-based visualization. Vis. Inf. 2(2), 136–146 (2018) models. IEEE Access 11, 45181–45193 (2023)
20. Li, G., Wang, X., Aodeng, G., Zheng, S., Zhang, Y., Ou, C., Wang, 33. Moritz, D., Wang, C., Nelson, G.L., Lin, H., Smith, A.M., Howe,
S., Liu, C.H.: Visualization generation with large language models: B., Heer, J.: Formalizing visualization design knowledge as con-
an evaluation. CoRR arXiv:2401.11255 (2024). [Link] straints: actionable and extensible models in Draco. IEEE Trans.
48550/ARXIV.2401.11255. [Link] Vis. Comput. Graph. 25(1), 438–448 (2019)
11255 34. Narayan, A., Chami, I., Orr, L.J., Ré, C.: Can foundation models
21. Li, P., He, Y., Yan, C., Wang, Y., Chaudhuri, S.: Auto-tables: syn- wrangle your data? Proc. VLDB Endow. 16(4), 738–746 (2022)
thesizing multi-step transformations to relationalize tables without 35. Narechania, A., Srinivasan, A., Stasko, J.T.: NL4DV: a toolkit for
using examples. Proc. VLDB Endow. 16(11), 3391–3403 (2023) generating analytic specifications for data visualization from nat-
22. Li, Y., Dong, B., Guerin, F., Lin, C.: Compressing context to ural language queries. IEEE Trans. Vis. Comput. Graph. 27(2),
enhance inference efficiency of large language models. In: Pro- 369–379 (2021)
ceedings of the 2023 Conference on Empirical Methods in Natural 36. Nguyen, T., Wong, E.: In-context example selection with influ-
Language Processing, EMNLP 2023, Singapore, December 6–10, ences. CoRR arXiv:2302.11042 (2023). [Link]
2023, pp. 6342–6353. Association for Computational Linguistics ARXIV.2302.11042. [Link]
(2023) 37. Nye, M.I., Andreassen, A.J., Gur-Ari, G., Michalewski, H., Austin,
23. Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., Chen, W.: What J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D.,
makes good in-context examples for gpt-3? In: E. Agirre, M. Apidi- Sutton, C., Odena, A.: Show your work: Scratchpads for intermedi-
anaki, I. Vulic (eds.) Proceedings of Deep Learning Inside Out: The ate computation with language models. CoRR arXiv:2112.00114
3rd Workshop on Knowledge Extraction and Integration for Deep (2021). [Link]
Learning Architectures, DeeLIO@ACL 2022, Dublin, Ireland and 38. OpenAI: Introducing chatgpt (2022). [Link]
Online, May 27, 2022, pp. 100–114. Association for Computational chatgpt
Linguistics (2022) 39. OpenAI: Gpt-4 technical report (2023)
24. Lu, Y., Bartolo, M., Moore, A., Riedel, S., Stenetorp, P.: Fan- 40. Phang, J., Zhao, Y., Liu, P.J.: Investigating efficiently extending
tastically ordered prompts and where to find them: Overcoming transformers for long input summarization. In: Proceedings of
few-shot prompt order sensitivity. In: S. Muresan, P. Nakov, the 2023 Conference on Empirical Methods in Natural Language
A. Villavicencio (eds.) Proceedings of the 60th Annual Meeting Processing, EMNLP 2023, Singapore, December 6–10, 2023, pp.
of the Association for Computational Linguistics (Volume 1: Long 3946–3961. Association for Computational Linguistics (2023)
Papers), ACL 2022, Dublin, Ireland, May 22–27, 2022, pp. 8086– 41. Pourreza, M., Rafiei, D.: DIN-SQL: decomposed in-context learn-
8098. Association for Computational Linguistics (2022) ing of text-to-sql with self-correction. CoRR arXiv:2304.11015
25. Luo, Y., Chai, C., Qin, X., Tang, N., Li, G.: Visclean: interactive (2023). [Link]
cleaning for progressive visualization. Proc. VLDB Endow. 13(12), 42. Qian, X., Rossi, R.A., Du, F., Kim, S., Koh, E., Malik, S., Lee,
2821–2824 (2020) T.Y., Chan, J.: Learning to recommend visualizations from data.
26. Luo, Y., Qin, X., Tang, N., Li, G.: Deepeye: towards automatic In: F. Zhu, B.C. Ooi, C. Miao (eds.) KDD ’21: The 27th ACM
data visualization. In: 34th IEEE International Conference on Data SIGKDD Conference on Knowledge Discovery and Data Mining,
Engineering, ICDE 2018, Paris, France, April 16–19, 2018, pp. Virtual Event, Singapore, August 14–18, 2021, pp. 1359–1369.
101–112. IEEE Computer Society (2018) ACM (2021)
27. Luo, Y., Qin, X., Tang, N., Li, G., Wang, X.: Deepeye: creating 43. Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., Yang, D.: Is
good data visualizations by keyword search. In: G. Das, C.M. chatgpt a general-purpose natural language processing task solver?
Jermaine, P.A. Bernstein (eds.) Proceedings of the 2018 Interna- In: H. Bouamor, J. Pino, K. Bali (eds.) Proceedings of the 2023 Con-
tional Conference on Management of Data, SIGMOD Conference ference on Empirical Methods in Natural Language Processing,
2018, Houston, TX, USA, June 10–15, 2018, pp. 1733–1736. ACM EMNLP 2023, Singapore, December 6–10, 2023, pp. 1339–1384.
(2018) Association for Computational Linguistics (2023)
28. Luo, Y., Tang, N., Li, G., Chai, C., Li, W., Qin, X.: Synthesiz- 44. Qin, X., Luo, Y., Tang, N., Li, G.: Making data visualization more
ing natural language to visualization (NL2VIS) benchmarks from efficient and effective: a survey. VLDB J. 29(1), 93–117 (2020)
NL2SQL benchmarks. In: G. Li, Z. Li, S. Idreos, D. Srivastava 45. Reimers, N., Gurevych, I.: Sentence-bert: sentence embeddings
(eds.) SIGMOD ’21: International Conference on Management using siamese bert-networks. In: K. Inui, J. Jiang, V. Ng, X. Wan
of Data, Virtual Event, China, June 20–25, 2021, pp. 1235–1247. (eds.) Proceedings of the 2019 Conference on Empirical Methods in
ACM (2021) Natural Language Processing and the 9th International Joint Con-
29. Luo, Y., Tang, N., Li, G., Tang, J., Chai, C., Qin, X.: Natural lan- ference on Natural Language Processing, EMNLP-IJCNLP 2019,
guage to visualization by neural machine translation. IEEE Trans. Hong Kong, China, November 3–7, 2019, pp. 3980–3990. Associ-
Vis. Comput. Graph. 28(1), 217–226 (2022) ation for Computational Linguistics (2019)
30. Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., 46. Rubin, O., Herzig, J., Berant, J.: Learning to retrieve prompts for
Apidianaki, M., Callison-Burch, C.: Faithful chain-of-thought rea- in-context learning. In: M. Carpuat, M. de Marneffe, I.V.M. Ruíz
soning. CoRR arXiv:2301.13379 (2023). [Link] (eds.) Proceedings of the 2022 Conference of the North American
ARXIV.2301.13379. [Link] Chapter of the Association for Computational Linguistics: Human
31. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegr- Language Technologies, NAACL 2022, Seattle, WA, United States,
effe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, July 10–15, 2022, pp. 2655–2671. Association for Computational
S., Majumder, B.P., Hermann, K., Welleck, S., Yazdanbakhsh, A., Linguistics (2022)
Clark, P.: Self-refine: iterative refinement with self-feedback. In: 47. Satyanarayan, A., Moritz, D., Wongsuphasawat, K., Heer, J.: Vega-
A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, S. Levine lite: a grammar of interactive graphics. IEEE Trans. Vis. Comput.
(eds.) Advances in Neural Information Processing Systems 36: Graph. 23(1), 341–350 (2017)
Annual Conference on Neural Information Processing Systems

123
38 Page 26 of 26 S. Li et al.

48. Savvides, R., Henelius, A., Oikarinen, E., Puolamäki, K.: Signifi- 58. Wang, L., Zhang, S., Wang, Y., Lim, E., Wang, Y.: Llm4vis:
cance of patterns in data visualisations. In: A. Teredesai, V. Kumar, explainable visualization recommendation using chatgpt. In:
Y. Li, R. Rosales, E. Terzi, G. Karypis (eds.) Proceedings of the M. Wang, I. Zitouni (eds.) Proceedings of the 2023 Conference
25th ACM SIGKDD International Conference on Knowledge Dis- on Empirical Methods in Natural Language Processing: EMNLP
covery & Data Mining, KDD 2019, Anchorage, AK, USA, August 2023—Industry Track, Singapore, December 6–10, 2023, pp. 675–
4–8, 2019, pp. 1509–1517. ACM (2019) 692. Association for Computational Linguistics (2023)
49. Shannon, C.E.: A mathematical theory of communication. Bell 59. Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D.F., Chao,
Syst. Tech. J. 27(3), 379–423 (1948). [Link] L.S.: Learning deep transformer models for machine translation.
1538-7305.1948.tb01338.x In: Proceedings of the 57th Conference of the Association for Com-
50. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., Yao, S.: putational Linguistics, ACL 2019, Florence, Italy, July 28- August
Reflexion: language agents with verbal reinforcement learning. In: 2, 2019, Volume 1: Long Papers, pp. 1810–1822. Association for
A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, S. Levine Computational Linguistics (2019)
(eds.) Advances in Neural Information Processing Systems 36: 60. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F.,
Annual Conference on Neural Information Processing Systems Chi, E.H., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits
2023, NeurIPS 2023, New Orleans, LA, USA, December 10–16 reasoning in large language models. In: S. Koyejo, S. Mohamed,
(2023) A. Agarwal, D. Belgrave, K. Cho, A. Oh (eds.) Advances in Neu-
51. Siddiqui, T., Kim, A., Lee, J., Karahalios, K., Parameswaran, A.G.: ral Information Processing Systems 35: Annual Conference on
Effortless data exploration with zenvisage: an expressive and inter- Neural Information Processing Systems 2022, NeurIPS 2022, New
active visual analytics system. Proc. VLDB Endow. 10(4), 457–468 Orleans, LA, USA, November 28–December 9 (2022)
(2016) 61. Wickham, H.: ggplot2—Elegant Graphics for Data Analysis. Use
52. Song, Y., Zhao, X., Wong, R.C., Jiang, D.: Rgvisnet: a hybrid R. Springer (2009)
retrieval-generation neural framework towards automatic data visu- 62. Wu, E., Battle, L., Madden, S.: The case for data visualization
alization generation. In: A. Zhang, H. Rangwala (eds.) KDD ’22: management systems. Proc. VLDB Endow. 7(10), 903–906 (2014)
The 28th ACM SIGKDD Conference on Knowledge Discovery 63. Xu, Z., Jain, S., Kankanhalli, M.S.: Hallucination is inevitable:
and Data Mining, Washington, DC, USA, August 14–18, 2022, an innate limitation of large language models. CoRR
pp. 1646–1655. ACM (2022) arXiv:2401.11817 (2024). [Link]
53. Vartak, M., Huang, S., Siddiqui, T., Madden, S., Parameswaran, 2401.11817. [Link]
A.G.: Towards visualization recommendation systems. SIGMOD 64. Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li,
Rec. 45(4), 34–39 (2016) C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang,
54. Vartak, M., Madden, S., Parameswaran, A.G., Polyzotis, N.: J., Yang, J., Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J.,
SEEDB: automatically generating query visualizations. Proc. Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni,
VLDB Endow. 7(13), 1581–1584 (2014) N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang,
55. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou,
Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you X., Ren, X., Zhang, X., Wei, X., Ren, X., Fan, Y., Yao, Y., Zhang,
need. In: Advances in Neural Information Processing Systems 30: Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., Fan, Z.: Qwen2
Annual Conference on Neural Information Processing Systems technical report. arXiv preprint arXiv:2407.10671 (2024)
2017, December 4–9, 2017, Long Beach, CA, USA, pp. 5998– 65. Zhang, Y., Zhou, K., Liu, Z.: What makes good examples for
6008 (2017) visual in-context learning? In: Advances in Neural Information
56. Wang, B., Shin, R., Liu, X., Polozov, O., Richardson, M.: RAT- Processing Systems 36: Annual Conference on Neural Informa-
SQL: relation-aware schema encoding and linking for text-to-sql tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA,
parsers. In: Proceedings of the 58th Annual Meeting of the Associ- USA, December 10–16 (2023)
ation for Computational Linguistics, ACL 2020, Online, July 5–10,
2020, pp. 7567–7578. Association for Computational Linguistics
(2020)
Publisher’s Note Springer Nature remains neutral with regard to juris-
57. Wang, C., Thompson, J., Lee, B.: Data formulator: Ai-powered
dictional claims in published maps and institutional affiliations.
concept-driven visualization authoring. IEEE Trans. Vis. Comput.
Graph. 30(1), 1128–1138 (2024)

123