InstructPipe: Building Visual Programming Pipelines in Visual

Blocks with Human Instructions Using LLMs

Zhongyi Zhou, Jing Jin, Vrushank Phadnis, Xiuxiu Yuan, Jun Jiang, Xun Qian, Jingtao Zhou, Yiyi
Huang, Zheng Xu, Yinda Zhang, Kristen Wright, Jason Mayes, Mark Sherwood, Johnny Lee, Alex
Olwal, David Kim, Ram Iyengar, Na Li, Ruofei Du†
Google Research, USA

InstructPipe
Visual Blocks
Describe the pipeline you want:
Node 1 Node 2 Node 3
Describe the emotion of a person in one Node

image using emoji and show this emoji on

the user's face in the webcam.
Selector Node 4 ... ... Node N

Tag:

Multimodal
Pseudocode
Submit Code
Writer ### Input ###

...

Generated
### Output ###

...

Pipeline ### Processor ###

Code Interpreter ...

Intermediate Data LLM Module

Figure 1: Workflow of InstructPipe. First, users describe their desired pipeline in natural language and designate it with a
language, image, or multi-modal tag. InstructPipe then feeds user instructions into a node selector to identify a relevant
set of nodes. Subsequently, both the instructions and the relevant nodes with their description are input into a code writer
to produce pseudocode. Finally, a code interpreter parses the pseudocode, rectifies errors, and compiles a JSON-formatted
pipeline, allowing users to refine and interact with it further within Visual Blocks’s node-graph editor.
ABSTRACT their learning curve, and leverage open-ended commands to spark
Visual programming has the potential of providing novice program- innovative ideas.
mers with a low-code experience to build customized processing
pipelines. Existing systems typically require users to build pipelines CCS CONCEPTS
from scratch, implying that novice users are expected to set up and • Computing methodologies → Visual analytics; Machine
link appropriate nodes from a blank workspace. In this paper, we learning; • Software and its engineering → Visual languages.
introduce InstructPipe, an AI assistant for prototyping machine
learning (ML) pipelines with text instructions. We contribute two KEYWORDS
large language model (LLM) modules and a code interpreter as Visual Programming; Large Language Models; Visual Prototyping;
part of our framework. The LLM modules generate pseudocode Node-graph Editor; Graph Compiler; Low-code Development; Deep
for a target pipeline, and the interpreter renders the pipeline in Neural Networks; Deep Learning; Visual Analytics
the node-graph editor for further human-AI collaboration. Both
ACM Reference Format:
technical and user evaluation (N=16) shows that InstructPipe
Zhongyi Zhou, Jing Jin, Vrushank Phadnis, Xiuxiu Yuan, Jun Jiang, Xun
empowers users to streamline their ML pipeline workflow, reduce
Qian, Jingtao Zhou, Yiyi Huang, Zheng Xu, Yinda Zhang, Kristen Wright,
Jason Mayes, Mark Sherwood, Johnny Lee, Alex Olwal, David Kim, Ram
∗ Corresponding author: me [at] duruofei [dot] com; Also contact: [Link] Iyengar, Na Li, Ruofei Du†. 2024. InstructPipe: Building Visual Programming
[at] [Link] Pipelines in Visual Blocks with Human Instructions Using LLMs . In CHI
’25, April 26 - May 1, 2025, Yokohama, Japan. ACM, New York, NY, USA,
23 pages. [Link]
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for third-party components of this work must be honored.
1 INTRODUCTION
For all other uses, contact the owner/author(s). A visual programming interface provides users with a node-graph
CHI ’25, April 26 – May 1, 2025, Yokohama, Japan editor to program through interaction with visual elements. As
© 2024 Copyright held by the owner/author(s).
ACM ISBN XXX-X-XXXX-XXXX-X/XX/XX. opposed to writing code in a code editor, the node graph allows
[Link] users to design pipelines by configuring nodes and connecting them
CHI ’25, April 26 – May 1, 2025, Yokohama, Japan Zhou et al.

with edges in a visual workspace. This alternative user interface users select and connect nodes, compared to building them from
approach often accelerates experimentation and exploration in scratch. This can potentially streamline the development process,
the prototyping phases of creative applications, and can make and allows users to focus on more novice-friendly interactions like
advanced technology more accessible to beginners. Advances parameter-tuning and human-in-the-loop verification. Our system
in machine learning (ML) further stimulate growing interest in evaluation with 16 participants demonstrated that InstructPipe
visual programming. Open-source ML hubs (e.g., TF-Hub [1], significantly reduced users’ workload in their creative process.
PyTorch-Hub [53], and Hugging Face [79]) contribute large Qualitative results further reveal that InstructPipe effectively
numbers of encapsulated modules that accelerate AI project supports novices’ on-boarding experience of visual programming
development and experimentation, and such libraries provide systems and allows them to easily prototype concepts for various
important resources for an ML-based visual programming platform. purposes. In our experiments, we also observed new challenges
Recent advancements in large language models (LLMs) [3, 7, 73] caused by human cognitive characteristics, and propose future
and findings on Chain-of-Thought [77] have further stimulated technical directions towards open-ended AI prototyping assistants.
community-wide interest in visual programming [4, 18, 80, 82], In summary, we contribute:
suggesting further potential in the interactive exploration of AI (1) InstructPipe, a visual programming AI assistant that enables
chains. users to generate ML pipelines from human instructions by
Despite the development of visual programming platforms in automating node selection and connection,
various domains, we observed that existing systems share one (2) System design and technical development of InstructPipe.
similar characteristic: users usually initiate a creative process in The system consists of two LLM modules and a code
the workspace “from scratch”. This implies that users need to 1) interpreter, which generate the specification for the visual
select nodes, 2) ideate the pipeline structure, and finally, 3) connect programming pipeline, compile the code, and render the
nodes within a completely empty workspace. For users unfamiliar pipeline in an interactive node-graph editor,
with a particular visual programming platform, such processes can (3) Technical and user evaluations that characterize the effec-
be overwhelming, degrading their overall programming experience. tiveness of InstructPipe, and contribute findings that reveal
Providing pipeline templates may reduce on-boarding efforts [8, 19], new challenges for the HCI community.
but the templates inherently lack flexibility and are not easily
adaptable to users’ specific needs. Similar issues also arise when 2 RELATED WORK
users write programs using text-based editors (there exist many
built-in functions in a particular programming language and 2.1 Visual Programming
multiple variables in a program), but advances in LLM assistants A computer program defines the operation of computer systems.
show that such challenges can be effectively reduced. For example, However, “the program given to a computer for solving a problem
GitHub Copilot [21] enables users to generate code by simply need not be in a written format” [69]. This future-looking statement,
describing users’ requirements in natural language. Even though the dating back to the 1960s, inspired several generations of researchers
generated code is not absolutely correct, the AI assistance usually to design and build visual programming systems.
finishes a large portion of the task, and programmers may only Today, visual programming systems (e.g., LabView [37], Unity
need to make a few edits to achieve a correct result [11, 36]. To this Graph Editor [72], PromptChainer [80], ComfyUI [12] and Visual
end, we raise the following question that motivates our work: How Blocks [17]) typically feature a node graph editor, providing
can we build visual programming assistants to accelerate design and users with a visual workspace to “write” their program using
prototyping of ML pipelines? “building blocks” [26, 64, 85]. Recent work further explored the
This paper introduces InstructPipe, a visual programming AI application of visual programming in education [8, 33, 38], XR
assistant that enables ML pipeline generation and design through creativity support [84, 87, 88], and robotics [13, 28, 29]. For
natural language instructions. InstructPipe facilitates node connec- example, Zhang et al. [88] connected the visual programming
tion and selection, allowing users to focus on more creative tasks tool to the concept of teaching by demonstration [42, 46, 93],
like parameter tuning and interactive analysis within the visual allowing users to rapidly customize AR effects in video creation.
programming workspace. We focus our AI assistant exploration FlowMatic [87] extended traditional visual programming interfaces
on ML-based pipelines, and therefore implemented InstructPipe as into 3D virtual environments, providing users with immersive
an extension to Visual Blocks [17], a visual programming system authoring experiences.
for prototyping ML pipelines. One major technical challenge in Advancements in AI has introduced many repositories of
implementing InstructPipe lies in the lack of visual programming advanced ML models [31, 62], and an increasing number of
data, making it impractical to finetune a dedicated code-LLM similar researchers are exploring AI chains [39, 82]. This progress has
to how developers build text-editor-based copilots [11, 21, 36]. We motivated HCI researchers to design and build a range of visual
addressed this issue by decomposing the generation process into programming interfaces to support the AI development process [12,
three steps (Fig. 1). InstructPipe’s first LLM module scopes the 41, 80]. For example, ChainForge is a web-based platform for
potentially useful nodes, while the second LLM module generates developers to explore various LLM-related configuration and
pseudocode for a pipeline. InstructPipe then parses the pseudocode designs in a wide range of applications [4]. Visual Blocks enables
and renders the pipeline in the node-graph editor to facilitate creation and interaction of advanced ML pipelines that can leverage
further user interaction. Our technical evaluation suggests that state-of-the-art computer vision and computer graphics models in
InstructPipe reduces the necessary user interactions by 81.1% when the browser [17].
InstructPipe CHI ’25, April 26 – May 1, 2025, Yokohama, Japan

This work contributes the technical system, implementation and 3.1 User Workflow
evaluation of a novel AI assistant that enables the use of text-based To generate a pipeline, users first click the “InstructPipe” button
instructions in visual programming of ML pipelines. Compared to in the top-right corner of the interface (Fig. 2b). The system
typical workflows in which people manually build their pipelines, then activates a simple dialog (Fig. 2a) in which users provide
InstructPipe has the potential to accelerate ML pipeline prototyping a description and a tag for their desired pipeline. The tag
in visual programming. can be “language”, “visual”, or “multimodal”, and helps guide
the pipeline generation. After users click the “Submit” button,
InstructPipe generates a visual pipeline in the node-graph editor.
2.2 Interactive Systems with LLMs
More specifically, InstructPipe generates a directed acyclic graph
The advances in LLMs bring many research directions for (DAG) of a visual programming pipeline. This implies that it uses
HCI researchers. Researchers have started designing new LLM default node parameters (e.g., the “temporature” or “max_tokens”
interfaces, to advance beyond the currently dominant chatbot value of an LLM node). Therefore, after the generation, the user
interface (e.g., OpenAI ChatGPT, Google Gemini). For example, needs to 1) finish the graphic structure if necessary, and 2) perform
Graphologue [34] augmented LLM responses with interactive parameter tuning as well as human-in-the-loop evaluation of the
diagrams that visualize response texts in a structured format. pipeline quality interactively in the visual programming platform.
Sensecape [67] provides users with a workspace to explore long As we will show in our evaluation, this new human-AI collaboration
LLM responses in a hierarchical structure. approach reduces users’ workload on the technical portion of the
Many HCI researchers integrated LLMs in conventional inter- visual programming tasks (selecting and connecting nodes), and
active systems and demonstrated that such enhanced machine thus provide a more novice-friendly experiences for technical visual
intelligence can provide new user experiences [44, 52, 56, 74]. This programming platforms.
research principle is widely applied in many downstream HCI
applications including visualization [61, 76], explainable AI [75, 81], 3.2 Primitive Nodes
and social science [43, 51]. For example, Chen et al. [10] utilized
InstructPipe supports 27 primitive nodes in Visual Blocks. We
LLMs to bridge low-level sensor information with high-level human
achieved this node library of InstructPipe by filtering out nodes
requests. Experiments showed that such connection allows users to
without explicit definition of their functions1 . For example, ‘TFLite
“construct their personalized contexts [for an intelligent system] more
model runner’ is an implicitly defined node: the user needs to input
quickly, accurately, and naturally”. To interface human intention
a tensorflow hub link to define its functionality. As we mentioned
with machine operations, researchers typically utilized LLMs by
previously, InstructPipe focuses on generating a DAG and leaves
following the ReAct (reasoning and acting) paradigm [83]. For
the parameter tuning task to users. Adding such implicit nodes
example, Park et al. [51] simulated human behaviors in an artifical
without clear definition of the functionality can easily confuse our
social system by leveraging LLMs as intelligent agents that can
AI assistant in the generation process, and thus we decide to exclude
perceiving the environment, planning their behaviors, and acting
these nodes in the node library of InstructPipe.
in the environment. Automated Visualization (AutoViz) researchers
The 27 nodes in our library include three input nodes, four output
employ LLMs for data analysis and reasoning for presenting the
nodes and 20 processor nodes. The following shows an example
visualization [45, 47, 59]. For example, LIDA features four modules
node in each category, and we leave the full node library description
in the visualization pipeline to 1) summarize a structured dataset, 2)
in Appendix A:
explore user’s goal, 3) generate code for visualization, and 4) render
visualization [15]. ChartGPT further constructs a dedicated dataset • “live camera” (an input node): Capture video stream
for chart visualization, and finetunes an LLM for fully automating through your device camera
the data visualization pipeline [70]. • “markdown viewer” (an output node): Render Mark-
InstructPipe extends the application of ReAct-like LLM frame- down strings into stylized HTML.
works [15, 70] to visual programming and demonstrates its • “imagen” (a processor node): Generate an image based
effectiveness to support rapid prototyping with lower user on a text prompt.
workload. Additionally, introducing visual programming to the We distributed 20 processor nodes based on the data type of
ReAct framework showcases an interface solution for human-AI its I/O edges and visualized it in Fig. 3. For example. “Google
collaboration. That being said, our work values partially correct AI Web Search” takes “Texts” information as input and output new
generation, though the previous literature considers it as a complete “Texts”, and “OCR” takes an image (vision-based information) as
generation failure [15, 22]. We render the imperfect results in a input and output “Texts”. “Features” in Fig. 3 indicates a wide
visual programming platform, so that even novices can intuitively range of intermediate data format used in ML pipelines, including
stand on the shoulder of the AI generation for their work. segmentation masks, a pose landmarks, URLs and etc. As shown
in the matrix, InstructPipe contains a wide range of nodes that
supports the creation of complex ML pipelines. Compared to
3 INSTRUCTPIPE related work that automates ad hoc ML inferences in specific use
InstructPipe is an AI assistant that enables users to generate scenarios [22, 68], InstructPipe makes one more step towards the
a visual programming pipeline by simply providing text-based open-ended assistants with a more diverse set of primitive nodes.
instructions. We implemented InstructPipe on Visual Blocks [17], a 1 Notethat Visual Blocks is a system that is actively being updated, and there are more
visual programming system for prototyping ML pipelines. nodes now.
CHI ’25, April 26 – May 1, 2025, Yokohama, Japan Zhou et al.

(a) InstructPipe’s instruction dialog.

(b) InstructPipe’s visual programming interface.

Figure 2: The user interface of InstructPipe. The user can first click on the “InstructPipe” button on the top-right corner of
the interface in (b). A dialog will appear, and the user can input the instruction and select a category tag. InstructPipe then
renders a pipeline on (b), in which the user can interactively explore and revise.

INPUT 4 PIPELINE GENERATION FROM

Texts Vision F e at u r e s INSTRUCTIONS
PaLM /
PaLM / 

Texts Google Web Search
/ Text Processor
PaLI / OCR Google Sheet InstructPipe leverages LLMs to generate visual programming
pipelines from text instructions. There are two prevailing ap-
OUTPUT

Body Segmentation
Vision Imagen / Google / Image Processor / Mask Visualizer /
Image Search Image Mixer / Tensor2DepthMap
Virtual Sticker proaches for LLM-customization, fine-tuning [44, 58], and few-shot
F e at u r e s PaLM
Face Landmark /
Pose Landmark /
URL2HTML /
prompting [22, 51]. Fine-tuning would require a substantial volume
Portrait Depth String Picker
of annotated data, with pairs of pipelines and prompts, and it is hard
to achieve for a specific visual programming platform. Additionally,
Figure 3: The distribution of 20 primitive processor nodes
a growing list of nodes would consistently require 1) new data
supported by InstructPipe. Note that “PaLM” represents two
annotation and 2) retraining the model, making this approach less
nodes in InstructPipe, i.e., a text generation model and a chat
sustainable. In comparison, few-shot prompting is a more practical
model of PaLM [3].
approach for prototyping an interaction concept to understand
Further extending our node library can effectively empower the the new experience it would bring to the community [22, 77, 83].
capability of our AI assistant, which we leave as critical future One major challenge of applying LLMs in visual programming
work. In the current implementation of InstructPipe, we focus on AI assistant lies in designing efficient prompts that fit within a
demonstrating its capability based on our focused node library, and reasonable number of tokens. Even though we focus our exploration
explore what new experiences this AI assistant can bring to our on 27 nodes, the node configuration file alone includes 8200 tokens.
users. Further formulating pipeline examples as in-context few-shot
examples would result in a combinatorial explosion, causing an
overwhelming number of tokens in the prompt.
InstructPipe CHI ’25, April 26 – May 1, 2025, Yokohama, Japan

task, and focuses on providing technical assistants on selecting and

connecting nodes.
In the following content, we provide detailed explanation on
the pseudocode design and implementation. As we mentioned
above, Fig. 4 provides an example of a pipeline (Fig. 4a) and its
(a) Pipeline. corresponding pseudocode (Fig. 4b). The syntax design is inspired
by TypeScript, and the overall structure is inspired by how academic
papers present pseudocode [89] in an algorithm block. In Fig. 4b,
we highlight the first line under the processor module (i.e., the
operation of the PaLI node) in different colors, representing four
different components in the programming language. “pali_1” is
the unique node ID. The green symbol after the colon, i.e., “pali”,
specifies the node type. In this example, node ID “𝑝𝑎𝑙𝑖_1” is a
“pali” node. The arguments in brackets, i.e., “image=input_image_1,
(b) pseudocode. prompt=input_text_1”, specifies the input variables (or input edges
Figure 4: A pair example of pipeline and pseudocode. In in the graph) of this node. “pali_1_out” represents the output
the first line of code under “processor”, pali_1_out, pali_1, variable name. For input nodes, the output variable name is the
pali and image=input_image_1, prompt=input_text_1 rep- same as the node id, so we do not annotate the output variable
resents output variable id, node id, node type, and node with a separate name (e.g., “input_image_1: input_image()” instead
arguments, respectively. of “input_image_1 = input_image_1: input_image()”). Note that
To this end, we implement InstructPipe with a two-stage InstructPipe generates texts (i.e., the node parameter) in the “input
LLM refinement prompting strategy, followed by a pseudocode text” node. Therefore, the argument in “text=“caption this image in
interpretation step to render a pipeline. Fig. 1 illustrates the high- detail”” does not indicate that the “input_text” node accepts input
level workflow of the InstructPipe implementation. InstructPipe edges, but accepts the node parameter input as a special case.
leverages two LLM modules (highlighted in red); a Node Selector
(§4.2), and a Code Writer (§4.3). Given a user instruction and a 4.2 Node Selector
pipeline tag, we first devise the Node Selector to identify a list of Node Selector filters out unrelated nodes by providing the LLM
potential nodes that would be used according to the instruction. In with a short description of each node. Fig. 5 shows the prompt
the Node Selector, we prompt the LLM with a very brief description we use in Node Selector. Followed by a general task description
of each node, aiming to filter out unrelated nodes for a target and guidelines, we list all node types with a short description
pipeline. The selected nodes and the original user input (the prompt that explains the function of each. Several nodes come with
and the tag) are then fed into the Code Writer, which generates recommendation(s) when the users interact with Visual Blocks,
pseudocode for the desired pipeline. In Code Writer, we provide the and we also include such node recommendations in the prompt.
LLM with detailed description and examples of each selected nodes The main intuition of this prompt design is based on how existing
to ensure the LLM has extensive context for each candidate node. open-source libraries (e.g., Numpy [23]) present a high-level
Finally, we employ a Code Interpreter to parse the pseudocode and overview of all functions2 . The documentation typically presents
render a visual programming pipeline for the user to interact with. a list of supported functions (in each category), followed by a
short description so that developers can quickly find their desired
functions. At the end of the prompt, we provide a list of Q&As as
few-shot examples to support the LLM to learn and adapt to the
4.1 Pipeline Representation
context of the task.
The Visual Blocks system takes a JSON-format data as input and
renders a directed acyclic graph (DAG) in the visual programming 4.3 Code Writer
workspace. Therefore, the ultimate goal of InstructPipe is to
With a pool of selected nodes, the Code Writer module can write
generate the JSON file; however, directly generating the long JSON
pipeline rendering pseudocode. Fig. 6 shows the structure of the
file is computationally expensive. For example, the JSON file for
prompt utilized in this LLM module. Similar to §4.2, the prompt
rendering the pipeline in Fig. 4a contains approximately 2.8k tokens.
starts with a general introduction and several guidelines. The major
To address this issue, we utilize the pseudocode representation of
difference in the prompt design in this stage lies in the granularity
a DAG, and define this token-efficient data format as the output
of each node introduction. We provide a detailed configuration for
data format of our LLM module. Fig. 4b shows the corresponding
each selected node with additional information, including 1) input
pseudocode representation of the pipeline in Fig. 4a, and the it only
data types, 2) output data types, and 3) an example, represented in
contains 123 tokens. The pseudocode representation simply stores
pseudocode, for how this node connects to other nodes. We include
the DAG information of a visual programming pipeline without
a detailed explanation of the full node configuration in Section B.1.2.
other information such as node parameters (e.g., the “max_tokens”
Similar to the previous LLM module (§4.2), the prompt design here
configuration of an LLM module) and the layouts of the nodes.
This indicates that InstructPipe leaves the task of node parameter 2 Seean example in the following link: [Link]
tuning to the user, which we believe is a more novice-friendly [Link]
CHI ’25, April 26 – May 1, 2025, Yokohama, Japan Zhou et al.

is also inspired by the documentation of existing software libraries. 5 TECHNICAL EVALUATION

Specifically, we gain inspiration from low-level function-specific InstructPipe contributes a framework for generating specifications
documentation3 , which typically includes 1) a detailed description, for visual programming pipelines based on text prompts from users.
2) data types in the input/output, followed by 3) one or more To characterize the system’s performance, we designed a technical
examples of a few lines of code for how developers can use the evaluation to asses the accuracy of the generated graphs for a
function. variety of prompts.
The prompt also includes a Q&A list as few-shot examples.
However, providing few-shot examples in this stage is non-trivial. 5.1 Data Collection
The reason lies in the dynamics of the node selection pool. A
To compute the accuracy of our generated pipelines, we need to
combination of all the nodes causes many possible options, and
collect a corpus with pairs of instructions and their corresponding
it is impossible to design a dedicated list of few-shot examples in
ground-truth pipelines. Therefore, we organized a two-day hybrid
each possible case. Therefore, we only created an example list for
workshop with 23 participants, aiming to collect real pipelines
each pipeline tag (i.e., “language”, “visual”, and “multimodal”) and
that Visual Blocks users would build for their creative usage.
intended to utilize these few-shot pipelines to teach LLMs example
The 23 participants (F: 6; M: 17) are composed of five software
use cases in each category. This implies that in-context pipelines
engineers, four research scientists, four students, three designers,
may include nodes that were not selected for the prompt. This can
two project managers, and two engineering mangers. Six attendees
potentially lead to LLM hallucinations [30], i.e., utilizing the nodes
claimed that they had prior experience in using Visual Blocks. As
that do not exist in our node library. We mitigated this issue by
this was a data collection study rather than a user study, where
adding specific prompts that explicitly show a list of supported
each participant here served as a data creator and annotator, we
nodes (i.e., the contents start with “the following is a full list of
did not restrict participation to individuals who self-identified as
...” in Fig. 6). However, LLM hallucination is a community-wise
novices. The workshop began with a 15-minute Visual Blocks
challenge, and we also find that our approach cannot completely
tutorial walking the participants through the nodes and the
eliminate this issue in visual programming. Therefore, InstructPipe
pipeline-building process. After the tutorial, attendees created
conduct a sanity check for the Code Writer outputs and directly
pipelines independently. Once they finished creating the pipelines,
disposes of the line of pseudocode with such hallucinated nodes.
participants were required to caption their pipelines and upload
This can ensure that the generated code is in a valid data format
them. We utilized this corpus of data pairs (caption/pipeline) as the
for rendering the pipeline in Visual Blocks.
data set for the technical evaluation.
The workshop was an open-ended creation process, in which
4.4 Code Interpreter participants were free to use any node available in Visual Blocks
After our LLM modules generate the pseudocode, InstructPipe with more than the 27 nodes covered by InstructPipe. The
employs a code interpreter to parse the generated pseudocode and InstructPipe feature was not available in the workshop. After the
compile a JSON-formatted pipeline with an automatic layout. Since workshop, we post-processed our collected data and achieved
we incorporated standard approaches to achieving such conversion 48 pipelines (23 language pipelines, seven visual pipelines and
from the pseudocode to the JSON file, which we do not intend 18 multi-modal pipelines) for our technical evaluations. The
to claim as our main contributions, we briefly summarize our post-processing procedure details are presented in Appendix C.1.
implementation into the following three steps for simplicity and
elaborate low-level implementation details at Appendix B.2: 5.2 Metric: The Number of User Interactions
(1) Lexical Analysis: InstructPipe first tokenizes each line of To quantify the efficacy of InstructPipe based on our goal of
the pseudocode into output variable id, node id, node type, accelerating and streamlining pipeline creation, we defined the
and node arguments (§4.1). metric Number of User Interactions as follow:
(2) Graph Generation with Default Node Parameters: We The Number of User Interactions is defined as the minimal
generated a DAG based on the tokenized results, and apply number of user interactions needed to complete the pipeline
predefined default node parameters in each generated node. from a generated pipeline.
For example, by default, the temperature and the max output
This definition is mainly inspired by Graph Edit Distance
tokens for the PaLM node are set to 0.5 and 256, respectively.
(GED) in graph theory [20]. Note that there are countless ways
If users are not satisfied with the default values, they can
to modify a generated pipeline toward a complete pipeline in
interactively adjust the parameters in the node-graph editor.
practice. Nevertheless, the minimal number of user interactions
(3) Layout Optimation: When pseudocode is converted into a
is deterministic, and this is an objective metric that can fairly
JSON file, default node parameters will cause sub-optimal
estimate the amount of effort users need to spend to achieve their
visual effects (Fig. 7a). InstructPipe conducts a layout
goal. A pipeline is considered complete when it satisfies the given
optimization process using the breadth-first search (BFS)
instruction. We calculate the number of interactions across two
algorithm, which re-arranges the layout for better presenta-
types of events: 1) adding/deleting a node, and 2) adding/deleting
tion of the pipeline (Fig. 7b).
an edge between nodes. In the technical evaluation, we report the
average ratio of user interactions required to complete a pipeline
3 See
an example in [Link]: [Link] “from our generated pipeline” compared to “from scratch” as our
[Link]#numpy-shape target metric. For example, if it takes 3 interactions to complete
InstructPipe CHI ’25, April 26 – May 1, 2025, Yokohama, Japan

Table 1: The ratio of human interactions in the technical using the Latin Square design still requires us to double the
evaluation. Results are reported as mean ± standard devia- number of participants. Additionally, novice participants are likely
tion. to progressively gain experiences within the study, and such
Overall Language Visual Multimodal learning effects will weaken the design of partial counterbalancing.
We believe that two pipelines with fully counterbalancing is a
18.9 ± 20.3% 17.4 ± 20.6% 17.6 ± 23.7% 20.8 ± 16.0% reasonable experiment setup in this work, and future work could
consider extending and scaling up these experiments.
a pipeline from our generated results and takes 10 interactions
to complete from scratch, then the ratio of interactions is 30%. 6.1.2 Pipeline selection. Given the fixed number of pipelines
Appendix C.2 contains further discussion of this metric. we can evaluate with users and the potential bias introduced by few-
shot prompts [91], it is important how we select the two pipelines
5.3 Experiment Setups and Results for user study. There are two critical factors that we considered:
We ran our generation algorithm on the pipeline captions six times representativeness and diversity. Representativeness implies that
(three times for each caption × two captions for each pipeline), and the selected pipelines should represent the average performance of
computed an averaged performance among the six trials for each InstructPipe. Diversity suggests that the selected pipelines should
pipeline. Table 1 summarizes the results of the technical evaluation. provide various experiences to simulate the actual use scenarios
Compared to building a pipeline from scratch, InstructPipe allows in which the performance of InstructPipe may vary. Following
the user to complete a pipeline with 18.9% of the user interactions, this guideline, we selected four candidates, and the final decision
demonstrating the potential of InstructPipe to require more than was made after a pilot study with one participant to test the level
5X fewer interactions. Seven generated pipelines directly satisfied of pipeline difficulty. The two resulting pipelines are composed
with instructions without user interactions in all six trials, and 38 of eight nodes with seven edges, and six nodes with six edges,
generated pipelines completed at least once in any of the six trials. respectively. Using the instructions from two authors, the averaged
ratio of human interactions in these two pipelines are 27.8% and
6 USER EVALUATION 5.2%, respectively. See Section D.2 for more detail on the pipelines.
While the technical evaluation demonstrates the accuracy of 6.2 Procedure
InstructPipe among various real pipelines created by participants,
Each study session takes 55 - 65 minutes in total. The study started
it is still unclear what is the actual user experience when real users
with 10-15 minute hands-on training for both conditions. The
go through the entire system workflow. Therefore, we conducted
training included 1) all the Visual Blocks interactions needed to
an in-person user study of InstructPipe with another group of
complete the subsequent steps of the experiment, and 2) all the
participants, aiming to provide more insights into our system
nodes that participants will need to use for pipeline creation in
performance as well as explore new user experiences brought by
the main session. Participants were also encouraged to experiment
InstructPipe. The study recruitment was in accordance with the
with building a pipeline independently, and to ask questions.
ethics board of our institution. We obtained participant consent
After the training, participants progressed to a formal study
before the study began.
session where they were asked to build and complete pipelines
under the given conditions. We verbally described the pipelines to
6.1 Study Design
participants as below, and participants could not see our scripts:
In the user evaluation, we aimed to investigate how the interface
• Text-based pipeline: get the latest news about New York
condition (with InstructPipe and without InstructPipe; the inde-
using Google Search and compile a high-level summary of
pendent variable) affects the user experience and behaviors (the
one of the results.
dependent variable). We will refer to these two interface conditions
• Real-time multimodal pipeline: create a virtual sun-
as “InstructPipe” and “Visual Blocks” in the following content. Fig. 8
glasses try-on experience using your web camera.
visualizes the complete study flow. In each condition, participants
completed the two pipelines with counterbalance (referred to as A pipeline is considered complete when the aforementioned
Task 1 and Task 2 in Fig. 8). functions run in the user’s visual programming workspace. For
We carefully designed the experiment to create a fair study that example, we consider the “real-time multi-model pipeline” as
could be completed with reasonable effort. In the following content, complete when the pipeline registers the sunglasses on the user’s
we elaborate how we made two important decisions related to the face, with real-time tracking and following of the head movement.
study’s rigor: During the task, participants were allowed to consult with us
for technical help. If participants were unable to make progress,
6.1.1 Two controlled pipelines with full counterbalancing. we provided hints. We provided much more hints in the baseline
Our user evaluation focuses on two controlled pipelines with full condition, and we made this decision to ensure every novice-level
counterbalancing. While we acknowledge that more pipelines participant can finish their tasks within a reasonable amount of time.
(e.g., four, six, or more) could enhance generalizability, such Appendix D.3 contains more details and discussion of the assistance
designs would also inevitably increase the size of the required we provided in the study. As an optional extension to the study,
user groups, even without fully counterbalancing. For example, eight participants were offered an open-ended pipeline creation,
fully counterbalancing four controlled pipelines require 12× where participants prototyped their own ideas using InstructPipe.
more participants. Partially counterbalancing with four pipelines This optional section was offered based on the progress of the
CHI ’25, April 26 – May 1, 2025, Yokohama, Japan Zhou et al.

Table 2: Task completion time and the number of human Wilcoxon signed ranks test found significant differences on both
interactions in the user study (N=16). We use ∗ ∗ ∗ to denote scales (𝑝 < .001).
𝑝 < .001. Fig. 9 further visualizes the results of users’ perceived workload
Time (secs) # Interactions in six sub-scales. The Wilcoxon signed ranks test revealed signif-
System icant differences on five sub-scales, all except “Mental Demand”
Median IQR p Median IQR p
(see §7.2 for more explanations and discussion). Furthermore, the
InstructPipe 203.5 156.25 5.0 4.25 test indicates that all participants unanimously considered that
*** ***
Visual Blocks 304.5 124.25 16.0 6.0 InstructPipe provides lower or equal workload on the subscales
of “Physical Demand”, “Temporal Demand”, “Performance” and
participant in the previous sections, and time constraints so that
“Effort” (𝑊 = 0). These quantitative results, with both objective and
the study duration was controlled within the time we guaranteed
subjective metrics, demonstrate the potential for InstructPipe to
in our recruitment process.
dramatically reduce users’ workload during the pipeline creation
After conducting all pipeline-condition combinations, par-
process.
ticipants answered open-ended questions in a semi-structured
Users’ qualitative feedback is also aligned with our quantitative
interview. The interview script is available in the appendix D.1.
results. Participants complimented that InstructPipe is “helpful”
Participants provided their general impression for each condition,
[P16] and “obviously easier (to use) than [Visual Blocks]” [P1]. P11
listed pros and cons, identified potential future use cases, and
and P6 further elaborated how InstructPipe enhances the user
critiqued the user interface for future improvements. We transcribed
experience when the user builds a visual programming pipeline:
the interviews and conducted the open coding analysis on the
qualitative data [65, 66]. More specifically, we categorized the “I feel like I can talk in natural language, and it (InstructPipe)
quotes based on our observations and then refined the code for can write the code for me.” [P11]
presentation. 6.5.2 On-boarding Support of Visual Programming. P1, P5,
and P9 explicitly mentioned that there is a “learning curve” in
6.3 Participants visual programming systems, which validates our statements and
We recruited 16 participants from our internal participant pool, motivation in §1.
which is specifically designed for UX research within our institution.
“There is a learning curve to it (using the visual programming
Importantly, none of the participants was involved in our project,
system) for sure, because you have to, like, read each node
and authors in charge of the study did not personally know any of
carefully.” [P1]
the participants. We screened participants on their self-reported
programming experience and machine learning skills. All of the P1’s comment matches with our observation of participants’
16 selected participants rated their “Programming Experience” and behaviors during the study. In the Visual Blocks condition, we
“’Machine Learning Skill’ as “Intermediate” or below (See Table 3 observed that people were more easily stuck in their creative
for a full breakdown). We intentionally recruited novice users, as purposes, which required our support. Typical support included 1)
we envision them as intended users of InstructPipe. guiding participants if they went too far away from the correct
pipeline, and 2) reminding them of an important node for the
6.4 Metrics pipeline, although we introduced all the necessary nodes in our
training session.
In addition to the qualitative data from the interview, we measured
To this end, participants commented that InstructPipe is a good
the following quantitative data.
onboarding tool in visual programming systems, especially for
6.4.1 Task Completion Time. Back-end logs were used to collect non-experts, to get familiarized with the system by having a ready
timestamps for starting and ending events. Then, completion time solution.
for each condition were calculated per task for each participant. “[InstructPipe] lets you know these nodes exist [when the pipeline
6.4.2 The Number of User Interactions. We used the number appears after the instruction]. It’s like a super speedy tutorial.”
of user interactions (introduced in §5.2) to measure the user’s [P7]
objective workload. Unlike the results in §5.3, we report an absolute “If you don’t have experience in visual programming, you will
value here because all the pipelines are controlled in the system appreciate [InstructPipe] much more ... With [InstructPipe], the
evaluation. structure is there, and I feel less worried about making mistakes.
It’s, like, giving you examples. It’s easier than starting from
6.4.3 Perceived Workload. The raw task load index (Raw-TLX) scratch.” [P5]
questionnaire was used to measure participant’s perceived work-
load [24]. This questionnaire was a subset of the NASA-TLX (part Anecdotally, three participants asked for InstructPipe during the
I). Participants filled out the questionnaire after each condition Visual Blocks condition.
(InstructPipe or Visual Blocks). 6.5.3 Integration into the Existing Workflow. InstructPipe is
a feature available in Visual Blocks. In the interviews, participants
6.5 Results particularly expressed their strong appreciation of this design as
6.5.1 InstructPipe Reduces Users’ Workload. Table 2 shows an AI assistant that enhances, instead of completely replacing, the
the results of two objective metrics measured in the study. The existing user workflow:
InstructPipe CHI ’25, April 26 – May 1, 2025, Yokohama, Japan

“[The pipeline generated by [InstructPipe] could be pretty close “[Using InstructPipe] is a little mentally demanding ... I have to
to what I want ... Or maybe sometimes not, but that’s okay. I debug ... If it doesn’t help (generating an almost 100% correct
got most of the blocks there, and then it’s up to me to figure out pipeline), I have to go through all the nodes ... I don’t like
how to connect them.” [P6] debugging.” [P13]
While most participants, like P6, appreciated the integration Additionally, we observed that several participants spent more
of the AI assistant into the standard visual programming work- time crafting their prompts than others. P15 spent the most time
flow, P15 expressed a concern about this approach. In visual writing the prompt. The following comments provide insights into
programming, users typically rely on visual thinking to construct how the prompting process caused extra mental workload:
pipelines, but the new prompt-based method introduces a shift
toward text-based reasoning. This blend of cognitive processes “I’m a relatively visual thinker ... Getting the prompt right
could potentially increase users’ mental workload: requires me to think in a way that is a lot more like precise
and like getting it figured out without working it out live ...
“ [the participant is talking about s/he wants to fix an [When writing prompts, ] you’re just putting them (every detail
unsuccessful generation by changing the prompt instead of in a whole pipeline) all out [in one short prompt]” [P15]
performing visual programming here] ... because I just spent so
much time figuring out what the prompt should be. That’s kind of In addition to the lack of the original visual thinking experience
like already where my brain was and I knew that something in visual programming, P13 also warned that such simplification of
was wrong there (the prompt), but I would have to switch over the creative process into prompting experience may sacrifice users’
to the other mode (visual programming) of figuring out what hands-on experiences:
was wrong in the pipeline ... [this is very overwhelming]” [P15] “I’m very hands-on with techs. I would like to understand what’s
going on [rather than prompting LLMs to generate everything
6.5.4 Use Scenarios: Accessible ML Prototyping and Educa-
for me]. I want to like think for myself and then compile all the
tion. In the open-ended session, we observed that participants
information myself.” [P13]
could efficiently utilize InstructPipe to prototype a pipeline for
various daily life or business purposes. For example, P14 tried
InstructPipe with “summarize real estate price increase in San Diego 7 DISCUSSION
California over 2023”. Compared to using LLM chatbots, InstructPipe 7.1 Human-AI Collaboration in Prototyping
helps the user quickly build a more explainable pipeline in which the Open-ended ML Pipelines
user can track (or modify) the information resources. P4 prototyped
Our technical evaluation (§5.3) shows that InstructPipe reduces the
an interactive VQA app by “Describe the product in the camera”.
number of user interactions to 18.9 % (±20.3%) There are two key
P13 further shared his thoughts on how this rapid and accessible
implications from the results:
prototyping experience can support future business:
“It (InstructPipe) is going to facilitate prototype building for PMs • InstructPipe automates most pipeline components with a
(Product Managers) ... I have lots of ideas, but my challenge single prompt.
is how to translate an idea into the technical world and see a • InstructPipe is not able to automate all the pipeline creation
prototype. I think that this app expedites me in that process a processes.
lot.” [P13] Such takeaways differ from early-stage findings that show LLMs
Another emerging theme was regarding educating kids on can achieve full automation of ML inference [22, 68]. The main
programming: reason is that existing work built their ad hoc solutions for target
use scenarios, respectively. In contrast, InstructPipe covers a larger
“With [InstructPipe], I don’t need to teach them (kids) to code range of ML models (§3.2) and aims for an open-ended use case. Our
for them to build something ... Some kids like to code, some kids results show that LLMs (we used GPT-3.5-turbo in the study) still
like to create stuff but don’t want to be bored with learning the fail to write robust code with prompt engineering techniques. This
syntax of coding ... Using [InstructPipe], I can see kids can build, aligns with the latest research findings that shows that even the
like, customized chat-bots or interactive Wikipedia.” [P13] latest LLMs still fail to formulate a whole working pipeline [58, 78].
While LLMs cannot generate a fully executable pipeline, our
6.5.5 Limitations and Future Directions. Across the study
work shows that AI can successfully render a certain portion
sessions, we consistently observed a specific user behavior pattern:
of a pipeline for users. Both technical and user evaluations
participants typically paused their pace when a generated pipeline
highlight the important values here. We believe such values provide
appeared in the workspace. At these times, some participants used
useful takeaways for HCI researchers to explore more human-AI
soliloquy, as in saying “Let me see”, while others kept a focused stare
collaboration approaches and designs in visual programming.
on the workspace. These human behaviors suggest that InstructPipe
led participants to engage in deeper, contemplative thought.
The observation suggests that participants needed time to 7.2 Three Attributes to Mental Workload
perceive the generated pipelines as they appeared in the workspace. Results in §6.5.1 show that InstructPipe failed to significantly reduce
Such sense-making processes bring new challenges to the creative novice users’ mental demand. We summarized its major causes into
process: three aspects.
CHI ’25, April 26 – May 1, 2025, Yokohama, Japan Zhou et al.

7.2.1 Instruction. P15’s comment in §6.5.5 summarizes the first (Fig. 10b). The user only needs to turn “Imagen” into another
aspect that causes mental burden. Although the “instruction-to- mode so that it also accepts the input “image” node. Revisiting
pipeline” process is fast and seems effortless, the process of framing the improved instruction, we instructed InstructPipe with “generate
a prompt is one factor that may overwhelm users, especially those a cat image based on THE updated caption”, which actually missed
who are more accustomed to visual thinking. InstructPipe requires the input image.
its users to 1) be clear about the problem they want to solve, and 2) The important takeaway is while natural languages are proven
be able to clearly articulate the desired pipeline. Such requirements to be one promising communication media that connects humans
cause a mental burden to the user. We believe that our results can and AI systems [10, 74], instructions may not be the best format to
reinforce Zamfirescu-Pereira et al.’s findings on how non-experts facilitate such connection. We believe the reason is that instructions
may not prompt LLMs well [86] in the visual programming domain. are still not intuitive to humans: AI typically requires flawless
and unambiguous instructions while humans tend to express their
7.2.2 Perception. The integration of LLM support into the
intention using ambiguous natural languages in conversations.
visual programming interface enables a “multimodal programming”
We encourage future work to investigate alternative interaction
experience [16], in which, users can program through both
mediums beyond instructions to further enhance user experience
verbal and visual approaches. However, this flexibility increases
in human-AI collaboration.
perceptual burden as users switch between visual and text-based
thinking [49]. Interestingly, our results seemingly contradict
psychological findings based on the Dual Coding Theory (DCT) that 8 LIMITATIONS AND FUTURE WORK
show a combination of verbal and visual information actually helps 8.1 Assisting Humans to Prompt AI Copilot in
humans’ memory process4 [48, 50]. Therefore, we believe that the Visual Programming
mental workload stems not from dislike of multimodal workspaces,
InstructPipe introduces a novel user interaction technique for
but from the lack of a transparent interface that aligns users’ mental
visual programming, along with its set of challenges – prompting
models with AI reasoning both verbally and visually. That being
AI is not easy [86]. While latest research has explored prompt
said, a next-generation copilot should visualize a pipeline (i.e., visual
writing assistants [6, 40], creating such assistants in visual
info) while the user is prompting the system (i.e., verbal info),
programming poses unique challenges, as discussed in §7.2, and
constituting and interfacing multimodal processing in humans’
requires further dedicated investigation. Despite these challenges,
brain.
the visual programming workspace offers unique opportunity
7.2.3 Debugging. When a rendered pipeline does not match – it provides an interactive and visual medium for delivering
users’ expectations, users need to debug (see P13’s comment AI-generated information. For example, a prompt writing assistant
§6.5.5). Specifically, users need to “invest extra effort to review and could provide “a pipeline preview” in real time via a lightweight
understand the generated content” [90] and then solve the issues LLM. Visualizing estimated outcomes, such as unexpected pipeline
caused by the AI assistant. In essence, debugging is a professional results (as illustrated in Fig. 10a), highlights the potential of these
programming skill, which understandably can be mentally over- tools to guide users in refining their instructions effectively.
whelming for beginner-level users. While InstructPipe visualizes
generated code in the visual programming platform, our results 8.2 Node Parameter Tuning
suggests that future systems should provide more guidance for
InstructPipe focuses on generating the graph structure in the
beginners to better proceed with their programming tasks.
pipeline (§4.1), and InstructPipe is not able to generate node
parameters. Latest research in AI agents show great potentials
7.3 Instructing LLMs Poses Challenges for of distributing a systematic task among multiple LLMs and let them
Both Novices and, Potentially, Experts solve the problem collaboratively [35]. We encourage future work
As we discussed above, non-experts found it challenging to instruct to extend such distributed AI agent approaches to generate suitable
LLM. More interestingly, we found that even we, the inventors of node parameters to further reduce users’ workload in tuning them.
InstructPipe, failed to write optimal instructions. For instance, the
two captions of Fig. 13c are “Describe the image and turn it into a cat 8.3 A Larger and Dynamic Node Library
image” and “Edit an image by updating the image caption”. Neither
InstructPipe is a AI assistant prototype on a small scale library
caption explicitly describes the detailed pipeline flow clearly, and
with 27 nodes. Similar to other tool-calling LLM systems [14, 60],
therefore, all the six evaluation trials (§5) were incomplete (see
InstructPipe cannot generate any out-of-scope node, and thus there
Fig. 10a for one example). The average ratio of user interactions
is a limited scope of pipelines that InstructPipe can generate. Future
is 45.8%, more than twice the average value for our multimodal
work should investigate a scale-up problem by creating an assistant
pipelines (20.8%). To further understand the cause of the failure,
that support a large-scale nodes [57]. What new technical challenge
another author improved the instruction into “Caption a tiger image
will emerge? Will such a large-scale library provide practical human
using VQA, modify the character in the caption into a cat using LLM,
value? If yes, what are the concrete new user experience it opens
and finally generate a cat image based on the updated caption”. The
up in visual programming?
resulting pipeline is significantly improved but still not perfect
Additionally, future work should explore a dynamic solution of
4 Forexample, people feel it easier to remember a new word if they learn the word the node library, in which an AI assistant can define necessary
using a vocabulary card with a figure that explains the texts. nodes in visual programming on the fly. HuggingGPT [62] is
InstructPipe CHI ’25, April 26 – May 1, 2025, Yokohama, Japan

a pioneering project that shares similar vision as this goal, but 8.6 Responsible AI
existing investigations show that the accuracy of such open-ended InstructPipe currently cannot detect harmful data or misuse of AI.
generation is still unsatisfactory [54, 58]. How can we design an We believe such safety features are crucial, especially in the context
interface to bridge such imperfect AI and human users in visual of the potential for future dynamic node libraries, which would
programming copilot? What will be the interaction paradigm in an greatly enhance the generalizability of ML pipeline prototyping
interactive system with a dynamic node library? capability. Future work must study effective methods to eliminate
potential harmful uses when AI assistants become increasingly
open-ended.
8.4 Refining System Component Design
9 CONCLUSION
InstructPipe provides a system contribution, and we verified
the usefulness of InstructPipe via two evaluations that assess This paper introduces InstructPipe, an AI assistant that empowers
InstructPipe as a whole system. One important future direction users to accelerate their design of ML visual programming pipelines
would be to verify (or even challenge) each technical component of using text instructions. We design and implement InstructPipe by
our system, as elaborated below: decomposing the task into three modules: a node selection module, a
Pseudocode. We designed the pseudocode order based on how code writer, and a code compiler. Results in our technical and system
algorithm papers present their algorithm blocks. Is this design the evaluations suggest that InstructPipe provides users’ satisfactory
best approach among all the possible candidates? If not, how can “on-boarding” experience of visual programming systems and
we further improve the design of pseudocode language? allows them to rapidly prototype an idea, potentially with more
Prompt Design. We leveraged in-context learning capability than 5X fewer interactions. We further discuss the issues we
of LLMs in our prompt design. Prior work shows that few-shot observed concerning LLMs in visual programming, related to
examples cause bias effects in practice [92], and thus we encourage both human factors and technical implementations. We hope
future work to mitigate this bias by collecting a large dataset and that InstructPipe will inspire the community to continue work
finetuning LLM on the dataset. in accelerated human-AI collaboration for increased expressivity
Divide-and-conquer at Scale. We adopt the strategy of and creativity, for machine learning pipelines, and beyond.
divide-and-conquer [63] with a two-stage LLM pipeline. Despite (8,101 words)
its effectiveness with a small node library and simple graphs, its
effectiveness is unknown when generating complex graphs. Ex- REFERENCES
ploring agent-based approaches [27, 32] with Retrieval Augmented [1] 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.
[Link] Software available from [Link].
Generation (RAG) would be a promising future direction to manage [2] Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and
complex graph generation in a divide-and-conquer manner [63]. Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem
solving with operation-based formalisms. arXiv preprint arXiv:1905.13319 (2019).
We encourage future work to contribute high-quality datasets as [3] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin,
well as an interactive LLM system with RAG that provides users Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng
with better experiences from AI agents. Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang,
Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin
Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang,
Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha,
James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng,
8.5 Evaluation Metrics and Long-term Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément
Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz,
Evaluation Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus
Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven
In the technical evaluation, we assessed the performance of AI Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy
assistants based on the number of user interactions. Existing related Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen
metrics, predominantly from the code synthesis literature [2, 25], Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin
Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao
largely focuses on two categories: correctness-based metrics [5, 9] Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua
that rely on test case verification, and similarity-based metrics [71]. Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric
Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov,
Very little research falls outside these two categories [55]. Our metric Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro
incorporates human factors by objectively estimating user effort Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone,
through graph theory, addressing a gap in the visual programming Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay
Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang,
literature where human-centric considerations are crucial. While John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin,
our work advances metric development in this domain, further Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav
formal research is essential to establish comprehensive standards Petrov, and Yonghui Wu. 2023. PaLM 2 Technical Report. arXiv:2305.10403 [[Link]]
[4] Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and
for visual programming evaluation. Elena L. Glassman. 2024. ChainForge: A Visual Toolkit for Prompt Engineering
In the user evaluation, we conducted a lab study to understand and LLM Hypothesis Testing. In Proceedings of the CHI Conference on Human
Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for
the user experience of InstructPipe. As future work, we plan to Computing Machinery, New York, NY, USA, Article 304, 18 pages. https:
conduct longer-term studies and gather more realistic insights than //[Link]/10.1145/3613904.3642016
those we obtained from the lab study. This is critical for us to [5] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk
Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le,
understand the long-term usefulness of our assistant for beginners, et al. 2021. Program synthesis with large language models. arXiv preprint
as well as collecting feedback to inform our system design. arXiv:2108.07732 (2021).
CHI ’25, April 26 – May 1, 2025, Yokohama, Japan Zhou et al.

[6] Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. arXiv.2211.11559
2023. Promptify: Text-to-Image Generation through Interactive Prompt [23] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers,
Exploration with Large Language Models. In Proceedings of the 36th Annual Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg,
ACM Symposium on User Interface Software and Technology (San Francisco, CA, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van
USA) (UIST ’23). Association for Computing Machinery, New York, NY, USA, Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe,
Article 96, 14 pages. [Link] Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. 2020.
Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Array programming with NumPy. Nature 585, 7825 (Sept. 2020), 357–362. https:
Askell, et al. 2020. Language Models are Few-shot Learners. Advances in Neural //[Link]/10.1038/s41586-020-2649-2
Information Processing Systems 33 (2020), 1877–1901. [Link] [24] Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX
arXiv.2005.14165 (Task Load Index): Results of empirical and theoretical research. In Advances in
[8] Liuqing Chen, Shuhong Xiao, Yunnong Chen, Yaxuan Song, Ruoyu Wu, and psychology. Vol. 52. Elsevier, 139–183.
Lingyun Sun. 2024. ChatScratch: An AI-Augmented System Toward Autonomous [25] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora,
Visual Programming Learning for Children Aged 6-12. In Proceedings of the CHI Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al.
Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI 2021. Measuring coding challenge competence with apps. arXiv preprint
’24). Association for Computing Machinery, New York, NY, USA, Article 649, arXiv:2105.09938 (2021).
19 pages. [Link] [26] Thomas T Hewett. 2005. Informing the design of computer-based environments
[9] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde to support creativity. International Journal of Human-Computer Studies 63, 4-5
De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, (2005), 383–409.
Greg Brockman, et al. 2021. Evaluating large language models trained on code. [27] Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui.
arXiv preprint arXiv:2107.03374 (2021). 2023. Agentcoder: Multi-agent-based code generation with iterative testing
[10] Weihao Chen, Chun Yu, Huadong Wang, Zheng Wang, Lichen Yang, Yukun Wang, and optimisation. arXiv preprint arXiv:2312.13010 (2023).
Weinan Shi, and Yuanchun Shi. 2023. From Gap to Synergy: Enhancing Contextual [28] Justin Huang and Maya Cakmak. 2017. Code3: A system for end-to-end
Understanding through Human-Machine Collaboration in Personalized Systems. programming of mobile manipulator robots for novices and experts. In
In Proceedings of the 36th Annual ACM Symposium on User Interface Software Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot
and Technology (San Francisco, CA, USA) (UIST ’23). Association for Computing Interaction. 453–462.
Machinery, New York, NY, USA, Article 110, 15 pages. [Link] [29] Justin Huang, Tessa Lau, and Maya Cakmak. 2016. Design and evaluation of a
3586183.3606741 rapid programming system for service robots. In 2016 11th ACM/IEEE International
[11] Shrestha Basu Mallick Chris Perry. 2023. AI-powered coding, free of charge with Conference on Human-Robot Interaction (HRI). IEEE, 295–302.
Colab. Retrieved Sep 10, 2024 from [Link] [30] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian
google-colab-ai-coding-features/ Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting
[12] ComfyUI. 2023. ComfyUI. [Link] Liu. 2023. A Survey on Hallucination in Large Language Models: Principles,
[13] Chandan Datta, Chandimal Jayawardena, I Han Kuo, and Bruce A MacDonald. Taxonomy, Challenges, and Open Questions. arXiv:2311.05232 [[Link]] https:
2012. RoboStudio: A visual programming environment for rapid authoring //[Link]/abs/2311.05232
and customization of complex services on a personal service robot. In 2012 [31] HuggingFace. 2022. Spaces. [Link]
IEEE/RSJ International Conference on Intelligent Robots and Systems. 2352–2357. preprocessing
[Link] [32] Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024.
[14] Fernanda De La Torre, Cathy Mengying Fang, Han Huang, Andrzej Banburski- MapCoder: Multi-Agent Code Generation for Competitive Problem Solving. arXiv
Fahey, Judith Amores Fernandez, and Jaron Lanier. 2024. LLMR: Real-time preprint arXiv:2405.11403 (2024).
Prompting of Interactive Worlds using Large Language Models. In Proceedings [33] Peiling Jiang. 2023. Positional Control in Node-Based Programming. In Extended
of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems
HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, (Hamburg, Germany) (CHI EA ’23). Association for Computing Machinery, New
Article 600, 22 pages. [Link] York, NY, USA, Article 231, 7 pages. [Link]
[15] Victor Dibia. 2023. LIDA: A Tool for Automatic Generation of Grammar- [34] Peiling Jiang, Jude Rayan, Steven P Dow, and Haijun Xia. 2023. Graphologue:
Agnostic Visualizations and Infographics using Large Language Models. Exploring Large Language Model Responses with Interactive Diagrams. arXiv
arXiv:2303.02927 [[Link]] [Link] preprint arXiv:2305.11473 (2023).
[16] Griffin Dietz, Nadin Tamer, Carina Ly, Jimmy K Le, and James A. Landay. 2023. [35] Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. 2024.
Visual StoryCoder: A Multimodal Programming Environment for Children’s From LLMs to LLM-based Agents for Software Engineering: A Survey of Current,
Creation of Stories. In Proceedings of the 2023 CHI Conference on Human Factors in Challenges and Future. arXiv:2408.02479 [[Link]] [Link]
Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing [36] Jupyter-ai. 2024. Jupyter AI. Retrieved Sep 10, 2024 from [Link]
Machinery, New York, NY, USA, Article 96, 16 pages. [Link] jupyterlab/jupyter-ai
3544548.3580981 [37] Jeffrey Kodosky. 2020. LabVIEW. Proc. ACM Program. Lang. 4, HOPL, Article 78
[17] Ruofei Du, Na Li, Jing Jin, Michelle Carney, Scott Miles, Maria Kleiner, (jun 2020), 54 pages. [Link]
Xiuxiu Yuan, Yinda Zhang, Anuva Kulkarni, Xingyu Liu, Ahmed Sabie, Sergio [38] Anastasia Kovalkov, Avi Segal, and Kobi Gal. 2020. Inferring Creativity in Visual
Orts-Escolano, Abhishek Kar, Ping Yu, Ram Iyengar, Adarsh Kowdle, and Alex Programming Environments. In Proceedings of the Seventh ACM Conference on
Olwal. 2023. Rapsai: Accelerating Machine Learning Prototyping of Multimedia Learning @ Scale (Virtual Event, USA) (L@S ’20). Association for Computing
Applications Through Visual Programming. In Proceedings of the 2023 CHI Machinery, New York, NY, USA, 269–272. [Link]
Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI 3406725
’23). Association for Computing Machinery, New York, NY, USA, Article 125, [39] LangChain. 2023. LangChain. [Link]
23 pages. [Link] [40] LangChain. 2024. Promptim: an experimental library for prompt optimization.
[18] Ruofei Du, Na Li, Jing Jin, Michelle Carney, Xiuxiu Yuan, Kristen Wright, Mark Retrieved Nov 26, 2024 from [Link]
Sherwood, Jason Mayes, Lin Chen, Jun Jiang, Jingtao Zhou, Zhongyi Zhou, Ping [41] LangFlow. 2023. LangFlow. [Link]
Yu, Adarsh Kowdle, Ram Iyengar, and Alex Olwal. 2023. Experiencing Visual [42] Yang Li and James A. Landay. 2005. Informal Prototyping of Continuous
Blocks for ML: Visual Prototyping of AI Pipelines. In Adjunct Proceedings of the Graphical Interactions by Demonstration. In Proceedings of the 18th Annual
33rd Annual ACM Symposium on User Interface Software and Technology (UIST). ACM Symposium on User Interface Software and Technology (Seattle, WA, USA)
ACM. [Link] (UIST ’05). Association for Computing Machinery, New York, NY, USA, 221–230.
[19] James Fogarty, Jodi Forlizzi, and Scott E. Hudson. 2001. Aesthetic information [Link]
collages: generating decorative displays that contain information. In Proceedings [43] Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai,
of the 14th Annual ACM Symposium on User Interface Software and Technology Diyi Yang, and Soroush Vosoughi. 2023. Training socially aligned language
(Orlando, Florida) (UIST ’01). Association for Computing Machinery, New York, models on simulated social interactions. arXiv preprint arXiv:2305.16960 (2023).
NY, USA, 141–150. [Link] [44] Xingyu Liu, Vladimir Kirilyuk, Xiuxiu Yuan, Alex Olwal, Peggy Chi, Xiang Chen,
[20] Xinbo Gao, Bing Xiao, Dacheng Tao, and Xuelong Li. 2010. A survey of graph and Ruofei Du. 2023. Visual Captions: Augmenting Verbal Communication With
edit distance. Pattern Analysis and applications 13 (2010), 113–129. On-the-fly Visuals. In Proceedings of the 2023 CHI Conference on Human Factors
[21] GitHub. 2023. GitHub Copilot · Your AI pair programmer. [Link] in Computing Systems (CHI). ACM, 20 pages. [Link]
features/copilot 3581566
[22] Tanmay Gupta and Aniruddha Kembhavi. 2023. Visual Programming: [45] Rishab Mitra, Arpit Narechania, Alex Endert, and John Stasko. 2022. Facilitating
Compositional Visual Reasoning Without Training. In 2023 IEEE/CVF Conference Conversational Interaction in Natural Language Interfaces for Visualization. In
on Computer Vision and Pattern Recognition. arXiv. [Link] 2022 IEEE Visualization Conference (VIS). IEEE. [Link]
InstructPipe CHI ’25, April 26 – May 1, 2025, Yokohama, Japan

2207.00189 [68] Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference
[46] B. A. Myers. 1986. Visual Programming, Programming by Example, and Program via python execution for reasoning. arXiv preprint arXiv:2303.08128 (2023).
Visualization: A Taxonomy. In Proceedings of the SIGCHI Conference on Human [69] William Robert Sutherland. 1966. The on-line graphical specification of computer
Factors in Computing Systems (Boston, Massachusetts, USA) (CHI ’86). Association procedures. Ph.D. Dissertation. Massachusetts Institute of Technology.
for Computing Machinery, New York, NY, USA, 59–66. [Link] [70] Yuan Tian, Weiwei Cui, Dazhen Deng, Xinjing Yi, Yurun Yang, Haidong Zhang,
22627.22349 and Yingcai Wu. 2024. Chartgpt: Leveraging llms to generate charts from abstract
[47] Arpit Narechania, Arjun Srinivasan, and John Stasko. 2021. NL4DV: A Toolkit for natural language. IEEE Transactions on Visualization and Computer Graphics
Generating Analytic Specifications for Data Visualization from Natural Language (2024).
Queries. IEEE Transactions on Visualization and Computer Graphics (TVCG) (2021). [71] Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. 2019.
[Link] Does BLEU score work for code migration?. In 2019 IEEE/ACM 27th International
[48] Allan Paivio. 1969. Mental imagery in associative learning and memory. Conference on Program Comprehension (ICPC). IEEE, 165–176.
Psychological review 76, 3 (1969), 241. [72] Unity. 2023. Unity’s Graph Editor. [Link]
[49] Allan Paivio. 1991. Dual coding theory: Retrospect and current status. Canadian visualscripting@1.7/manual/[Link]#the-graph-editor
Journal of Psychology/Revue canadienne de psychologie 45, 3 (1991), 255. [73] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
[50] Allan Paivio, James M Clark, et al. 2006. Dual coding theory and education. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All
Pathways to literacy achievement for high poverty children (2006), 1–20. You Need. Advances in Neural Information Processing Systems 30 (2017). https:
[51] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy //[Link]/10.5555/3295222.3295349
Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra [74] Bryan Wang, Gang Li, and Yang Li. 2023. Enabling Conversational Interaction
of Human Behavior. arXiv:2304.03442 [[Link]] With Mobile UI Using Large Language Models. In Proceedings of the 2023 CHI
[52] Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI
Liang, and Michael S. Bernstein. 2022. Social Simulacra: Creating Populated ’23). Association for Computing Machinery, New York, NY, USA, Article 432,
Prototypes for Social Computing Systems. In Proceedings of the 35th Annual ACM 17 pages. [Link]
Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). [75] Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao.
Association for Computing Machinery, New York, NY, USA, Article 74, 18 pages. 2024. Human-LLM Collaborative Annotation Through Effective Verification of
[Link] LLM Labels. In Proceedings of the CHI Conference on Human Factors in Computing
[53] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery,
Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. New York, NY, USA, Article 303, 21 pages. [Link]
Pytorch: An imperative style, high-performance deep learning library. Advances 3641960
in neural information processing systems 32 (2019). [76] Yun Wang, Zhitao Hou, Leixian Shen, Tongshuang Wu, Jiaqi Wang, He Huang,
[54] Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Haidong Zhang, and Dongmei Zhang. 2023. Towards Natural Language-Based
Gorilla: Large language model connected with massive apis. arXiv preprint Visualization Authoring. IEEE Transactions on Visualization and Computer
arXiv:2305.15334 (2023). Graphics 29, 1 (2023), 1222–1232. [Link]
[55] Debalina Ghosh Paul, Hong Zhu, and Ian Bayley. 2024. Benchmarks and [77] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia,
Metrics for Evaluations of Code Generation: A Critical Review. arXiv preprint Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits
arXiv:2406.12655 (2024). Reasoning in Large Language Models. arXiv:2201.11903 [[Link]]
[56] Zhenhui Peng, Xingbo Wang, Qiushi Han, Junkai Zhu, Xiaojuan Ma, and Huamin [78] Chao Wen, Jacqueline Staub, and Adish Singla. 2024. Program Syn-
Qu. 2023. Storyfier: Exploring Vocabulary Learning Support with Text Generation thesis Benchmark for Visual Programming in XLogoOnline Environment.
Models. arXiv:2308.03864 [[Link]] arXiv:2406.11334 [[Link]] [Link]
[57] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin [79] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.
models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789 (2023). 2019. Huggingface’s Transformers: State-of-the-Art Natural Language Processing.
[58] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin ArXiv Preprint ArXiv:1910.03771 (2019). [Link]
Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing [80] Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina,
Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. Michael Terry, and Carrie J Cai. 2022. PromptChainer: Chaining Large Language
ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. Model Prompts through Visual Programming. In Extended Abstracts of the 2022
arXiv:2307.16789 [[Link]] [Link] CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA)
[59] Subham Sah, Rishab Mitra, Arpit Narechania, Alex Endert, John Stasko, and (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article
Wenwen Dou. 2024. Generating Analytic Specifications for Data Visualization 359, 10 pages. [Link]
from Natural Language Queries using Large Language Models. Presented at the [81] Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S. Weld. 2021.
NLVIZ Workshop, IEEE VIS 2024. arXiv:2408.13391 [[Link]] [Link] Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving
abs/2408.13391 Models. arXiv:2101.00288 [[Link]]
[60] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, [82] Tongshuang Wu, Michael Terry, and Carrie Cai. 2022. AI Chains: Transparent
Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. and Controllable Human-AI Interaction by Chaining Large Language Model
Toolformer: Language models can teach themselves to use tools. Advances in Prompts. In CHI Conference on Human Factors in Computing Systems. ACM.
Neural Information Processing Systems 36 (2024). [Link]
[61] Leixian Shen, Enya Shen, Yuyu Luo, Xiaocong Yang, Xuming Hu, Xiongshuai [83] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan,
Zhang, Zhiwei Tai, and Jianmin Wang. 2023. Towards Natural Language and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language
Interfaces for Data Visualization: A Survey. IEEE Transactions on Visualization Models. arXiv:2210.03629 [[Link]]
and Computer Graphics 29, 6 (2023), 3121–3144. [Link] [84] Hui Ye, Jiaye Leng, Pengfei Xu, Karan Singh, and Hongbo Fu. 2024. ProInterAR:
2022.3148007 A Visual Programming Platform for Creating Immersive AR Interactions. In
[62] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Proceedings of the CHI Conference on Human Factors in Computing Systems
Zhuang. 2024. Hugginggpt: Solving ai tasks with chatgpt and its friends in (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York,
hugging face. Advances in Neural Information Processing Systems 36 (2024). NY, USA, Article 610, 15 pages. [Link]
[63] Douglas R Smith. 1985. The design of divide and conquer algorithms. Science of [85] Zhengyan Yu, Hun Namkung, Jiang Guo, Henry Milner, Joel Goldfoot, Yang
Computer Programming 5 (1985), 37–58. Wang, and Vyas Sekar. 2024. SEAM-EZ: Simplifying Stateful Analytics through
[64] Zefan Sramek, Arissa J. Sato, Zhongyi Zhou, Simo Hosio, and Koji Yatani. 2023. Visual Programming. In Proceedings of the CHI Conference on Human Factors in
SoundTraveller: Exploring Abstraction and Entanglement in Timbre Creation Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing
Interfaces for Synthesizers. In Proceedings of the 2023 ACM Designing Interactive Machinery, New York, NY, USA, Article 1041, 23 pages. [Link]
Systems Conference (Pittsburgh, PA, USA) (DIS ’23). Association for Computing 3613904.3642055
Machinery, New York, NY, USA, 95–114. [Link] [86] J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang.
[65] Anselm Strauss, Juliet Corbin, et al. 1990. Basics of qualitative research. Vol. 15. 2023. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design
sage Newbury Park, CA. LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in
[66] Anselm L Strauss. 1987. Qualitative analysis for social scientists. Cambridge Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing
university press. Machinery, New York, NY, USA, Article 437, 21 pages. [Link]
[67] Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: Enabling 3544548.3581388
Multilevel Exploration and Sensemaking with Large Language Models. arXiv [87] Lei Zhang and Steve Oney. 2020. FlowMatic: An Immersive Authoring Tool for
preprint arXiv:2305.11483 (2023). Creating Interactive Scenes in Virtual Reality. In Proceedings of the 33rd Annual
ACM Symposium on User Interface Software and Technology (Virtual Event, USA)
CHI ’25, April 26 – May 1, 2025, Yokohama, Japan Zhou et al.

(UIST ’20). Association for Computing Machinery, New York, NY, USA, 342–353.
[Link]
[88] Yongqi Zhang, Cuong Nguyen, Rubaiat Habib Kazi, and Lap-Fai Yu. 2023.
PoseVEC: Authoring Adaptive Pose-aware Effects Using Visual Programming and
Demonstrations. In ACM Symposium on User Interface Software and Technology.
[89] Zhenyu Zhang, Xuxi Chen, Tianlong Chen, and Zhangyang Wang. 2021. Efficient
lottery ticket finding: Less data is more. In International Conference on Machine
Learning. PMLR, 12380–12390.
[90] Zheng Zhang, Jie Gao, Ranjodh Singh Dhaliwal, and Toby Jia-Jun Li. 2023. VISAR:
A Human-AI Argumentative Writing Assistant with Visual Programming and
Rapid Draft Prototyping. arXiv preprint arXiv:2304.07810 (2023).
[91] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate
Before Use: Improving Few-shot Performance of Language Models. In Proceedings
of the 38th International Conference on Machine Learning (Proceedings of Machine
Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 12697–
12706. [Link]
[92] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate
Before Use: Improving Few-shot Performance of Language Models. In Proceedings
of the 38th International Conference on Machine Learning (Proceedings of Machine
Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 12697–
12706. [Link]
[93] Zhongyi Zhou and Koji Yatani. 2022. Gesture-Aware Interactive Machine
Teaching with In-Situ Object Annotations. In Proceedings of the 35th Annual
ACM Symposium on User Interface Software and Technology (Bend, OR, USA)
(UIST ’22). Association for Computing Machinery, New York, NY, USA, Article
27, 14 pages. [Link]

Figure 5: The prompt structure for the Node Selection module.

Each node description is formated as "{node types}: {short
descriptions of the nodes}; {recommended node(s)}". The node
recommendation is optional.
You are a programmer responsible for helping the user design an AI
pipeline.
Upon receiving a concise description from the user about the
desired functionality of the pipeline, you should generate the whole
pipeline using pseudocode.

Guidelines:
1. Respond solely in pseudocode, without additional commentary.
2. Utilize ONLY the nodes listed below; introducing new nodes is not
permitted.
3. Ensure there's a minimum of one line in each pseudocode
category: 'input', 'output', and 'processor'.

Below are the nodes you can incorporate into the pipeline:
… // detailed node configurations for each selected node

The following is a full list of nodes you may also use but those not
included above are not recommended:
… // a full list of node types supported by LLM2Pipeline

Examples:
Q:
{'description': 'generate a photo and validate whether it is real or
generated.', 'tag': 'multimodal'}
A:
… // pipeline pseudocode

… // more in-context examples

Figure 6: The prompt structure for the Code Writer module.

Detailed node configurations, see the appendix for examples,
are listed in the highlighted region.
InstructPipe CHI ’25, April 26 – May 1, 2025, Yokohama, Japan

(a) Before layout optimization. (b) After layout optimization.

Figure 7: A comparison of the same generated pipeline before and after layout optimization.

Pre-study
Training task
survey

InstructPipe Visual Blocks

Task 1 + Task 2

Switch
NASA TLX
condition

Yes Any
conditions
left?

No
Optional
open-end task

Post-study
interview

Figure 8: A flow diagram of the user study. After a training

session, participants completed the two tasks in each con-
dition in the sequence determined by the counterbalancing
protocol.
CHI ’25, April 26 – May 1, 2025, Yokohama, Japan Zhou et al.

Figure 9: Raw-TLX results. The statistic significance is annotated with ∗ , ∗∗ , or ∗∗∗ (representing 𝑝<.05, 𝑝<.01, and 𝑝<.001,
respectively).

(a) (b)
Figure 10: A comparison of InstructPipe generated by two instructions: (a) “Edit an image by updating the image caption” ; (b)
“Caption a tiger image using VQA, modify the character in the caption into a cat using LLM, and finally generate a cat image
based on the updated caption”. See Fig. 13c for the complete pipeline.
InstructPipe CHI ’25, April 26 – May 1, 2025, Yokohama, Japan

APPENDIX {
"nodeSpecId": "body_segmentation",
"description": "Segment out people in images.",
{
"nodeSpecId": "pali",
"description": "Answer questions about an image using a
"category": "processor", vision-language model.",

A A FULL LIST OF 27 NODES IN "inputSpecs": {

"image": {
"type": "image"
"category": "processor",
"inputSpecs": {
"image": {

INSTRUCTPIPE },
}

"outputSpecs": {
},
"type": "image"

"prompt": {
"segmentationResult": { "type": "string"
The following content shows 27 nodes InstructPipe covers in the "type": "masks",
"recommendedNodes": [ },
}

"mask_visualizer" "outputSpecs": {
generation process and their corresponding short description used ] "answer": {
} "type": "string"
in the Node Selector (§4.2): },
"examples": [ },
}

"live_camera_xhjtec: "examples": [
live_camera();\nbody_segmentation_xctd1p_out = "input_image_f1ohfa: input_image();\ninput_text_04ejnm:
body_segmentation_xctd1p: input_text(text=\"What is the person in the image
body_segmentation(image=live_camera_xhjtec);\nmask_visualizer_frjz doing?\");\npali_2pzuwn_out = pali_2pzuwn:

A.1 Input Nodes ga_out = mask_visualizer_frjzga:

mask_visualizer(image=live_camera_xhjtec,
segmentationResult=body_segmentation_xctd1p_out);\n"
pali(image=input_image_f1ohfa,
prompt=input_text_04ejnm);\nmarkdown_viewer_6wqe86:
markdown_viewer(markdownString=pali_2pzuwn_out);\n"
] ]
(1) live_camera: Capture video stream through your device } }

camera.
(a) Body segmentation (b) PaLI
(2) input_image: Select images to use as input to your pipeline.
Figure 11: Examples of node configuration used in Code
You can also upload your own images.
Writer. The configuration is structured in a JSON format.
(3) input_text: Add text to use as input to your pipeline.
(18) palm_model: Generate text using a large language model
based on prompt and context.
A.2 Output Nodes
(19) imagen: Generate an image based on a text prompt.
(1) image_viewer: View images. (20) input_sheet: Read string data from Google Sheets.
(2) threed_photo: Create a 3D photo effect from depthmap
tensors. B SYSTEM IMPLEMENTATION
(3) markdown_viewer: Render Markdown strings into styl-
ized HTML. B.1 System Prompts Used in LLM Modules
(4) html_viewer: Show HTML content with styles Here we provide more details about the prompts we utilized
in InstructPipe. The original txt files are also attached in the
supplementary zip file.
A.3 Processor Nodes
(1) google_search: Use Google to search the web that returns B.1.1 Node Selector. Please see our supplementary file
a list of URLs based on a given keyword; usually selected (node_select.txt) for the full prompt we use in this stage.
with string_picker.
B.1.2 Code Writer. The prompt in Code Writer is dynamic, which
(2) body_segmentation: Segment out people in images; usu-
is dependent on the nodes chosen in Node Selector. Therefore,
ally selected with mask_visualizer.
we cannot provide all the possible prompts in the supplementary
(3) tensor_to_depthmap: Display tensor data as a depth map.
materials. Here, we will focus on providing examples of two detailed
(4) portrait_depth: Generate a 3D depth map for an image;
node configurations utilized in InstructPipe. Fig. 6 shows the
usually selected with tensor_to_depthmap, threed_photo.
structure of the prompt utilized in this LLM stage. Fig. 11 provides
(5) face_landmark: Detect faces in images. Each face contains
two examples of node configurations (i.e., “Body segmentation”
468 keypoints; usually selected with landmark_visualizer,
and “PaLI”) that InstructPipe may chose into the highlighted line(s).
virtual_sticker.
Each configuration includes keys of “nodeSpecId” (i.e., node types),
(6) pose_landmark: Generate body positional mappings for
“description”, “category” and “examples”. For those nodes that
people detected in images; usually selected with land-
support input and output edges, “inputSpecs” and “outputSpecs”
mark_visualizer.
specify the sockets and their corresponding valid data types.
(7) image_processor: Process an image (crop, resize, shear,
For example, the output socket name of “Body segmentation” is
rotate, change brightness or contrast, add blur or noise).
“segmentationResult”, and its data type is “masks”. Some nodes (e.g.,
(8) text_processor: Reformat and combine multiple text inputs.
“Body segmentation”) include recommended node(s) (e.g., “Mask
(9) mask_visualizer: Visualize masks.
visualizer” for “Body segmentation”), and our configuration also
(10) string_picker: Select one string from a list of strings;
contains such information in the dictionary.
usually used with google_search.
(11) image_mixer: Combine images and text into one output
B.2 Code Interpreter
image. Requires two image inputs.
(12) virtual_sticker: Use face landmarks data to overlay virtual Here we provide more low-level implementation details on Code
stickers on images. Interpreter. The Code Interpreter parses generated pseudocode into
(13) palm_textgen: Generate Text using a large language model. a visual programming pipeline for visualization at the Visual Blocks
(14) keywords_to_image: Search for images by keywords. workspace. Fig. 12 shows the data type definition of graphs, nodes,
(15) url_to_html: Crawl the website by a given URL. and edges in the system. The example JSON file to be parsed into
(16) image_to_text: Extract text from an image using OCR the Typescript interface is available at the Visual Blocks website5 .
service. 5 [Link] JSON file is available for data structure
(17) pali: Answer questions about an image using a vision- exploration by 1) entering an example project and 2) clicking on the “Export” button
language model. on the top-right corner.
CHI ’25, April 26 – May 1, 2025, Yokohama, Japan Zhou et al.

Our code defines a visual programming pipeline into an array of Algorithm 1: Code Interpreter
serialized nodes, 𝐺 (𝑁 ). When the user adds a new node to an
1 Input: 𝐶: the generated texts (i.e., pseudocode) in the string
pipeline, it is adding a new “SerializedNode”, where it contains
format.
the edge definition between this new node and other nodes in the
2 Output: 𝐺 (𝑁 ): a visual programming pipeline
current workspace, to the current “SerialedGraph”. This mechanism
(𝑆𝑒𝑟𝑖𝑎𝑙𝑖𝑧𝑒𝑑𝐺𝑟𝑎𝑝ℎ) that mainly stores an array of
ensures that nodes can be incrementally added in the order they
𝑆𝑒𝑖𝑟𝑖𝑎𝑙𝑖𝑧𝑒𝑑𝑁𝑜𝑑𝑒 (Fig. 12).
appear in the pseudocode order while maintaining the integrity of
3 Variables: 𝑇 : a dictionary of parsed tokens that contains
the graph by clearly defining dependencies and data flow between
𝑜𝑢𝑡𝑝𝑢𝑡_𝑣𝑎𝑟𝑖𝑎𝑏𝑙𝑒_𝑖𝑑, 𝑛𝑜𝑑𝑒_𝑖𝑑, 𝑛𝑜𝑑𝑒_𝑡𝑦𝑝𝑒,
nodes. Algorithm 1 further shows how InstructPipe parses code
𝑛𝑜𝑑𝑒_𝑎𝑟𝑔𝑢𝑚𝑒𝑛𝑡𝑠; 𝑒: the incoming edges of a new node, in
and incrementally add nodes to formulate a final serialized graph.
the format of 𝑆𝑒𝑟𝑖𝑎𝑙𝑖𝑧𝑒𝑑𝐼𝑛𝑐𝑜𝑚𝑖𝑛𝑔𝐸𝑑𝑔𝑒; 𝑛: a new node in
/** A serialized graph. */ the format of 𝑆𝑒𝑟𝑖𝑎𝑙𝑖𝑧𝑒𝑑𝑁𝑜𝑑𝑒.
export declare interface SerializedGraph { 4 𝐺 : 𝑆𝑒𝑟𝑖𝑎𝑙𝑖𝑧𝑒𝑑𝑁𝑜𝑑𝑒 [] = [] // Initialize 𝐺 as an
nodes: SerializedNode[]; empty array
5 𝑙𝑖𝑛𝑒𝑠 = 𝑙𝑖𝑛𝑒_𝑝𝑎𝑟𝑠𝑒𝑟 (𝐶) // Parse 𝐶 into lines of code
/** other properties */ with no pseudocode order changed.
}
6 for 𝑙𝑖𝑛𝑒 in 𝑙𝑖𝑛𝑒𝑠 do
/* Example: */
/** A serialized node. */ /* 𝑝𝑎𝑙𝑖_1_𝑜𝑢𝑡 = 𝑝𝑎𝑙𝑖_1 : 𝑝𝑎𝑙𝑖 (𝑖𝑚𝑎𝑔𝑒 =
export declare interface SerializedNode { 𝑖𝑛𝑝𝑢𝑡_𝑖𝑚𝑎𝑔𝑒_1, 𝑝𝑟𝑜𝑚𝑝𝑡 = 𝑖𝑛𝑝𝑢𝑡_𝑡𝑒𝑥𝑡_1) */
/** The id of the node, e.g., pali_1. */ /* –> */
id: string; /* ‘𝑝𝑎𝑙𝑖_1_𝑜𝑢𝑡’, ‘𝑝𝑎𝑙𝑖_1’, ‘𝑝𝑎𝑙𝑖’ and
[‘𝑖𝑚𝑎𝑔𝑒 = 𝑖𝑛𝑝𝑢𝑡_𝑖𝑚𝑎𝑔𝑒_1’, ‘𝑝𝑟𝑜𝑚𝑝𝑡 = 𝑖𝑛𝑝𝑢𝑡_𝑡𝑒𝑥𝑡_1’]
/** The node spec id, e.g., pali. */ */
nodeSpecId: string; 7 𝑇 = 𝑡𝑜𝑘𝑒𝑛𝑖𝑧𝑒𝑟 (𝑙𝑖𝑛𝑒)

8 𝑒 : 𝑖𝑛𝑐𝑜𝑚𝑖𝑛𝑔𝐸𝑑𝑔𝑒𝑠 = 𝑐𝑟𝑒𝑎𝑡𝑒_𝑖𝑛𝑐𝑜𝑚𝑖𝑛𝑔_𝑒𝑑𝑔𝑒𝑠 (𝑇 )
/** Serialized incoming edges that connect to this node. */ // create incoming edges for the new node
incomingEdges?: {[inputId: string]: SerializedIncomingEdge[]};
9 𝑛 : 𝑆𝑒𝑟𝑖𝑎𝑙𝑖𝑧𝑒𝑑𝑁𝑜𝑑𝑒 = 𝑐𝑟𝑒𝑎𝑡𝑒_𝑛𝑜𝑑𝑒 (𝑇 , 𝑒) // create a
/** other properties */ new SerializedNode with the incomingEdges
} and the parsed dictionary

10 𝐺 .𝑎𝑑𝑑_𝑠𝑒𝑟𝑖𝑎𝑙𝑖𝑧𝑒𝑑𝑛𝑜𝑑𝑒 (𝑛) // add the new

/** A serialized incoming edge. */
SerializedNode to the graph
export declare interface SerializedIncomingEdge {
/** The id of the source node. */ 11 𝐺 = 𝑜𝑝𝑡𝑖𝑚_𝑙𝑎𝑦𝑜𝑢𝑡 (𝐺) // Perform the UI layout
sourceNodeId: string; optimization, as shown in Fig. 7
12 return 𝐺
/** The id of the output in the source node. */
outputId: string; the main graph, is defined as a graph (or a node) that has no
} connection to the main graph in the pipeline that provides
the main functionality of the pipeline (e.g., the “Image viewer”
Figure 12: The definition of a graph, a node and an edge node on the bottom-left corner of Fig. 7b). We observed that
in the system using the Typescript language. Only core some participants typically would like to explore the system
properties of graphic structure definition are presented in by working on a separate sub-space. While we acknowledge
this figure. its usefulness, leaving such “redundant” graphs in the raw
data for the evaluation would cause issues when we calculate
C TECHNICAL EVALUATION the number of user interactions (i.e., the metric used in the
evaluation that will be defined in the next subsection).
C.1 Data Post-Processing • Low-quality captions. While we explicitly required the
After the workshops, one author carefully examined each collected participants to write descriptive captions, we found some
pipeline and found several critical issues in the raw data: captions written by the participants were either empty
• Incomplete pipelines. There exist pipelines uploaded by or low-quality (e.g., “newsletter”, “image editing” and
the participants that were incomplete. “[participant name]-demo”).
• Isolated graphs. There exist pipelines that include at least
one isolated subgraph. The isolated subgraph, as opposed to
InstructPipe CHI ’25, April 26 – May 1, 2025, Yokohama, Japan

(a) Search news from Google, summarize it and then conduct fact check. Input: a keyword for Google Search; Output: a summarization of the
news and a fact-check result.

(b) Generating an emoji from a photo. Input: a photo uploaded by the user; Output: an emoji generated from the photo.

(c) Turning a tiger into a cat. Input: an image of a tiger; Output: an image of a cat.
Figure 13: Example pipelines participants built in the workshops.
CHI ’25, April 26 – May 1, 2025, Yokohama, Japan Zhou et al.

The observation motivated us to post-process the raw data complete a pipeline “from our generated pipeline” to that “from
to present more rigorous evaluation results. We first removed scratch” as our target metric in the technical evaluation.
incomplete pipelines and the isolated graphs in each pipeline
(if there are any). To further enhance the annotation quality, D USER EVALUATION
two authors individually annotated the caption of each pipeline
D.1 Semi-structured Interview Script
separately by referring to the original captions and pipelines
authored by the participants. It is important to note that we finished [ Introduction ] ( Start timing! 60 min max. )
the workshop and the data annotation task before we completed Hello, my name is X.
the system implementation. The two authors had no experience First, I would like to thank you for your participation and
using InstructPipe before completing the annotation. We believed completing the consent form. Today, you will be a participant in a
this process could effectively enhance the quality of the captions user study regarding machine learning and visual programming.
while maintain fairness of the technical evaluation. Your data will be kept anonymous. Additionally, as a researcher I
As we clarified in §5.1, the workshop is designed to be an open- have no position on this topic and ask that you be as open, honest,
ended creation process. This indicates that the dataset inevitably and detailed in your answers as possible. Do you have any questions
includes out-of-scope nodes like “custom scripts” (in which the before we begin?
participants write code to process the input data and return custom Basically, visual programming borrows the metaphor of block
outputs; see Fig. 13b for an example) and “TFLite model runner” building and allows novice users to develop digital functionalities
(which call a custom TensorFlow model with a URL input of the without writing codes.
model in the TF-Hub). [Show Visual Blocks]
We removed the pipelines that contain node(s) out of our focus Here, each block is called a node, and each node takes in specific
27 nodes, and selected all the remaining pipelines as our final inputs, then returns the desired outputs. What you can do is to
evaluation set. We argue that this post-processing is critical for connect a series of nodes together as a pipeline to achieve a high-
reporting a fair accuracy value since InstructPipe can only generate level goal.
pipelines based on its known node library. The final 48 pipelines We are going to walk you through our Visual Blocks system and
(out of 64 pipelines) are comprised of 23 language pipelines, seven ask you to actually use Visual Blocks in two conditions to create a
visual pipelines, and 18 multi-modal pipelines. Fig. 13 shows three few applications.
pipelines created by the participants. Fig. 13b is an example of [ Tutorial ]( Start timing! 10 min max. )
the pipelines that include out-of-scope nodes, and therefore is not Before we get started, let us do a tutorial of our system.
included in the final 48 pipelines. In the technical evaluation, we ran [ Study and TLX ]( Start timing! about 30 min )
our generation algorithm on the pipeline captions six times (three [Leverage the counter-balanced sheet and give user a task]
times for each caption × two captions from two authors for each [Think aloud. Have a short discussion with the user. What’s the
pipelines) and evaluated the generation results using the metric user’s plan to achieve this given functionality?]
that will be introduced below. [ Interview ]( Start timing! about 15 min )
1. What’s your impression of Visual Blocks / InstructPipe
[counterbalanced]? Do you need many edits / operations to make
C.2 Evaluation Metric: The Number of User it work?
Interactions 2. Are there any pipelines you come up with in work scenarios /
Our definition of the number of user interactions has two important casual scenarios?
implications. First, a complete pipeline after user interaction does 3. What works with InstructPipe? In what specific scenarios will
not need to be the same as the corresponding pipeline in the dataset. InstructPipe be very helpful?
As long as it fulfills the task described in the caption, we consider 4. What does not work with InstructPipe? Would you give me
the pipeline complete. Second, our definition does not consider an example?
interactions of modifying the node parameters, e.g., typing in a text 5. Do you have any suggestions to improve the design of both
box or selecting a value in a drop-down box. We argue that such systems?
interactions are highly node-dependent and are hard to quantify 6. Which kinds of technologies would be interesting to add?
objectively. More importantly, as we explain in §4.1, the generation 7. What applications do you want to start with InstructPipe?
of node parameters is out of the scope of this work. And what applications do you want start without it?
In the technical evaluation with various pipelines, it is unfair That’s all for our user study. Thank you for your participation
to report an averaged absolute value of user interactions because and we will compensate for your time.
the complexity of the pipelines varies dramatically. For instance,
the user may need to make three edits based on a generated result D.2 User Study Pipelines
to complete a large pipeline that requires 20 edits from scratch. Fig. 14 and Fig. 15 visualize two pipelines we required the
In another pipeline, the user also needs to do three edits starting participants to complete in our user study. Fig. 15 is a multimodal
from the generated result, but the whole pipeline only takes three pipeline that allows participants to interact with AR effects in
edits to finish. Averaging these absolute values does not provide real time. Our technical evaluation shows that InstructPipe can
reasonable insights into how accurate the generation is. Therefore, generate this pipeline accurately: the averaged ratio of human
we reported an averaged ratio of user interactions required to interactions = 5.2%. Fig. 15 is a text-based pipeline that provides
InstructPipe CHI ’25, April 26 – May 1, 2025, Yokohama, Japan

Table 3: Participant demographics for the user study, showing various demographic characteristics and skills relevant to
InstructPipe.
Self-identified Age Programming Machine LLM
ID Job Title
Gender Group Experience Learninig Skill Usage
P1 Product Manager Woman 25 - 34 Beginner Beginner At least once a month
P2 Image Tuning Engineer Man 35 - 44 Intermediate Beginner At least once a week
P3 Program Manager Woman 45 - 54 No experience No experience At least once a week
P4 Hardware Engineer Man 35 - 44 Intermediate No experience At least once a month
P5 Technical Program Manager Man 35 - 44 Beginner No experience At least once a day
P6 Senior Hardware Engineer Man 35 - 44 Beginner No experience At least once a month
P7 Technical Program Manager Woman 18 - 24 Beginner Beginner Never used it
P8 Technical program manager Man 25 - 34 No experience No experience Multiple hours every day
P9 Solutions Engineer Man 25 - 34 Beginner No experience At least once a month
P10 Program Manager Man 55 - 64 Beginner Beginner At least once a month
P11 Program Manager Woman 35 - 44 No experience No experience Never used it
P12 Lab Manager Man 35 - 44 Intermediate Beginner At least once a week
P13 Partner Development Manager Man 25 - 34 Beginner Beginner At least once a week
P14 Hardware Engineer Man 25 - 34 Beginner Beginner At least once a week
P15 Global Supply Manager Man 25 - 34 Beginner No experience At least once a month
P16 Global Supply Manager Woman 55 - 64 No experience No experience At least once a week

Figure 14: Text-based pipeline. The “String picker” node provides users a drop-down menus to select one URL from a list of
URLs returned by “Google Search”. “PaLM Text Generator” is an LLM used to summarize the full HTML page.

LLM capability to process the texts instead of simply combining

two texts.
Note that even though InstructPipe may be able to complete the
pipeline structure in Fig. 15 from users’ instruction, we observed
that participants still need to fine-tune their keywords to get an
Figure 15: Real-time multimodal pipeline. The “Keyword to ideal pair of sunglasses. Additionally, the default anchor value is
image” node is used to search sunglasses image, and the “Face top”, so participants need to use the drop-down menu on the
“Virtual sticker” node anchors the sunglasses onto users’ “Virtual sticker” node to change the value to “Eyes”. This further
face. motivates us to use the metric of “Time” in addition to the number
of user interactions in our study. Our demo video also covers the
participants with a summary of the news searched from Google. workflows of these two pipelines.
The technical evaluation reveals that InstructPipe cannot generate
this pipeline accurately without further human interaction, and the D.3 Assistant Provided to the Participants in
averaged ratio of additional human interactions is 27.8%. While the
generated diagram (with error) is not deterministic, we observed
the User Evaluation
that InstructPipe commonly generates the pipeline in Figure 14 In the user evaluation, our goal is to make the interface condition
without “URL to HTML” or “PaLM Text Generator” nodes. The (either InstructPipe or Visual Blocks) as the only independent
error implies that the LLM may misinterpret 1) the data from the variable that changes our dependent variables (§5.2). Similar to user
“selected text” port of the “String picker” node is the texts on the evaluations of other early-stage HCI research, we had to improvise
web instead of the web URL and 2) that “Text processor” has the for lacking system affordances. As an example, we would include
help menus and error recovery modals in the future versions of
CHI ’25, April 26 – May 1, 2025, Yokohama, Japan Zhou et al.

our system but at this early stage we relied on in-person help to

nudge and assist our user study participants. We took actions (i.e.,
assistants) in the user evaluation to ensure the study is under an
appropriate amount of control as well as maintain the fairness of
our study.
Here we elaborate two examples of assistants we provided in
the user study.
In the InstructPipe condition, one participant started their
“instructions” by dragging a text box into the visual programming
workspace and began typing. When noticing this issue, we kindly
asked the participant whether s/he wanted to write instructions
or build a pipeline from scratch. S/he then noticed this issue and
clicked on the “InstructPipe” button to write prompts. Note that
we explicitly taught every participant how to use InstructPipe
and asked participants themselves to go through the instruction
processes in the training task (Fig. 8).
In the Visual Blocks condition, one participant first dragged a
“Virtual sticker” into the workspace when s/he wants to build the
multimodal pipeline as required (Fig. 15). After a while, s/he asked
us for the meaning of “landmarks” on the first input port of the
“Virtual sticker” node (Fig. 15). We then answered this question and
provided a hint on the “Face landmark” node (Fig. 15) that could
produce the “Face landmarks” required by the “Virtual sticker”.
While we had explained all the nodes that the participants need
to use in the study in the training task (Fig. 8), we consider such
technical questions as reasonable because all of our participants
are non-experts. Programming itself is a difficult skill, and it is
quite common that people may forget some of the knowledge
that they have just learned. Instead of being silent and keeping
the participants stuck on a technical issue, we believed offering
technical help is an important action we must take to ensure the
data quality we collected in the study.
These anecdotes in the user evaluation reveal several limitations
of the visual programming system: some designs may not be very
intuitive to non-experts. Since the goal of our user evaluation is
understanding the benefits of InstructPipe compared to Visual
Blocks (without AI assistants), we made our best efforts to
take actions to prevent the effects caused by other factors from
influencing our data. Meanwhile, we also encourage future work
to further explore the system design so that future users can more
easily use our assistant in visual programming.

D.4 Counter-Balancing and The Replication

Number
Table 4 presents how we perform counterbalance in the user evalu-
ation. We counterbalanced both the interface factor (“InstructPipe”
and “Visual Blocks”) and the pipeline factors to reduce the learning
effects. We then replicated the order four times so that we collected
multiple data from different participants in each unique study order.
This helps strengthen the power of the data we collected in the
study. Note that, in the group of P5 - P8, we flipped the orders
within P5 and P6 as well as P7 and P8, but this does not cause a
difference in the counterbalance process.
InstructPipe CHI ’25, April 26 – May 1, 2025, Yokohama, Japan

Table 4: The counterbalance sheet of the user evaluation. Each cell is in the format of “Interface / Pipeline”. “Instruct” and
“VB” mean the “InstructPipe” and “Visual Blocks” conditions, respectively. “Search” and “Tryon” represent the “text-based
pipeline” (Fig. 14) and the “real-time multimodal pipeline” (Fig. 15), respectively.
ID Step 1 Step 2 Step 3 Step 4
P1 Instruct / Tryon Instruct / Search VB / Tryon VB / Search
P2 VB / Tryon VB / Search Instruct / Tryon Instruct / Search
P3 Instruct / Search Instruct / Tryon VB / Search VB / Tryon
P4 VB / Search VB / Tryon Instruct / Search Instruct / Tryon
P5 VB / Tryon VB / Search Instruct / Tryon Instruct / Search
P6 Instruct / Tryon Instruct / Search VB / Tryon VB / Search
P7 VB / Search VB / Tryon Instruct / Search Instruct / Tryon
P8 Instruct / Search Instruct / Tryon VB / Search VB / Tryon
P9 Instruct / Tryon Instruct / Search VB / Tryon VB / Search
P10 VB / Tryon VB / Search Instruct / Tryon Instruct / Search
P11 Instruct / Search Instruct / Tryon VB / Search VB / Tryon
P12 VB / Search VB / Tryon Instruct / Search Instruct / Tryon
P13 VB / Tryon VB / Search Instruct / Tryon Instruct / Search
P14 Instruct / Tryon Instruct / Search VB / Tryon VB / Search
P15 VB / Search VB / Tryon Instruct / Search Instruct / Tryon
P16 Instruct / Search Instruct / Tryon VB / Search VB / Tryon