Software and Systems Modeling

[Link]

REGULAR PAPER

Model-based GUI automation

Joshua Spinak1

Received: 27 October 2023 / Revised: 12 June 2025 / Accepted: 22 June 2025

© The Author(s) 2025

Abstract
This paper addresses fundamental limitations in visual GUI automation by redefining current approaches as process-based
and introducing a novel model-based alternative. By addressing the inherent stochasticity in GUI environments through
insights from software testing, machine vision, robotics, and human cognition, I present a theoretical framework that enables
integration and unit testing of GUI automation and significantly improves automation robustness. The approach reduces
code complexity and supports larger automation applications, potentially enabling GUI automation in previously impractical
domains.

Keywords Model-based · GUI · Automation · Brobot · Visual API

Mathematics Subject Classification Primary 68N19; Secondary 68N01

1 Introduction GUI automation as a domain. Section 4 looks at visual GUI

automation in software testing, emphasizing the limitations
Imagine programming a robot to go to the grocery store to of existing process-based techniques. Section 5 presents for-
buy diapers and wine. If you had to program the steps in mal mathematical models of GUI automation, and Sect. 6
advance, unexpected scenarios such as the diaper delivery explores how these models can be implemented in a frame-
truck blocking the entrance would confuse your robot. Since work. The DoT app provides practical implementations of
testing it would require watching it perform, every failure key concepts in the paper and is introduced in Sect. 7.
would take time to uncover and provide data for just one new Section 8 examines stochasticity in GUI automation and
workaround. Getting this robot to and from the store may be Sect. 9 looks at the transformation from implicit to explicit
possible, but at what cost? How complex must the algorithm environment representation. Section 10 explores robustness,
be and how long would it take to code it? code complexity, and scalability through practical exam-
This robot represents the state of visual GUI automation ples and mathematical analysis, demonstrating the real-world
until the emergence of model-based GUI automation, first advantages of model-based GUI automation. Section 11 dis-
implemented in the open-source Brobot framework appear- cusses the novel approach to integration and unit testing
ing in January 2022. Model-based GUI automation makes of GUI automation applications. Finally, Sect. 12 explores
it possible to reliably solve complex tasks, like the GUI the transformative applications enabled by this model-based
automation equivalent of sending a robot to the grocery store approach, illustrating its potential impact across various
for diapers and wine. domains.
The remainder of the paper is organized as follows. Sec-
tion 2 provides the foundational concepts of GUI automation
and key concepts in adjacent domains. Section 3 discusses 2 Foundations of GUI automation
related work and defines the boundaries of model-based
This section lays the groundwork for understanding model-
Communicated by Zhenjiang Hu. based GUI automation by introducing key concepts and
definitions. It explores the nature of GUI environments and
B Joshua Spinak
fundamentals of GUI automation.
jspinak@[Link]
The GUI environment for a specific activity is the set of all
1 Düsseldorf, Germany possible screens displayed on the monitor during this activity.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

When a person uses a computer to perform a task, such as deterministic and does not depend on a trained neural net or
booking a flight, they do so with an understanding of how to latent spaces.
use the computer, its operating system, the web browser, and
the specific website. This understanding represents a mental
model of the GUI environment for that activity. 3 Related work and domain boundaries
GUI automation is the process of manipulating the graph-
ical user interface with software. GUI automation, whether Although this paper focuses on automation rather than
for productivity, game playing, software testing, or any other testing, the academic literature on GUI automation pre-
activity, involves at least 2 applications: (1) the application dominantly addresses testing applications. Understanding
controlling the automation, and (2) the application(s) acted this landscape is crucial for positioning model-based GUI
on, or being automated. In software testing, the automated automation as a distinct domain. In this section, I analyze
application is referred to as the AUT, Application Under Test, the distinctions between continuous and discrete processes,
or SUT, Software Under Test (Banerjee et al. 2013) [1]. establish clear boundaries between model-based testing and
In the context of GUI automation, a GUI action is an indi- model-based GUI automation, and clarify terminology to
vidual operation performed during automation. Examples of prevent confusion.
GUI actions include clicking a location, searching for an
image, dragging the mouse, and typing (Yeh et al. 2009) [2]. 3.1 Continuous versus discrete processes: a
A GUI process is a sequence of GUI actions and can contain fundamental mismatch
one or more actions.
Process-based GUI automation tools use scripts or work- Software testing is a discrete [4] activity. Each test case
flows to perform actions on the GUI. Scripts are sequences operates as an independent unit designed to verify a spe-
of commands or instructions and can be coded manually in a cific functionality. When a test finishes—whether passing or
programming or domain-specific language, assembled using failing—the system resets to its initial state before running
a GUI, or recorded for later replay. GUI automation is tradi- the next test. This discreteness is essential for testing, where
tionally a process-based activity. isolation between test cases ensures clear diagnostics.
GUI automation can be divided into two categories: Process-based visual GUI automation tools align well with
declarative and visual. An example of a tool that does declar- these discrete test cases, as both follow predetermined action
ative GUI automation is Selenium, which accesses web sequences with clear start and end points. However, this cre-
elements based on their attributes or hierarchical position ates a fundamental mismatch when applying these same tools
within the HTML document (Selenium, 2024) [3]. to other GUI activities. Most real-world GUI interactions are
Visual GUI automation acts as a human agent, uses continuous processes. For example, when someone buys a
machine vision to understand the GUI, and performs key- flight online and encounters an unexpected issue like a bro-
board and mouse actions (e.g., click, type, drag) to manip- ken link, they do not abandon the entire task and restart from
ulate it. Visual GUI test automation tools can recognize the beginning. Instead, they adapt, finding an alternative path
GUI elements based on appearance (e.g., color, shape, text) forward from their current position.
instead of internal properties (e.g., name, ID, class). They This incompatibility explains why traditional visual GUI
can work with any application regardless of its platform or automation tools often struggle with non-testing automation
technology. Perhaps the best-known visual GUI automation tasks. Three key characteristics highlight this mismatch:
software is SikuliX1 (Yeh et al. 2009) [2].
Pattern matching is the type of image recognition primar- • Situation Continuity: Continuous processes have fluid,
ily used in visual GUI automation and visual GUI testing. non-resettable states that contrast sharply with the dis-
One implementation of pattern matching is the OpenCV tinct, repeatable steps of discrete test cases.
library’s matchTemplate function, which slides an image • Adaptive Goal Pursuit: They require dynamically
across the search region and compares overlapping pixels. adjusting actions based on current conditions, unlike dis-
Typically, an image on file is compared to the pixels in the crete processes that follow predetermined paths.
GUI. When a part of the screen meets a specified minimum • Cumulative Historical Impact: Past actions shape
similarity score, the observation is successful and returns the future possibilities, a stark contrast to discrete processes
on-screen coordinates of the image match. This process is where each step operates more independently.

1
Figure 1 illustrates this critical distinction. Software test-
SikuliX, originally called Sikuli, is an open-source project initially
developed by Tsung-Hsiang Chang and Tom Yeh and maintained by
ing naturally aligns with process-based, discrete approaches
Raimund Hocke. The author of this paper has also contributed code to whether performed manually or through automation. How-
SikuliX. ever, activities such as game playing and productivity

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

Fig. 1 Traditional GUI

automation differs from many
manually performed activities,
although AI-driven tools can
also adopt a model-based
approach

tasks are fundamentally continuous. While humans naturally 3.3 Terminological clarifications
approach these continuous activities with a mental model of
the GUI environment, the process-based approach of tradi- The term model-based automation is used interchangeably
tional automation tools creates an inherent limitation. This in common discourse with the terms model-based testing and
misalignment between tool capabilities and activity require- model-based test automation.
ments explains why conventional GUI automation often fails In model-based automation and model-based GUI automa-
in non-testing contexts. tion, the words model-based and automation have different
meanings, as captured in Table 1.
In model-based automation, automation refers to the gen-
3.2 Model-based testing eration of test cases and not to their execution, which can be
automated or performed manually. The term model-based in
Using a model of the AUT, a model-based testing tool can this context refers to the use of a model of the AUT to generate
generate test cases that cover different paths in the AUT test cases. When automated, test cases employ process-based
(Petrenko et al. 2012) [5]. The model in model-based testing GUI automation (do A, then do B, etc.).
is a map of possible GUI actions (Utting et al. 2011) [6]. In contrast, in model-based GUI automation, automation
Figure 2 demonstrates how model-based testing works. refers specifically to GUI automation (manipulating the GUI
Figure 3 shows a model of an AUT characteristic of model- without user input). The term model-based refers to the use
based testing. The model is a directed graph comprised of of a comprehensive model of GUI automation, including the
nodes (corresponding to actions taken on GUI elements) GUI environment, GUI action execution, pathfinding, path
and edges (showing which actions are possible from the last traversal, and state management.
action taken). For example, the edge from the blue node to the
yellow node shows that the yellow action can be performed 3.4 Testing automation versus automated testing
after the blue action. Model-based testing uses the model to
describe all possible action chains within the AUT, referred One of the principal innovations of model-based GUI
to as the event-interaction graph (EIG) by Yuan and Memon automation is the ability to systematically test the automation
(2010) [7]. Typically, the model is used to generate a list of code itself. Given that GUI automation is a cornerstone of
test scripts that are then analyzed to find deficiencies in the modern software testing, it is crucial to distinguish between
AUT. The generated test scripts are linear processes, and the two related but distinct concepts: automated testing and test-
execution of each script remains rigid and sequential. ing automation.
The number of paths in an EIG and the time it takes to
run all tests increase exponentially with a linear increase in
connected states (Moreira et al. 2017) [8]. This scalability
issue is known as the state explosion problem (Clarke et al.
2012) [9].

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

Fig. 2 In model-based testing, test cases are generated from a model of the AUT

Fig. 3 Nodes represent actions

taken on GUI widgets, and
edges show potential next
actions

Table 1 An explanation of the differences between similar-looking terminology

Terminology: model-based... Automation means Model-based means Primary goal

Automation Test cases are generated A model of the AUT is used To verify the correctness of
Testing from code to generate test cases the application under test
(AUT)
Test automation
GUI automation An application controls the A model of GUI automa- To reliably perform com-
GUI with the mouse and key- tion (the GUI environment, plex, continuous tasks
board action execution, pathfind- within a GUI environment
ing, state management, etc.)

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

• Automated testing is the familiar practice of using soft- • Unexpected delays in the process flow: Visual GUI
ware to test another piece of software (the Application automation tools execute scripts sequentially based on pre-
Under Test). To use a metaphor, this is like building a defined timings or events. However, the execution speed
robot to test a car’s brakes. The focus is on the car’s per- and order of scripts can be affected by factors such as net-
formance. work latency, system load, and application responsiveness.
• Testing automation, a novel capability enabled by the These factors can lead to unexpected delays or interrup-
model-based approach, refers to the process of testing tions that can cause scripts to fail (Garousi et al. 2017;
the automation code itself. In the metaphor, this is akin Nass et al. 2021) [11, 13].
to testing the robot to ensure its own arms, sensors, and • Dynamic content generation: Many modern applications
logic function correctly before it ever touches the car. have dynamic or adaptive GUIs that can change based on
user actions, data inputs, and context information, render-
ing automation scripts ineffective (Nass et al. 2021) [13].
Model-based GUI automation makes this second type of
• GUI changes between versions: Any unanticipated change
testing feasible for the first time in a structured way, applying
to the GUI can cause scripts to fail. An AUT with frequent
standard software engineering practices to a domain where
updates easily breaks test scripts built for static environ-
they were previously impractical. Integration and unit testing
ments (Nass et al. 2021; Yandrapally et al. 2014) [13, 14].
of GUI automation are discussed in depth in Sect. 11.
Aho and Vos (2018) wrote that GUI tests are process flows
and process flows are innately fragile: “Usually a GUI test
4 Challenges in traditional GUI automation case contains a sequence of events or interactions...an incor-
rect GUI state in the middle of the sequence may lead to an
Traditional GUI automation approaches face persistent chal- unexpected screen, making further test case execution use-
lenges that limit their effectiveness in complex, real-world less [4].”
applications. Despite decades of incremental improvements, Empirical evidence confirms the pervasiveness of this
visual GUI automation continues to suffer from fundamen- problem [4, 11–21]. Tests can fail despite having a functional
tal problems that impede its broader adoption. This section AUT and false positives occur frequently due to automation
examines the core limitations of process-based GUI automa- failures (Alégroth et al. 2014; Memon and Soffa 2003) [16,
tion approaches, with particular emphasis on script fragility 21]. Alégroth et al. (2014) found that 16% of 17,567 tests gave
and the relationship between robustness and code complex- false positives due to script fragility while only 3% identi-
ity. Understanding these challenges helps us to appreciate fied actual defects [16]. More dramatically, Memon and Soffa
why a fundamentally different approach is necessary. (2003) reported that 72% of tests failed on new releases of
Adobe Acrobat Reader due to changes in the GUI layout and
4.1 Script fragility: the fundamental problem appearance [21]. Despite 20 years of research, these chal-
lenges have seen very little progress (Nass et al. 2019) [13].
Compared to manual testing, automated testing improves
accuracy and scalability but introduces complexities such 4.2 Complexity-robustness trade-off
as maintenance and coding challenges (Dobslaw et al. 2019)
[10]. The Achilles heel of visual GUI test automation is script The fundamental challenge in addressing script fragility lies
fragility, which refers to the tendency of automation to fail in the inverse relationship between robustness and code com-
due to changes in the GUI environment. These failures are plexity. Thummalapenta et al. (2013) observed this trade-off:
prominent in visual GUI testing and are discussed exten- “For pragmatic reasons, some amount of script brittleness
sively in the academic literature. Script fragility is caused by might be acceptable when weighted against the coding effort
various factors, such as: required for increasing change-resiliency [19].”
Attempts to improve robustness in process-based
• Images not found or found erroneously: Pattern-matching approaches typically involve:
algorithms can be sensitive to variations in resolution,
scaling, color depth, brightness, contrast, etc., and can • Adding exception handling for each potential failure
be confused by similar-looking or overlapping elements. point
Such factors can lead to image recognition failures clas- • Creating alternative paths to accommodate various GUI
sified as false negatives (the target images exist but are states
not found) or false positives (the images found are not the • Implementing recognition redundancy (multiple ways to
target images) (Garousi et al. 2017; Wiklund et al. 2017; identify the same element)
Nass et al. 2021; Yandrapally et al. 2014) [11–14]. • Building in wait mechanisms and retry strategies

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

Each of these strategies multiplies the code complexity, 5.1 General and applied models
often exponentially. For example, adding alternative paths for
each step in a process with n steps can lead to a combinatorial This subsection describes the differences between general
explosion of paths that must be coded and maintained. Sim- and applied models and illustrates how they contribute to a
ilarly, for each GUI element that might change appearance comprehensive understanding of GUI automation.
or position, developers must implement multiple recognition A general model provides a high-level, abstract framework
methods and fallback approaches. for understanding GUI automation fundamentals. It inten-
As automation tasks grow more complex, this trade- tionally omits implementation details specific to particular
off becomes increasingly problematic. The challenge of GUIs or tasks, focusing instead on essential features relevant
maintaining code that addresses all possible exceptions and across diverse scenarios. Section 5 defines several general
variations quickly outweighs the benefits of automation models: the Overall Model of GUI automation, the Action
itself. This creates a practical ceiling on the complexity of Model, the state management model, the Transition Model,
tasks that can be reliably automated using process-based the path model, and the path traversal model.
approaches. In contrast, an applied model operates at a lower level of
The complexity-robustness trade-off explains why, despite abstraction and represents a specific automation application.
its potential benefits, GUI automation has seen limited It integrates the implementation-specific details necessary to
adoption in areas where reliability is crucial: The cost of automate a particular GUI environment and fulfill specific
developing and maintaining sufficiently robust solutions automation objectives. The applied model provides all the
becomes prohibitive as complexity increases. Model-based components needed to build a model-based GUI automation
GUI automation offers solutions to this fundamental prob- application in practice.
lem, and Sect. 12 explores the practical applications this The applied model in model-based GUI automation func-
unlocks. tions as a digital twin of the actual GUI environment. Digital
twins are virtual representations that typically model phys-
ical systems or environments. In this context, model-based
GUI automation presents an atypical case, as it involves cre-
ating a digital model of a digital environment. While general
5 Formal mathematical model of GUI models provide the theoretical framework, the applied model
automation instantiates this framework with specific details, creating a
virtual representation that can predict and interact with the
Having established the fundamental challenges of script GUI environment in the same way a traditional digital twin
fragility and the complexity-robustness trade-off inherent in would represent a physical system.
traditional, process-based GUI automation, this section intro-
duces a formal mathematical model designed to overcome 5.2 Theoretical foundations
these challenges. The core of this model-based approach is
a shift from writing rigid, sequential procedures to creating Model-based GUI automation addresses unpredictable envi-
an explicit, navigable map of the GUI environment, referred ronments using principles from robotics and human cogni-
to as the State Structure. This explicit representation allows tion. A model-based GUI automation application comprises
an automation framework to find its own paths and recover three key components: a map of the environment, a plan-
from errors dynamically, much as a human user would adapt ning mechanism, and an automation agent. This structure
to unexpected changes. mirrors human problem-solving, combining understanding,
The formal model presented here captures the essential planning, and execution. In its use of graph-based models
elements and dynamics of these GUI interactions, providing and heuristics, it navigates digital interfaces like robots tra-
a rigorous foundation for this new automation paradigm. It verse physical spaces, allowing for dynamic pathfinding and
formalizes key concepts such as GUI states, actions, tran- adaptability.
sitions, and path traversal while directly accounting for the In their paper “Bounded Rationality in Problem Solv-
stochasticity of GUI environments. By offering precise def- ing,” Langley et al. (2014) discussed how humans approach
initions and relationships between these components, the problem-solving in scenarios with incomplete information
model enables clearer reasoning about automation processes [22]. Situations, understood by the brain as collections of
and lays the theoretical groundwork for practical imple- symbols, are organized into a directed graph. Humans use this
mentation in frameworks like Brobot. This model serves conceptual graph of situations to imagine the steps required
as the basis for analyzing the advantages of model-based to reach a goal. A different type of symbol, operators repre-
GUI automation in terms of robustness and code complexity, senting actions, is employed to move from one situation to
which are explored in subsequent sections. another.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

Langley et al. divided knowledge into two categories: • τ , the Transition Model, for executing transitions between
domain knowledge and strategic knowledge. Situations and states;
available actions define domain knowledge and are specific • §, the Path Traversal Model, used for moving within the
to the problem. In model-based GUI automation, this cor- GUI environment;
responds to the GUI environment specific to an automation
task. Strategic knowledge is problem-agnostic and allows the
brain to build, reorganize, and traverse the directed graph of The visible GUI  contains:
situations and actions [22]. This is represented in model-
based GUI automation as pathfinding, path traversal, state
management, and action execution. Strategic knowledge • the scene, or the pixel output of the screen;
allows for awareness and manipulation of the GUI environ- • E  = f () ⊆ E, the set of all GUI elements in the
ment. Since strategic knowledge is problem-independent, its visible GUI.
implementation should be the responsibility of a framework.
In this context, the domain knowledge is represented by a
The State Structure  = (E, S, T ) is defined by:
problem-specific entity called the State Structure (see Fig. 4).
The State Structure provides a complete map of the prob-
lem space. It describes the specific GUI environment to be • E = {e1 , e2 , . . . , en }, the set of all GUI elements selected
automated—most importantly, what exists in this environ- to model the environment (images, regions, locations,
ment and how to move from point A to point B. etc.);
• S, the set of all GUI states:
5.3 The overall model
– Each state s ∈ S is a subset of E.
The Overall Model serves as a comprehensive framework – Multiple states can be active simultaneously; thus,
that unifies the fundamental components of model-based the visible GUI at a particular time can be described
GUI automation into a cohesive system. It specifies how the as a set of active states S ⊆ S.
automation system perceives the GUI, manages states, per- – s ∈ S if and only if s ∩ E  = ∅.
forms actions, navigates through the interface, and responds • T , the set of all transitions t between states.
to dynamic changes.
Figure 5 shows the critical dependencies between compo-
nents of the Overall Model, separated into domain knowledge The GUI environment, the set of all possible screens dur-
and strategic knowledge. The Action Model (a) bridges both ing an activity, is represented in the model by the set of GUI
domains—it receives input from and modifies the visible elements (E) organized into GUI states (S). The current scene
GUI (), provides the building blocks for transitions in the (), the screen at a specific time, provides the real-time data
State Structure (), and takes state elements from  as input. necessary for the automation system to perceive, interpret,
Actions and transitions provide information to the State Man- and interact with the GUI. Since the GUI environment is
agement system (M), which tracks active states. Given the conceptual and must be abstracted to finite sets of elements
set of active states, the Path Traversal Model (§) navigates and states, only its proxies - elements, states, and scenes—are
the GUI with help from ’s knowledge of transitions and included in the model.
graph of the GUI environment. A state in model-based GUI automation is a collection of
This architecture separates environment representation related GUI elements. State objects often are grouped spa-
(what exists) from interaction mechanisms (how to navigate), tially or appear at the same time. Objects used together in a
allowing the framework to handle strategic knowledge while process are likely candidates for belonging to the same state.
applications focus on domain knowledge and business logic. However, these configurations are not absolute rules, and
The Overall Model of GUI automation is a tuple (, , the definition of a state is subjective. A state has a meaning
a, M, τ, §) comprising: within the automated environment that can vary depending
on the automation goals, and a state configuration should
• , representing the visible GUI; make sense in the context of the automation application.
• , the State Structure, an abstraction of the GUI environ- In Fig. 6, the left side of the screen has a browser open
ment; to an AI chatbot, and the right side has a different browser
• a, the Action Model, for performing individual actions with a spreadsheet. These spaces can be clearly defined as
on the GUI; separate states since actions performed on one browser will
• M, the State Management system, responsible for main- not affect the other. The orange box around the open menu
taining the active states. in the spreadsheet defines a third state.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

Fig. 4 A simple State Structure, the expression of domain knowledge for a specific problem

Fig. 5 The overall model: (, , a, M, τ, §)

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

• Transition Model (τ ): Manages the execution of transi-

tions between states;
• Path Traversal Model (§): Enables navigation through the
GUI environment.

The application utilizes these components via API calls,

eliminating the need to implement complex logic such as
pathfinding.

Automation instructions ()

Fig. 6 A GUI screenshot with three states

Automation instructions define the application’s business

5.3.1 A practical example: the DoT app logic, specifying the tasks to be executed. This logic is pri-
marily implemented in the SaveLabeledImages class,
To make the abstract components of the Overall Model which contains the main loop for navigating to islands and
(, , a, M, τ, §) more concrete, I use the DoT app, an saving images. Additional functionality is provided by helper
experimental application designed to automate tasks in the classes such as GetIslandType, which supports classifi-
mobile game Dawn of Titans. This app will serve as a running cation and retrieval tasks.
example to illustrate how the theoretical models are realized
in practice. The following is a breakdown of how the app’s 5.3.2 Design principles and component necessity
components map to the model.
The Overall Model’s architecture addresses fundamental
challenges in GUI automation through a separation of con-
Domain knowledge: the state structure Ä
cerns principle. Each component serves a critical role that, if
absent, would reintroduce the fragility and complexity prob-
The State Structure represents the domain knowledge of
lems of process-based approaches:
the automation system. It provides a structured map of the
problem space, defining the states, elements, and transitions
within the GUI environment. Visible GUI (4)

The critical distinction of this model is its treatment of the

• States (S): These are represented by the Home, World, Visible GUI () as an explicit component and a verifiable
and Island classes, which model discrete sections of ground truth. Unlike process-based approaches that use the
the game’s interface; screen as a transient input for sequential actions, this model
• Elements (E): These are the specific images and regions enables a crucial feedback loop: it continuously compares
defined within state classes, such as the toWorld its internal belief of active states (S ) against the reality of
Button in the Home state or the nameRegion in the the screen (). This verification allows the system to detect
Island state; unexpected events and confirm transition outcomes. Lack of
• Transitions (T ): These are defined in the Home systematic verification is one of the main causes of fragility in
Transitions, WorldTransitions, and Island process-based approaches, where automation often assumes
Transitions classes, which handle the logic for mov- an expected GUI state.
ing between states.
State structure (Ä)
Strategic knowledge: the framework F
Process-based approaches embed environment knowledge
Strategic knowledge is provided by the framework, which within action sequences, creating tight coupling. The State
implements problem-independent automation capabilities. Structure externalizes this knowledge, enabling:
The following components ensure efficient GUI interaction:
• Independent testing of environment representation;
• Action Model (a): Defines atomic operations for inter- • Reusable state definitions across multiple automation
acting with GUI elements; tasks;
• State Management (M): Maintains the set of active states • Clear separation between “what exists” (states) and “how
and their relationships; to navigate” (transitions).

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

Without , environment knowledge remains scattered across of active states gives the system an accurate understand-
procedures, making maintenance exponentially complex as ing of its current context from which to attempt recovery.
the automation grows. • Enabling Dynamic Pathfinding: Evaluating alternative
paths requires knowing your current location. When a
Action model (a) step in a rigid, sequential process fails, the script’s “loca-
tion” becomes invalid. It has no reliable, independent
While process-based tools also perform actions, they lack a knowledge of the GUI’s state to begin a new path. The
unified model for action execution and result interpretation. State Management system acts as the reliable “You Are
The Action Model establishes the following: Here” marker on the map. After a failed transition, the
Path Traversal Model can query M for the current, valid
• Interface Contract: The model defines a standardized set of active states (S ) and use that precise starting point
result structure (ra ) for every action. This creates a consis- to find a new, viable path to the target.
tent interface contract, ensuring that the success or failure
of any action—regardless of its type—can be interpreted The State Management system maintains an explicit,
uniformly by other system components. It allows the state dynamic awareness of all active states (S ), enabling robust
manager, for example, to reliably process the outcome adaptation and recovery. Without M, the system cannot reli-
without needing custom logic for each action. ably reason about its current position in the GUI environment,
• Implementation Scope: The model abstracts environ- a prerequisite for any complex or long-running automation
mental stochasticity () from an action’s internal logic. task.
This deliberately narrows the implementation scope of
each action to its core task (e.g., clicking a coordinate, Transition model () and path traversal (§)
finding an image). The action’s code is not responsible
for handling unpredictable events like pop-up windows The Transition Model manages the execution of defined
or network lag; instead, the responsibility for handling action sequences (τ ) that move between states. The Path
the consequences of such events is elevated to the frame- Traversal Model (§) is responsible for the higher-level strate-
work’s state management and path traversal components, gic task of finding a viable sequence of transitions to navigate
preventing code duplication and centralizing error recov- from a current state to a target state. This separation of defin-
ery logic. ing a transition from finding a path is a key architectural
choice.
State management (M)
Design principle summary
Process-based automation tracks the GUI’s state implicitly
through the current position in the action sequence. For exam- In summary, the design’s core principle is the explicit separa-
ple, if line 15 of the script is executing, the GUI must be on tion of concerns. Domain-specific knowledge is externalized
the “Payment Details” screen. This method fails in complex into an explicit map of the GUI environment (the State Struc-
scenarios because it lacks a formal, explicit representation ture, ), while the problem-agnostic strategic knowledge
of the GUI’s state. The State Management system (M) is (how to find, act, and navigate, implemented in a, M, τ, §) is
necessary to overcome these challenges: encapsulated within the framework. This separation allows
the framework to handle reusable, complex logic, while
• Handling Multiple Active States: A process-based tool applications focus on defining the environment and their own
lacks an architectural concept to represent the combina- business logic.
tion of {State A, StateB} as a single, coherent “world
view.” It only knows that two separate images were 5.3.3 Addressing the robustness-code complexity trade-off
found. A dedicated State Management system maintains
an explicit set of active states (S ). This allows the frame- This architecture directly resolves the robustness-code com-
work to reason holistically, for example, by looking for plexity trade-off identified in Sect. 4.2:
transitions that are available from any of the currently
active states, something that would require complex, cus- Robustness through modularity
tom, and brittle conditional logic to handle manually in
a process-based script. Robustness is achieved through the architecture’s modular-
• Recovering from Unexpected Events: The State Man- ity and its feedback mechanisms. Because the State Structure
agement system adjusts the active states using the results () is decoupled from execution logic, GUI changes often
of transitions and certain actions. Having an updated set require updating only a single, localized state or transition

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

definition. Furthermore, the explicit state tracking provided From these action results, state information is derived by
by the State Management system (M) combined with verifi- the function:
cation against the Visible GUI () allows the system to detect
and recover from unexpected events, rather than failing when Sa : ra → P(S × {T r ue, False}) (2)
the GUI diverges from an implicit assumption.
State information (Sa )
Complexity reduction through abstraction
The results of an action provide information about GUI ele-
ments that can be used to determine which states are active
The path traversal component (§) eliminates the need
and which are inactive. For example, when an observation
to explicitly code every possible path. This fundamen-
action successfully locates an image, this may indicate that
tally reduces development complexity. The process-based
the state associated with that image is active. Conversely,
approach requires manually accounting for a number of paths
when an image disappears from the GUI, this could suggest
that grows exponentially. In contrast, the model-based effort
that its associated state is no longer active.
is polynomial. The developer must define the n states and,
assuming an average of m transitions per state, approximately
The visible GUI (4)
n × m transitions. This reduces development complexity to
a manageable polynomial scale, O(nm).
The scene captures the visible state of the GUI at any
For the example in Sect. 10 with n = 30 states and m = 5
point during the automation process. Unlike state abstrac-
transitions per state, this is the difference between coding
tion, which captures higher-level information, the scene is
a number of paths on the order of trillions versus defining
a pixel-level representation of the entire GUI, including all
approximately 30 + (30 × 5) = 180 states and transitions.
elements, background images, and white space.
This represents a shift from a practically impossible task to
The detailed information in the scene is crucial for ana-
a well-defined and scalable one.
lyzing the effects of individual actions and determining the
next feasible steps in an automation process. By using the
5.4 The action model (a) scene, the model incorporates both expected and unexpected
changes, allowing the automation system to adapt dynam-
The Action Model serves as the fundamental building block ically to variations in the GUI. This approach is especially
of GUI automation, defining the atomic operations that can valuable in stochastic environments where GUI changes may
be performed on the graphical user interface. It encapsulates not follow predictable patterns.
individual interactions, such as clicking a button or entering
text, that collectively enable task automation. Environmental stochasticity (2)
An atomic action is a tuple a = (oa , E a , ζ ) comprising:
Unexpected environmental events, modeled by , can sig-
• oa , parameters or options associated with the action; nificantly influence the execution of GUI actions during
• E a , the elements acted on; automation. These events can alter the GUI unpredictably
• ζ , the success function specific to the action type. before or during action execution, leading to failures or unin-
tended behaviors. For example, consider a scenario where
a pop-up window appears and obscures a button that the
The action function automation intends to interact with. If this pop-up emerges
before an observation action attemps to locate the button, the
f a : (a, , ) → ( , ra ) (1) action will fail because the button is no longer visible. If the
pop-up surfaces before a click action is performed on the
button, it will act on the pop-up instead of on the intended
comprises:
button.
When unexpected events cause discrepancies between the
• a, the action, which can modify or observe the GUI; application’s perception and the reality of the GUI, the reli-
• , the visible GUI; ability of automation diminishes. In the previous example
• , environmental stochasticity: all external events and of the click action, the system believes it has successfully
environmental factors that can unpredictably influence clicked the button based on its observations. In reality,
the GUI state at the time of action execution; the action did not affect the intended target, and the GUI
•  , the resulting GUI after the action is performed; diverges from what the system anticipates. This misalign-
• ra , data describing the action’s results. ment means that subsequent actions, which depend on the

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

expected outcome of the click, may fail or behave unpre- – a success or failure status;
 de f
dictably. Therefore, maintaining synchronization between S if transition succeeds
– state information St = t .
the application’s knowledge of the GUI and the real GUI ∅ if transition fails
is crucial. Implementing mechanisms to detect and reconcile
discrepancies—such as verifying the GUI states after critical The process function
actions or handling exceptions when actions do not produce
expected changes—enhances the robustness and reliability f A : (A) → ( ) (5)
of automation.
builds on the action function f a by iteratively applying it
5.5 The state management model (M) to each action in the sequence, where each action operates
on the scene () produced by the previous action. The final
The state management model M is responsible for maintain-
output ( ) represents the cumulative effect of applying all
ing and updating the set of active states based on information
actions.
received from the Action Model and the Transition Model.
It ensures the GUI automation system has an accurate repre-
State information in transitions
sentation of the GUI’s current state.
The state management model is defined as M = (S ),
The state information St provides explicit state change details
where S ⊆ S is the current set of active states.
that complement the information inferred from actions.
The state management function processes updates from
Although not strictly necessary for state management, tran-
either actions or transitions:
sitions can significantly streamline the process through St by
 containing known relationships between states. For exam-
f M : (S , Sa , St ) → (S ∪ S+ ) \ S− = S (3)
ple, if state A always disappears when state B is activated,
St can include this rule directly in the transition, eliminating
where:
the need for additional actions to verify the absence of state
A. Without such rules included in St , the automation sys-
• Sa and St are state information from actions and transi-
tem would need to rely solely on action execution and Sa to
tions respectively, each a set of (s, value) pairs where
verify the presence or absence of related states. This makes
s ∈ S and value ∈ {T r ue, False};
St particularly valuable for optimizing state management in
• S+ = {s ∈ S | (s, T r ue) ∈ (Sa ∪ St )} are states to
GUI environments with well-defined state relationships.
activate;
• S− = {s ∈ S | (s, False) ∈ (Sa ∪ St )} are states to
deactivate; Transition set and relations
• S is the resulting set of active states.
The transition relation
5.6 The transition model ()
δ⊆ S×T (6)
The Transition Model defines how sequences of actions lead
to transitions between different GUI states. defines which transitions are accessible from which states,
de f where:
A transition is a tuple t = (A, St ) comprising:

• A, a process or sequence of actions A = (a 1 , a 2 , . . . , a n ) • S is the set of all possible states;

executed as part of the transition; • T is the set of all transitions;
de f
• St ∈ P(S × {T r ue, False}), the intended state infor- • an element (si , t j ) ∈ δ indicates that transition t j is acces-
mation if the transition succeeds. sible from state si .

The transition function 5.7 The path traversal model (§)

) → ( , rt )
de f
f τ : (A, St (4) The Path Traversal Model’s primary responsibility is to tran-
sition the GUI to the desired target state. It does so by
processes the transition, where: identifying appropriate paths in the State Structure, select-
ing a path to follow, and executing the path. This process
•  is the resulting GUI after the process is executed; repeats until the target state is reached or all available paths
• rt is the transition result, containing: have failed.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

Within the Path Traversal Model, a path is defined as a • State Cost Function: c S : S → R, assigns a cost to each
tuple ρ = (Sρ , Tρ ) comprising: state s ∈ S;
• Transition Cost Function: cT : T → R, assigns a cost to
• Sρ = [s0 , s1 , . . . , sn ], the sequence of states, where: each transition t ∈ T ;
• Path Cost:
– s0 ∈ S (the first state is in the set of active states);
– sn = starget (the last state is the target state).

n−1

• Tρ = [t0 , t1 , . . . , tn−1 ], the sequence of transitions, c(ρ) = [cT (ti ) + c S (si+1 )] (9)
i=0
where:
– ti ∈ T (each transition is in the set of all transitions); • Path Selection:
– (si , ti ) ∈ δ (a transition is accessible from its begin
state);
de f
– (si+1 , T r ue) ∈ Sti (the transition’s end state is acti- ρ ∗ = arg min [c(ρ)] . (10)
ρ∈P
vated by the transition if it succeeds).

where:
Path traversal is a tuple § = (, S , starget , H ) comprising:

• c S (si ) is the cost associated with state si ;

• , the State Structure;
• cT (ti ) is the cost associated with transition ti ;
• S , the set of active states;
• ρ ∗ is the optimal path selected for execution;
• starget ⊆ S, the state the automation aims to reach;
• P is the set of all possible paths.
• H : P → R, the heuristic used for path selection. For
any path ρ, the heuristic value is h ρ .
These costs serve as the heuristic (H ), reflecting execu-
The pathfinding function delivers all valid paths from the tion difficulty, time requirements, or likelihood of failure.
active states to the target state: This approach has similarities with the A* algorithm in pri-
oritizing lower-cost paths while dynamically adapting to
environmental feedback (Hart, Nilsson, & Raphael 1968)
f path f ind : (, S , starget ) → P(ρ) (7) [25].

where P(ρ) is the set of all possible paths from the active
states to the target state.
5.8 The applied model (App)
The path traversal function is
An application implementing model-based GUI automa-
tion can be modeled as components representing domain
f § : (, S , starget , H ) → ( ) (8) knowledge, strategic knowledge, and automation instruc-
tions. Domain knowledge is modeled by the State Structure
where  is the resulting GUI after path execution attempts. of a specific GUI environment. Strategic knowledge, contain-
The path traversal function f § internally uses the pathfind- ing algorithms that attempt to understand and manipulate this
ing function f path f ind to determine the set of possible paths environment, is environment-independent and implemented
P(ρ), applies the heuristic H to select an optimal path, and by a framework. The automation instructions are the business
then executes the selected path. During execution, each tran- logic for the specific automation application.
sition ti for i > 0 executes only if ti−1 is successful. The applied model is a tuple App = (domain, strategic,
In robotics, heuristics determine the expected cost of mov- instructions) = (, F, ι) comprising:
ing between locations and are used to select the transitions
and paths to pursue (Ferguson et al. 2005) [23]. Pohl (1970)
• Domain knowledge : , the State Structure;
described a heuristic as “any device that aids search efficiency
• Strategic knowledge : F, the framework used, a specific
by either restricting the region searched or appropriately
implementation of (a, M, τ, §);
ordering the search” [24]. Heuristics are valuable for path
• Automation instructions : ι.
traversal, as there may be many potential paths from which
to choose.
Path cost depends on individual state and transition costs These are the components needed to create a model-based
in the path: GUI automation application in practice.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

5.9 The visual API model provides the necessary implementations for the abstract func-
tions and processes in the general models, bridging the gap
The Visual API is an interface between automation agents between theory and application.
and GUI environments, defined as a tuple (, F) comprising The Brobot framework is the first implementation of
a State Structure () and a framework (F). model-based GUI automation principles. Throughout this
Application programming interfaces (APIs) that com- paper, it serves a dual purpose: demonstrating how the
municate with external applications are omnipresent in abstract models can be realized in a functional system and
the software domain, particularly in artificial intelligence providing empirical evidence that supports and validates the
research where reinforcement learning agents commonly theoretical foundations.
control games. In systems like AlphaGo (Silver et al. 2016) The practical utility of this Visual API concept, particu-
[26] and Starcraft (Vinyals et al. 2017) [27], game envi- larly for enabling complex game automation and sophisti-
ronments are represented without information loss, allowing cated reinforcement learning research, is further discussed
researchers to focus on algorithmic development instead of in Sect. 12.
the mechanics of environment manipulation. For example,
Starcraft provides an explicit API for retrieving detailed 6.1 The action implementation
information about the game and for moving game objects.
Most games do not provide explicit APIs, and creating a The Brobot framework implements common actions such
reinforcement learning agent for these games requires visual as click, t ype, drag, f ind, and move. In the action func-
GUI automation. In these scenarios, the Visual API takes the tion f a : (oa , E a , ζ, , ) → ( , ra ), action parameters
role of an explicit API, allowing the automation instructions (oa ) can include modifiers such as maximum search time,
(ι) to focus on strategy and business logic. minimum image similarity score, and mouse-hold duration.
For example, imagine using this system to play a game Brobot provides predefined success criteria (ζ ) for each
of chess through a standard GUI. The RL Agent, acting as action type and allows developers to implement custom eval-
the chess engine, would decide on a strategic move, such as uation logic.
moving a pawn from e2 to e4. It would send this high-level The following code represents a GUI action that searches
command as an RL Action, like move(piece: pawn, for two images. In Brobot, the ActionOptions variable repre-
from: e2, to: e4), to the Automation Instructions. sents oa , the ObjectCollection variable E a , and the Matches
The automation instructions (ι) would then translate this variable ra .
strategic command into a sequence of concrete GUI actions
for the Visual API, such as: ActionOptions findOptions = new
[Link]()
• click(e,2,pawn) .setSimilarity(.90)
• click(e,4,empty) .setMaxWait(2)
.build()
ObjectCollection elementsToFind = new
Finally, the Visual API (the framework F and State Struc- [Link]()
ture ) would handle the low-level mechanics of locating the .withImages(image1, image2)
relevant GUI elements on the screen (e.g., finding the pawn .build()
Matches findResults = [Link](
at coordinates matching “e2” and the empty square at ‘e4”) findOptions, elementsToFind);
and executing the clicks. This clear separation allows the RL
agent’s logic to remain focused on game strategy, completely
decoupled from the complexities of GUI manipulation.
Figure 7 illustrates how the components of a model-based 6.2 State management in Brobot
GUI automation application relate to reinforcement learning,
demonstrating how the Visual API bridges the gap between Recall from the general models that:
GUI automation and AI research.
• E is the set of all GUI elements in the environment
• S is the set of all states
6 Implementation • S is the set of currently active states
• Sa is the state information produced by actions
The previous section presented general models that define • ra represents the action results
the fundamental components and processes of model-based
GUI automation and an applied model that takes a specific Building on these definitions, for Brobot’s implementa-
framework as input. This section explains how the framework tion, I define:

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

Fig. 7 Model-based GUI

automation in the context of
reinforcement learning

• For each element e ∈ E, there is a function state(e) = se , where:

which maps an element to its parent state.
• For any observation action a executed on a set of elements (1) States are added to S when they are marked active in
E a ⊆ E, let E found ⊆ E a be the set of elements found either Sa or St
successfully. (2) States are only removed from S when explicitly marked
inactive in St (transition-based updates)
Scenario 1: element found (3) No state is removed based solely on action results (Sa
never contains pairs with False)
When an element e ∈ E a is found (that is, e ∈ E found ),
Brobot adds the corresponding state to Sa as active: For St (transition-based state updates), state changes occur
only when a transition fully succeeds. Although failed tran-
∀e ∈ E found : (se , True) ∈ Sa (11) sitions do not trigger their defined state changes, individual
actions within them may still modify the state set through Sa .
Scenario 2: element not found or not searched
6.3 The transition implementation
A state is considered active if at least one of its elements is
found. It is not marked as inactive based on missing elements: In Brobot, transitions are separated into two parts: From-
Transitions and ToTransitions (see Fig. 8). A FromTransition
∀e ∈ E a \ E found : (se , False) ∈
/ Sa (12) handles the initial actions for moving from the current state
to another state, and a ToTransition completes the transition.
This includes elements that were searched for but not located, For example, to go from State 1 to State 6, the framework
as well as elements that were not part of the search at all. Indi- first calls FromTransition 1 → 6 and then ToTransition →
vidual images can reliably signal a state’s existence but not its 6. The ToTransition contains actions that always occur when
absence; therefore, Brobot relies on transitions to deactivate activating the state, regardless of which state started the tran-
states. sition.

State management function implementation 6.4 The path traversal implementation

I formalize the state management function implementation When everything goes as planned, only one path is neces-
in Brobot as: sary for a successful path traversal. If a transition fails, the
active states may have changed and in this case all paths are
recalculated.
f M (S , Sa , St ) = (S ∪ { s ∈ S | (s, True) ∈ Sa ∪ St })
In Fig. 9, the rectangles represent states, and the circles
\ { s ∈ S | (s, False) ∈ St }. (13) represent transitions. When the first transition (C1) on the ini-

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

Fig. 8 Transitioning requires a FromTransition and ToTransition

Fig. 9 Dynamic path selection with transition failures

tially chosen path fails, the system selects a new path: star t tation website ([Link]
→ A1 → 2 → I1 → 9 → I2 → end. Subsequently, when basics/live-automation).
transition I1 also fails, the system selects another alternative The images in the DoT app are cut from screenshots of
path: 2 → H1 → 6 → G1 → end. This demonstrates how the GUI, and some of these images are shown in Fig. 11. The
the path traversal implementation handles multiple failures screenshots in Figs. 12, 13, and 14 show the three states in
through continuous re-evaluation and adaptation. the DoT app.

The DoT app as an implementation of the formal

7 Case study: implementing the DoT app models

As introduced in Sect. 5, the DoT app serves as a practical The DoT app demonstrates a practical implementation of the
implementation of the model-based approach. This section Applied Model App = (, F, ι) as defined in Sect. 5.8.
explores its design and key code components in greater detail. Domain knowledge () is represented through the game’s
The DoT app identifies and captures images of islands GUI environment, with the Home, World, and Island
in the game environment. Each image is then saved with classes modeling discrete sections of the interface as states,
a filename that denotes the island type (e.g., farms, mines, while the transitions between these states are managed
forests) along with a unique identifier. Figure 10 presents a by the HomeTransitions, WorldTransitions, and
screenshot of the application in operation, showcasing three IslandTransitions classes. Strategic knowledge (F)
key elements: the emulated game interface (middle left), the is provided by the Brobot framework: actions, transitions,
console output (bottom left) and a file directory (right) where path traversal, and state management appear as API calls in
the captured images are saved. The entire process, from the DoT app. Automation instructions (ι) comprise the busi-
game navigation to image capture and storage, is executed ness logic that accomplishes the specific task of collecting
autonomously by the DoT app. The video demonstration of and labeling island images. The complete implementation
this automation process is available on the Brobot documen- code is found in Appendix II.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

Fig. 10 A screenshot from the recording of live automation

8 Stochasticity in GUI automation

Stochasticity in the GUI environment is caused by events

that include the variable timing and appearance of GUI ele-
ments, unexpected application behavior, and interference by
other applications. Although less pronounced than in phys-
ical environments, this stochasticity makes process-based
GUI automation error prone, especially for complex tasks.

8.1 Empirical evidence

To empirically demonstrate stochasticity, I conducted an

experiment using optical character recognition (OCR). Run-
ning an application to collect island images and recognize Fig. 11 DoT elements
island names yielded variations in text recognition for the
word “Lakes” (Table 2). This exemplifies the stochasticity of
even a single, seemingly simple GUI action. • A: Observation actions performed on the GUI;
• R: Automation results from these actions.

8.2 A formal model of stochasticity in GUI

The model focuses on observation actions because their
automation
outcomes, unlike those of click or type actions, are deter-
ministic once the visible GUI () is fixed.2 This separation
The effects of stochasticity on GUI automation are modeled
allows for modeling GUI uncertainty independently to action
using a tuple (, , A, R):
execution.

• : Environmental stochasticity: all events affecting the 2 The observation actions are assumed to be themselves deterministic
GUI since the last observation; (i.e., pattern matching). AI-based observation methods are not deter-
• : The visible GUI state; ministic.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

Fig. 12 The HOME state

Fig. 13 The WORLD state

Environmental Stochasticity () represents the cumu- Automation Results (R) are determined by the function
lative effect of all external events and environmental factors R = h(, a), where h :  × A → R maps an observation of
that have unpredictably influenced the GUI state since it was the visible GUI to a specific result. These outcomes can be
last observed. This includes factors such as delays in ele- categorized by how they affect automation success: correct
ment rendering, unexpected pop-up windows, background matches, false negatives, and false positives.
processes, and application malfunctions—all possible causes Using the total probability theorem, the probability of
for variations in the GUI’s current appearance that are not obtaining result r is:
directly triggered by the automation itself.
The Visible GUI () exhibits a probability distribu- 
tion over its possible appearances. For each appearance ξi , P(R = r ) = P( = ξ ) × 1[h(ξ, a)=r ] (14)
ξ ∈
P( = ξi ) quantifies the likelihood
 the GUI has exactly that
pixel composition, where ξi P( = ξi ) = 1. This distri-
bution reflects the influence of environmental stochasticity where 1[h(ξ,a)=r ] is the indicator function. Equation 14
on the GUI. effectively weights the deterministic outcomes by the prob-
Observation Actions (A) search for elements within the abilities of the corresponding visual outputs, capturing how
GUI. The outcome of an observation depends on the state of the stochastic nature of the GUI influences the automation
the visible GUI when the action is performed. performance.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

Fig. 14 The ISLAND state and the WORLD state

Table 2 OCR text recognition variations for the words Lakes and their
frequency counts
OCR variations of the word “Lakes”
Variant Count Variant Count Variant Count

Lakes 18351 Lakeg 7 Lcikes 72

Iakes 24 Lokes 84 Lalces 50
Lakcs 123 Lakas 8 La1ces 5
Lokcs 2 Lcikcs 1
Fig. 15 A simple automation process

8.3 The effect of stochasticity on process-based GUI

automation sequential actions {A, B, C}. Figure 16 shows an enhanced
version where the developer has attempted to improve robust-
The inherent stochasticity of GUI environments signifi- ness by manually coding three alternative actions {A1, A2,
cantly impacts the reliability of process-based automation. A3} for the first step, providing fallback options if the pri-
In sequential processes where each action must succeed for mary action fails.
the overall process to succeed, the probability of success- Given independent actions with success probability p, the
ful execution decreases exponentially with the number of success probability of each complete path through the graph
actions. For an environment where each action has a success is p 3 . One might incorrectly conclude that the probability of
probability p = 0.90, a sequence of 7 actions has a success the overall process in Fig. 16 succeeding is 1 − (1 − p 3 )3 ,
probability of p 7 = 0.478, making failure more likely than which would yield 0.98 for p = 0.90. This analysis fails to
success. account for the continuous nature of GUI automation.
To increase robustness when using process-based meth- In continuous automation, alternative paths are only avail-
ods, the primary recourse is to manually introduce alternative able at specific decision points. If action A1 fails, the system
actions or paths to accommodate potential failures. For exam- can attempt A2, but if action B fails after a successful A1,
ple, consider the simple process in Fig. 15 consisting of three the entire process fails without recourse to alternative paths.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

series:


n
mk (21)
k=1

where:

• n is the total number of steps in the original process;

• m is the number of alternative actions at each step (includ-
ing the original);
• k is the step number or level in the action sequence.

In the example with n = 3 steps and m = 3 alterna-


tives per step, this yields 3k=1 3k = 39 individual actions,
Fig. 16 A more robust process with alternative first actions as shown in Fig. 17. The exponential growth in this series
demonstrates a fundamental limitation of process-based GUI
automation: achieving high reliability requires an explosion
The correct analysis for the first layer considers three
in the number of actions that must be programmed and main-
mutually exclusive scenarios:
tained. This manual effort makes it impractical to build truly
robust solutions for complex tasks. In stark contrast, a model-
P(first layer success) = P(A1) + P(¬A1 ∩ A2)
based approach, as detailed in this paper, avoids this manual
+ P(¬A1 ∩ ¬A2 ∩ A3) (15) proliferation. By defining a State Structure (), the frame-
= p + p(1 − p) + p(1 − p) 2
(16) work can dynamically discover and traverse paths, including
alternatives, thereby handling complexity and stochasticity
= 3 p − 3 p2 + p3 (17)
without requiring the developer to explicitly code each vari-
ant. The pathfinding mechanisms inherent in the model-based
where:
framework (see Sects. 5.7 and 10) manage this adaptability
systematically.
• p is the probability of successful execution for each
action;
• A1, A2, and A3 represent the alternative actions at the
first step; 9 From implicit to explicit environment
• ¬A1 denotes the failure of action A1. representation

Since the success probability remains p for both the sec- The stochasticity inherent in GUI environments directly
ond and third layers, the overall success probability is: undermines the robustness of process-based approaches. This
fundamental limitation stems from a structural mismatch:
P(success) = P(first layer success) · p · p (18) process-based GUI automation forces an inherently state-
based problem into sequential procedures. Since GUIs are
= (3 p − 3 p + p ) · p
2 3 2
(19) built for humans, who understand the GUI environment as
= 3 p3 − 3 p4 + p5 (20) collections of states, elements, and transitions, treating the
GUI as a series of actions is somewhat unnatural. Hidden
For p = 0.90, this yields a success probability of approxi- within the procedural code and fragmented across the exe-
mately 0.809, substantially lower than the incorrect intuitive cution paths of process-based GUI automation is an implicit
estimate of 0.98. model of the GUI environment. Developing GUI automa-
tion with process-based methods means interacting with the
Achieving high reliability implicit model, requiring a mental representation of the entire
state space. In this section, I demonstrate how these implicit
To achieve high reliability (e.g., 99%), the process requires models can be systematically transformed into explicit state
alternative paths for every action in the sequence. For a representations, aligning automation design with the natural
process with n original actions, where each action has m structure of GUI environments and human cognitive models.
alternatives (including the original), the total number of indi- Figure 18 modifies the process-based tree in Fig. 17 by
vidual actions in the robust algorithm follows the geometric adding a GUI state before and after each action. In Fig. 18,

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

Fig. 17 An action tree with alternative actions for each automation step

Fig. 18 States are added to the action tree before and after each action

the repetition of actions B1, B2, and B3 suggests their orig-

inating states (S2, S3, S4) are the same state. In Fig. 19, S2,
S3, and S4 are consolidated to state T2. This transformation
reveals redundant states and clarifies transition paths, making
previously hidden assumptions explicit.
Figure 20 shows the tree with descriptive state names.
Actions transitioning between the same states (e.g., all paths
from Home to World) can be combined into a single pro-
cess. Process A represents the combined actions A1, A2,
and A3, implemented as a series of conditional returns. Veri-
fied states (shown with dark backgrounds) require successful
image recognition, and transitions represent action sequences
between verified states, such as {B, C} defining the transition
from Home to World.
Explicit modeling transforms GUI automation develop-
ment by shifting from sequential to structural thinking. The
State Structure () is built before the automation instructions
(ι), prioritizing environmental representation over procedural
sequences. This provides two key advantages: First, modular
design allows components to be built, tested, and debugged
independently, with changes and troubleshooting localized
to specific states or transitions rather than requiring analy-
sis of entire action sequences. Second, the clear definition of
states through their GUI elements and the explicit mapping Fig. 19 The action tree after consolidating duplicate states
of transitions aligns with the GUI’s actual structure.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

Fig. 20 The tree after

consolidating states and actions

Fig. 21 The DoT app’s State

Structure

This transformation process demonstrates that process-

based GUI automation contains implicit models that can be
systematically converted to explicit representations. Process-
based and model-based approaches are not fundamentally
different paradigms, but rather different expressions of the
same underlying GUI environment—one implicit and frag-
mented, the other explicit and cohesive.

10 Robustness and code complexity

Model-based GUI automation offers significant advantages

over process-based techniques in robustness and code com-
plexity. A mathematical analysis of pathfinding complexity
reveals the fundamental differences between these approaches,
particularly as the number of states and transitions grows.

10.1 Path implementation: a practical comparison

using the DoT app

The simplicity of the DoT app makes it a good example Fig. 22 The process-based code in SikuliX
for comparing process-based and model-based path imple-
mentation. The paths in Fig. 21 would be implemented by
creating three states and two transitions. Figure 22 shows the Units state. In the model-based app, this requires adding
process-based implementation of the three paths in SikuliX. one new state and one new transition. Figure 23 shows the
To demonstrate the effects on code complexity of an new State Structure. The process-based app would require a
increase in automation complexity, we can introduce a new method for each of the three new paths.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

Fig. 23 The DoT app with an additional state

10.2 Path analysis as the probability that an edge leads to an unvisited node when
k nodes are already in the path. Although exact probabilities
To analyze the complexity implications of the two automa- can be calculated for specific graphs (Fig. 24), p(k) provides
tion approaches, I first define a formal model of paths an average across all possible configurations of type {n, c}.
through GUI states. I then use this model to demonstrate The average number of paths can be calculated by:
why process-based approaches become impractical at scale,
while model-based approaches remain manageable. 2 = p(1) × n × c = n × c (23)
Let G = (V , E) be a directed graph where:
∀k > 2, k = c × p(k − 1) × k−1 (24)
 n
• V is a finite set of nodes, with |V | = n ≥ 2; = (25)
T k
• E ⊆ V × V is the set of directed edges. k=2

Properties: where the summation in Equation 25 is performed over all

possible path lengths k, ranging from k = 2 nodes up to
• Out-degree: ∀v ∈ V , |(v, u) ∈ E : u ∈ V | = c where k = n nodes.
1 ≤ c ≤ n − 1; For concrete analysis, consider three scenarios, where
• No self-loops: ∀v ∈ V , (v, v) ∈ / E; nodes are states and edges are transitions:
• Connectivity: ∀u, v ∈ V , there exists a sequence of ver-
tices v1 , ..., vk where v1 = u, vk = v, and (vi , vi+1 ) ∈ E
(1) Simple automation task (n = 4, c = 2):
for 1 ≤ i < k.
• 2-state paths: 2 = n × c = 8;
Definition 1 (Path) A path ρ is a sequence of distinct nodes • 3-state paths: 3 = c × p(2) × 2 = 2 × 0.67 × 8 =
(v1 , ..., vk ) where (vi , vi+1 ) ∈ E for 1 ≤ i < k and k ≥ 2. 10.72;
• 4-state paths: 3 = c × p(3) × 3 = 2 × 0.33 ×
Let k = f (n, c, k) be the estimated number of paths of 10.72 = 7.08;
length k (in number of nodes) in the graph. Let T = f (n, c) • Total paths ≈ 25.8.
be the estimated total number of paths of all possible lengths,
from length 2 up to length n (nodes), where: (2) Moderate automation task (n = 30, c = 5):
• Total paths ≈ 6.36 × 1012 .
• n is the number of nodes in the graph;
• c is the number of outgoing edges per node; (3) Complex automation task (n = 500, c = 10):
• k is the length of the path in number of nodes (i.e., a • Path count exceeds my spreadsheet app’s numerical
path of length k has k nodes and k − 1 edges), where capacity of 1.79E+308.
2 ≤ k ≤ n.
Figure 25 shows the exponential growth in paths for the
To estimate paths in graphs with unknown connections, I
moderate automation task. Each path requires its own method
define:
in process-based GUI automation, making implementation
n−k impractical for nontrivial applications. This programming-
p(k) = (22) time complexity could be described as NP-hard: Not Pro-
n−1

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

Fig. 24 A path analysis of a graph of type {n=4, c=2}

gramming this! This is too hard!3 Model-based GUI automa- An atomic action remains defined as:
tion handles this complexity through framework-level pathfind-
ing, enabling the development of complex applications as a = (oa , E a , ζ ) (26)
detailed in Sect. 12.
The mock action function replaces the live GUI and environ-
mental stochasticity with the set of active states:
11 Integration and unit testing in GUI
automation f amock : (a, S ) → (ra , Sa ) (27)

A fundamental innovation of model-based GUI automation

The GUI element in mock runs includes its Action History:
is enabling systematic testing of the automation code itself.
Unit testing and integration testing, cornerstones of profes-
sional software development, have traditionally been absent e ∈ E a = (oe , AH ) (28)
from GUI automation due to the challenges of testing code
that interacts with dynamic interfaces. Model-based GUI where:
automation introduces formal mechanisms that make both
integration and unit testing of automation code possible, • S is the set of active states;
aligning GUI automation development with established soft- • ra gives the results of the action;
ware engineering practices. • Sa contains changes to the active states;
• E a is the set of all elements acted on by a;
11.1 Integration testing through mocking • oe contains the parameters of the element (e.g., filename
for images, bounding box for regions);
In model-based GUI automation, integration testing is • AH = AS1 , AS2 , ..., ASn : The Action History, where
achieved through mocking—a formal approach to simulat- each ASi is an Action Snapshot;
ing GUI interactions. During mock runs, the automation does • AS = (oah , S h , r h ), where historical components (h ) are
a
not have access to the live GUI environment (, ). Instead, structurally equivalent to their live components:
it operates on a predefined Action History (AH ) containing – oah ∼
= oa : action parameters;
recorded outcomes of GUI interactions. h ∼ S : active states;
– S = 
– rah ∼
= ra : action results.
3 This is a joke referring to the concept of NP-hardness in computational
complexity theory. Implementation requires two key functions:

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

Fig. 25 The paths of length k

for a graph with 30 nodes and 5
outgoing edges per node. This
graph has 6.36 trillion paths

(1) A matching function that identifies relevant historical • Elements (E): Tests verify that defined GUI elements are
actions by comparing: sufficient to enable navigation and interaction;
• States (S): Mock runs validate state definitions and reveal
• parameters of the action to be executed (oa ) to snap-
missing or redundant states;
shot parameters (oah );
• Transitions (T ): Testing exposes gaps or errors in the
• a snapshot’s active states (S
h ) to current active states
transition paths between states.
(S ).
(2) A selection function that chooses an action snapshot from For example, consider a mock run failure when attempting
matching candidates. to reach state st ∈ S from state s0 ∈ S. This could indicate:

The mock action function operates by retrieving appropriate (1) Missing transition: t ∈ T such that t connects s0 to st ;
historical results: (2) Incomplete element set: ∃e ∈ / E required for transition;
(3) State definition error: st or intermediate states are inade-
ra := rah and Sa := f (rah ) (29) quately defined.

The structural equivalence between mock outputs and live This iterative improvement through testing enables the
automation ensures that both the action snapshot (AS) and the development of increasingly accurate models of the GUI
live action function ( f a ) produce results (ra ) and state infor- environment, supporting more sophisticated automation tasks.
mation (Sa ) in identical formats, preserving computational
isomorphism across testing and production environments. 11.3 Unit testing model
Consequently, downstream components that consume these
outputs—such as the State Management system (M) and path While integration testing through mocking verifies system-
traversal function ( f § )—can operate without distinguishing level behavior, unit testing requires deterministic verification
between simulated and actual GUI interactions. of individual components. The unit testing model uses two
different action functions to ensure reproducible results:
11.2 Model validation through testing For observation actions:

Integration testing in model-based GUI automation serves a f aunit : (a, x ) → (ra , Sa ) (30)
dual purpose: validating automation instructions and verify-
ing the accuracy of the State Structure  = (E, S, T ). GUI where x represents a fixed scene (e.g., a screenshot).
elements, states, and transitions are testable components that For all other actions:
can be systematically evaluated and improved.
Mock runs provide insights into each component of : f amock : (a, S ) → (ra , Sa ) (31)

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

Table 3 Comparison of integration and unit tests in GUI automation

Aspect Integration tests Unit tests

Results source Historical action data from previous GUI interactions Static screenshots for observations, selected historical actions for others
Stochasticity Preserved through sampling of historical results Eliminated using fixed inputs for reproducibility
Purpose Verify system behavior and state transitions Test individual components in isolation

where the outputs are determined by selected historical data.

Observation actions perform pattern matching on static
screenshots, while interactive actions such as clicks and
keyboard inputs are simulated using predetermined results
from action snapshots. This hybrid approach enables fully
reproducible tests that validate real image recognition func-
tionality while deterministically controlling actions that
inherently require a live GUI environment.
The key distinctions between the integration and unit test-
ing approaches are summarized in Table 3:
This formal approach to testing addresses a critical gap in
GUI automation development. By enabling integration and
unit testing without dependence on a live GUI environment,
model-based GUI automation makes it possible to develop
and maintain complex automation applications with the same
rigorous testing practices used in other areas of software
development. Unit tests achieve reproducibility by operating Fig. 26 The effect of automation complexity on code complexity for
on static screenshots rather than a live GUI, while integra- process- and model-based approaches, assuming similar robustness
tion tests use historical data to simulate interactions with a
stochastic GUI environment.
applications (e.g., ERP, CRM, legacy systems). By treating
This ability to rigorously test the automation code itself is a
the entire multi-application environment as a unified state
cornerstone of developing reliable and maintainable systems,
space () managed by the framework (F), model-based GUI
unlocking new application possibilities such as robust GUI
automation enables robust automation of intricate end-to-end
testing within CI/CD pipelines, as explored in Sect. 12.
processes.

GUI testing
12 Applications and practical implications
Model-based GUI automation allows tests to evolve with
Throughout this paper, the DoT application (Sect. 7) served applications rather than breaking with UI modifications.
as a concrete example to demonstrate model-based GUI Decoupling the GUI model (State Structure ) from test logic
automation principles. The benefits observed—reduced code (automation instructions ι) facilitates:
complexity (Sect. 10.1), dynamic adaptation (Sect. 9), and
testability (Sect. 11)—scale dramatically with application
• Resilient Test Scenarios: UI changes primarily affect
complexity. Figure 26 illustrates this, showing that while
localized state definitions () or transitions (τ ), leaving
process-based approaches face practical limits (Sect. 10.2),
high-level test logic intact and reducing maintenance.
model-based GUI automation maintains manageable com-
• Deterministic CI/CD Pipeline Execution: Mock runs
plexity even for extensive state spaces. This section explores
(Sects. 11.1, 11.3) ensure consistent, reliable GUI test
how this scalability and robustness enable previously imprac-
execution in CI/CD pipelines, free from the unpre-
tical applications.
dictability of live UI interactions.
• Comprehensive Test Coverage for Automation Code:
Process automation Standard software testing practices can be applied to
the automation code itself, allowing measurement of
Traditional Robotic Process Automation (RPA) often strug- coverage metrics and enhancing the reliability of the
gles with complex workflows spanning multiple enterprise automation logic.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

This elevates the engineering rigor of GUI testing, making it becomes a reliable, testable, and maintainable component of
a more reliable component of development. larger, intelligent systems.

Dataset generation for machine learning

13 Conclusion
Model-based GUI automation excels at generating large-
scale, high-quality datasets for training ML models for GUI Model-based GUI automation represents a new approach in
interaction, offering key advantages: the field of GUI automation, addressing long-standing chal-
lenges that have plagued traditional process-based approaches.
• Structured Action Representation: The Action Model Drawing from principles in robotics, human cognition, and
(a) (Sect. 5.4) provides consistent, vectorizable inputs graph theory, this work provides a systematic approach to
(oa , E a ). designing, implementing, and testing GUI automation appli-
• Robust Continuous Operation: Dynamic path traversal cations through the explicit modeling of GUI environments.
(§) and state awareness (M) enable resilient, prolonged It offers solutions to the inherent stochasticity of these envi-
data collection, allowing controlled randomness to gen- ronments, which has been the root cause of fragility and code
erate diverse training sets for better model generalization. complexity in process-based GUI automation applications.
• Rich Contextual Annotation: The explicit state model Key advantages of model-based GUI automation include:
() provides semantic context beyond pixels, annotating
training instances with active states (S ), transitions (τ ), • Enhanced robustness: By employing graph-based mod-
and element relationships (E), creating stronger training els and heuristics, model-based GUI automation can
signals. dynamically navigate unpredictable GUI environments.
This adaptability reduces the impact of stochastic ele-
ments that often lead to failures in process-based meth-
This enables generation of millions of labeled interaction
ods.
sequences, a scale impractical with manual methods or frag-
• Reduced code complexity: The mathematical analy-
ile process-based scripts.
sis of pathfinding demonstrates that model-based GUI
automation can significantly reduce code complexity,
Complex game automation and reinforcement especially as the number of possible states and transi-
learning tions increases. This reduction in complexity makes it
possible to develop robust solutions to more complex
As conceptualized in Sect. 5.9, model-based GUI automation GUI automation tasks.
can serve as a robust Visual API for applications lacking • Improved testability: Unlike process-based methods,
programmatic APIs, especially video games: model-based GUI automation leverages simulation and
mocking to enable integration and unit testing. The
• Bridging for RL Research: For games without native ability to test GUI automation with standard software
RL interfaces, the Visual API provides the necessary engineering practices enhances development and quality
abstraction. The State Structure () models the visual assurance.
environment, the framework (F) translates GUI interac- • Explicit representation: The representation of the GUI
tions, and the automation instructions (ι) implement the environment through explicitly defined states and tran-
actions dictated by the agent’s policy. sitions fundamentally changes the approach to automa-
• Automating Complex Gameplay: Dynamic pathfinding tion design. Explicit modeling improves scalability and
(§) and state management (M) offer resilience to game maintainability as automation task complexity grows,
stochasticity and visual changes, enabling automation enabling modular development where individual states
for player assistance, testing, or repetitive tasks where and transitions can be built and tested independently.
process-based methods fail.
• Interaction without Formal APIs: The Visual API In conclusion, this paper introduces model-based GUI
allows external programs or AI agents to interact with automation as a new domain that addresses the fundamental
any GUI environment in a standardized way. limitations of traditional process-based approaches, offering
a more robust, efficient, and scalable solution for com-
This decouples strategic logic from low-level GUI mechan- plex GUI automation tasks. It makes contributions in two
ics, lowering barriers for complex interaction tasks. key areas: the development of a comprehensive theoretical
The shift from implicit to explicit environment modeling framework, which lays the groundwork for further advance-
enables a new class of applications where GUI automation ment in the field, and practical implementation through the

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

open-source Brobot framework, which facilitates empirical I examine how states, transitions, nodes, and edges carry
experimentation and validation of the formal models. This distinct meanings in each context, and how these differ-
integration of theory and practice provides a fresh approach ences reflect the broader distinctions between testing and
to GUI automation in fields ranging from software testing to automation. This comparison is particularly important for
reinforcement learning research. practitioners familiar with model-based testing who are inter-
ested in model-based GUI automation, as it helps prevent
potential misconceptions arising from the superficial simi-
Appendix I: model-based GUI automation larities between their graphical representations.
versus model-based testing Figure 27 compares aspects of model-based testing and
model-based GUI automation. The left side shows the model
Although both model-based GUI automation and model- in model-based testing, which is used to generate discrete
based testing use directed graphs to represent their domains, test cases. Nodes represent actions taken on GUI elements
the underlying concepts and purposes of these graphs differ and edges represent the next actions available after a specific
significantly. This appendix provides a detailed comparison action is taken. The right side portrays the State Structure
of these approaches, highlighting how similar visual rep- () as a directed graph where states correspond to conceptu-
resentations can encode fundamentally different concepts. ally cohesive sections of the GUI, sets of states represent the

Fig. 27 Graphs used by model-based testing and model-based GUI automation

Fig. 28 A graphical visualization of the active states before and after the transition

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

Table 4 The meaning of units in the directed graphs of model-based testing and model-based GUI automation
Unit Model-based testing Model-based GUI automation

◦ Nodes are actions taken on GUI elements States are conceptually coherent sections of the GUI
→ Edges shows which actions are possible as the next step in the process Transitions are action sequences that change the set of active states

Table 5 Model-based testing

Concept From the perspective of model-based GUI automation
concepts from the perspective of
model-based GUI automation ◦ Node A GUI action. It can be part of a transition
→ Edge In model-based GUI automation graphs, individual GUI actions
(which would be edges in model-based testing) are encapsulated
within transitions rather than explicitly represented as separate
graph elements

Table 6 Model-based GUI

Concept From the perspective of model-based testing
automation concepts from the
perspective of model-based ◦ State In model-based testing, cohesive sections of the GUI (states) are not
testing modeled as explicit concepts. Instead, the existence of GUI elements
that might constitute a state is implied through connected action
nodes
→ Transition A series of one or more connected nodes

visible contents of the GUI at a specific point in time, and patterns and practices that remain relevant for more complex
transitions are action sequences that modify the set of active automation tasks.
states.
Figure 28 shows the start states (in the blue oval in Fig. 27) // annotations
and the states after transition 2a (in the yellow oval in Fig. 27) public class Home {
as they would appear in the GUI. The transition shows that
public enum Name implements
clicking on the link in State 2 will activate State 3 and deac- StateEnum {
tivate State 2. State 1 is unaffected and remains active after HOME
the transition. }
The directed graphs in model-based GUI automation and
private StateImageObject
model-based testing may look similar, but their underly- toWorldButton =
ing concepts differ. The key difference is that model-based new [Link]()
testing is action-centric (focusing on sequences of discrete .withImage("toWorldButton")
actions), while model-based GUI automation is state-centric .isFixed(true)
.addSnapshot(new MatchSnapshot
(focusing on GUI configurations and transitions between (220, 600, 20, 20))
them). These differences are presented in Tables 4, 5, and .build();
6.
private State state = new State.
Builder(HOME)
.withImages(toWorldButton)
Appendix II: DoT application code .build();

The DoT application provides a practical demonstration public Home(StateService

stateService) {
of model-based GUI automation principles. This appendix [Link](state);
presents the complete implementation code, illustrating how }
the theoretical concepts discussed in the paper are realized }
in a working application. The code demonstrates state def-
inition, transition management, and path traversal. Despite
the intentional simplicity of the DoT app, with only three
states and basic transitions, its implementation reveals key

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

// annotations // annotations
public class World { public class Island {

public enum Name implements public enum Name implements

StateEnum { StateEnum {
WORLD ISLAND
} }

private StateImageObject private StateRegion nameRegion = new

searchButton = [Link]()
new [Link]() .called("island name region")
.withImage("searchButton") .addSnapshot(new MatchSnapshot.
.isFixed(true) Builder()
.build(); .setActionOptions(
[Link].GET_TEXT)
private State state = new State. .addString("Mines")
Builder([Link]) .addString("Lakess")
.withImages(searchButton) .addString("Farmz")
.build(); .build())
.build();
public World(StateService
stateService) { private StateRegion islandRegion =
[Link](state); new [Link]()
} .called("island image region")
} .build();

private State state = new State.

Builder([Link])
.withRegions(nameRegion,
islandRegion)
.build();

public Island(StateService
stateService) {
[Link](state);
}
}

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

// annotations // annotations
public class HomeTransitions { public class WorldTransitions {
// injected dependencies // injected dependencies

public HomeTransitions(/* injected public WorldTransitions(/* injected

dependencies */) { dependencies */) {
// injected dependencies // injected dependencies
StateTransitions transitions = StateTransitions transitions =
new [Link]( new [Link](
HOME) WORLD)
.addTransition(this:: .addTransitionFinish(this::
goToWorld, WORLD) finishTransition)
.build(); .addTransition(new
[Link]( [Link]()
transitions); .addToActivate(ISLAND)
} .setFunction(this::
goToIsland)
private boolean goToWorld() { .
return [Link]( setStaysVisibleAfterTransition(TRUE)
[Link], .build())
[Link]() .build();
).isSuccess(); [Link](
} transitions);
} }

private boolean finishTransition() {

ActionOptions find = new
[Link]()
.setMaxWait(10)
.build();
ObjectCollection worldImages =
new [Link]()
.withAllStateImages(world.
getState())
.build();
return [Link](find,
worldImages).isSuccess();
}

public boolean goToIsland() {

return [Link](
CLICK, [Link]()
).isSuccess();
}
}

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

// annotations // annotations
public class IslandTransitions { public class IslandActivities {
// injected dependencies // injected dependencies
private Image islandTypes = new
public IslandTransitions(/* injected Image(
dependencies */) { "castle", "mines", "farms", "
// injected dependencies forest", "mountains", "lakes");
StateTransitions transitions =
new [Link]( public boolean defineNameRegion() {
ISLAND) if ([Link]().
.addTransitionFinish(this:: defined()) return true;
finishTransition) ActionOptions defineWith = new
.build(); [Link]()
[Link]( .setAction(ActionOptions.
transitions); [Link])
} .setDefineAs(ActionOptions.
[Link])
public boolean finishTransition() { .setMaxWait(3)
if (!islandActivities. .build();
defineIslandRegion() || ObjectCollection nameImages =
!islandActivities. new [Link]()
defineNameRegion()) return false; .withImages(islandTypes).
return ![Link](). build();
isEmpty(); Region definedRegion =
} [Link](defineWith,
} nameImages).getDefinedRegion();
[Link]().
setSearchRegion(definedRegion);
return [Link]().
defined();
}

public boolean defineIslandRegion()

{
if ([Link]().
defined()) return true;
ActionOptions defineAbove = new
[Link]()
.setAction(ActionOptions.
[Link])
.setDefineAs(ActionOptions.
[Link])
.setAddX(-50)
.setAddY(-250)
.setAbsoluteWidth(200)
.setAbsoluteHeight(200)
.build();
ObjectCollection searchButton =
new [Link]()
.withImages(world.
getSearchButton()).build();
Region definedRegion =
[Link](defineAbove,
searchButton).getDefinedRegion();
[Link]().
setSearchRegion(definedRegion);
return [Link]().
defined();
}
}

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

// annotations // annotations
public class GetIslandType { public class SaveLabeledImages {
// injected dependencies // injected dependencies
private final Map<String, String>
islandTypes = [Link]( public void saveImages(int maxImages
"Burg", "Castle", "Mine", "Mines , String directory) {
", if (!stateTransitionsManagement.
"Farm", "Farms", "Moun", " openState(ISLAND)) return;
Mountains", for (int i=0; i<maxImages; i++)
"Fore", "Forest", "Lake", "Lakes {
" goToNewIsland();
); String newIslandType =
private String currentIslandType = [Link]()
""; ;
// output island type
public String getType() { if (![Link]()
String textRead = getIslandText &&
(); [Link]().
[Link]( defined()) {
textRead); imageUtils.
currentIslandType = saveRegionToFile(
getIslandTypeFromText(textRead); [Link]
return currentIslandType; ().getSearchRegion(),
} directory + "/" +
newIslandType
private String getIslandTypeFromText );
(String islandNameText) { }
for ([Link]<String, String> // output newline
type : [Link]()) { }
if ([Link]( }
[Link]()))
currentIslandType = type private boolean goToNewIsland() {
.getValue(); return worldTransitions.
} goToIsland() &&
return ""; islandTransitions.
} finishTransition();
}
private String getIslandText() { }
ActionOptions getText = new
[Link]()
.setAction(ActionOptions.
Action.GET_TEXT)
.setMaxWait(3)
.setPauseBeforeBegin(3)
.build();
ObjectCollection nameRegion =
new [Link]()
.withRegions(island.
getNameRegion())
.build();
return [Link](getText,
nameRegion)
.getSelectedText();
}
}

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

The classes and images are organized as follows in the

// annotations DoT app’s file structure.
public class GetLabeledDataApp {

public static void main(String[]

args) {
// configure Spring context

// setup brobot
[Link]("images
");
[Link] = false;

// find initial active States

InitialStates initialStates =
[Link](
[Link]);
[Link](90,
HOME);
[Link](10,
WORLD);
[Link]()
;

// get and save labeled images

SaveLabeledImages
saveLabeledImages =
[Link](
[Link]);
[Link]
(100, "labeledImages");
}
}

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

The DoT app consists of several key classes that work GetIslandType class
together:
• contains automation instructions;
State classes (Home, World, and Island) • reads and interprets island names from the screen;
• can handle text recognition errors such as “Minez”;
• defines the states with images and regions • returns an empty string if the island type is not recog-
nized.
– the Home state contains just one element: an image
of the button that leads to the World state);
– the Island state has two regions: one that shows the SaveLabeledImages class
island’s name and one that displays the island image.
• contains the main automation instructions:
Transition classes (HomeTransitions, (1) navigates to an island;
WorldTransitions, and (2) saves its image with the appropriate label;
IslandTransitions) (3) moves to the next island;
(4) repeats until enough images are collected.
• handle movement between states;
• uses the DoT’s Visual API (Brobot + the DoT State Struc-
• a good example is moving from World to Island state:
ture) for navigation and image handling.
– WorldTransitions starts the process with its
goToIsland method; The main executable (GetLabeledDataApp)
– IslandTransitions completes it with finish
Transition, which checks for a valid island name; • sets up the Spring Boot application;
– the World state stays visible during this transition • configures the image storage location;
(marked by staysVisibleAfterTransition • tells Brobot which states might be active at startup;
= TRUE). • runs the automation to collect 100 labeled island images.

IslandActivities class When moving from one island to another, the DoT app
directly calls the World → Island transition instead of
• defines where to look for important information in the relying on Brobot’s path traversal. This is necessary because
Island state; both the current and target locations are part of the same
• Uses two main methods: Island state. This demonstrates the subjectivity of State
– defineNameRegion: finds where the island name Structure design and emphasizes the importance of tailoring
appears on screen; the design to the needs of the automation task. The cur-
– defineIslandRegion: locates the island image rent DoT design was chosen for simplicity, but having a
by using the search button as a reference point. NewIsland state would have been a better choice given
the automation task.
The results of executing the DoT app are shown in Fig. 29.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

Fig. 29 The file folder containing the saved labeled images

Appendix III: integration test example

|CLICK [Link]| [Link]
This appendix provides an example of integration testing in 36:09 found=true <click> wait-0,2 <
click> CLICK 
model-based GUI automation, as implemented by the Brobot
|GET_TEXT wait-3,0 [Link] GET_TEXT 
framework. text = Farmz
The StateRegion definition below demonstrates how Save file with base path DOTislands/
action snapshots store historical results for mock runs: labeledImages/Farms

|CLICK [Link]| [Link]

36:12 found=true <click> wait-0,2 <
click> CLICK 
private StateRegion nameRegion = new |GET_TEXT wait-3,0 [Link] GET_TEXT 
[Link]() text = Lakes
.called("island name region") Save file with base path DOTislands/
.addSnapshot(new MatchSnapshot. labeledImages/Lakes
Builder()
.setActionOptions(ActionOptions.
Action.GET_TEXT)
.addString("Mines") // Declarations
Historical text recognition result
.addString("Lakess") //
Simulates OCR variation BROBOT LINKS • Documentation: [Link] •
.addString("Farmz") // Code: [Link]
Simulates OCR variation
.build()) Conflict of interest The work on model-based GUI automation, both
.build(); conceptually and in code, was the sole effort of the author and was
not funded or supported by any organization. Brobot is free and open-
source.

Open Access This article is licensed under a Creative Commons

Attribution 4.0 International License, which permits use, sharing, adap-
Mock runs use these snapshots to simulate GUI interac-
tation, distribution and reproduction in any medium or format, as
tion. The following output of an integration test shows state long as you give appropriate credit to the original author(s) and the
transitions and action results: source, provide a link to the Creative Commons licence, and indi-

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Model-based GUI automation

cate if changes were made. The images or other third party material Trans. Softw. Eng. Methodol. 32(3), 1–30 (2023). [Link]
in this article are included in the article’s Creative Commons licence, 10.1145/3571855
unless indicated otherwise in a credit line to the material. If material 16. Alégroth, E., Karlsson, A., Radway, A.: Continuous integration and
is not included in the article’s Creative Commons licence and your visual GUI testing: benefits and drawbacks in industrial practice
intended use is not permitted by statutory regulation or exceeds the (2018). [Link]
permitted use, you will need to obtain permission directly from the copy- 17. Grechanik, M., Xie, Q., Fu, C.: Experimental assessment of manual
right holder. To view a copy of this licence, visit [Link] versus tool-based maintenance of GUI-directed test scripts (2009).
[Link]/licenses/by/4.0/. [Link]
18. Eladawy, H.M., Mohamed, A., Salem, S.A.: A new algorithm
for repairing web-locators using optimization techniques (2018).
[Link]
References 19. Thummalapenta, S., Devaki, P., Sinha, S., Chandra, S., Gnana-
sundaram, S., Nagaraj, D.D., Kumar, S., Kumar, S.: Efficient and
1. Banerjee, I., Nguyen, B., Garousi, V., Memon, A.: Graphical change-resilient test automation: an industrial case study. In: 2013
user interface (GUI) testing: systematic mapping and repository. 35th International Conference on Software Engineering (ICSE)
Inf. Softw. Technol. 55(10), 1679–1694 (2013). [Link] (2013). [Link]
1016/[Link].2013.03.004 20. Zheng, Y., Huang, S., Hui, Z., Wu, Y.: A method of optimizing
2. Yeh, T., Chang, T.-H., Miller, R.C.: Sikuli: using GUI screenshots multi-locators based on machine learning (2018). [Link]
for search and automation. In Proceedings of the 22nd Annual ACM 10.1109/qrs-c.2018.00041
Symposium on User Interface Software and Technology (UIST 21. Memon, A.M., Soffa, M.L.: Regression testing of GUIs (2003).
’09), pp. 183–192. ACM (2009). [Link] [Link]
handle/1721.1/72686/miller_sikuli.pdf%3Fsequence%3D1 22. Langley, P., Pearce, C.J., Barley, M., Emery, M.: Bounded rational-
3. Selenium: selenium documentation (2023). [Link] ity in problem solving: guiding search with domain-independent
dev/documentation/ heuristics. Mind Soc. 13(1), 83–95 (2014). [Link]
4. Aho, P., Vos, T.E.J.: Challenges in automated testing through graph- s11299-014-0143-y
ical user interface. In: 2018 IEEE International Conference on Soft- 23. Ferguson, D., Likhachev, M., Stentz, A.: A guide to heuristic-based
ware Testing, Verification and Validation Workshops (ICSTW), pp. path planning. American Association for Artificial Intelligence
118–121 (2018). [Link] (2005). [Link]
5. Petrenko, A., Simao, A., Maldonado, J.C.: Model-based testing of [Link]
software and systems: recent advances and challenges. Int. J. Softw. 24. Pohl, I.: Heuristic search viewed as path finding in a graph.
Tools Technol. Transf. 14(4), 383–386 (2012). [Link] Artif. Intell. 1(3–4), 193–204 (1970). [Link]
1007/s10009-012-0240-3 0004-3702(70)90007-x
6. Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model- 25. Hart, P., Nilsson, N., Raphael, B.: A formal basis for the heuris-
based testing approaches. Softw. Test. Verif. Reliab. 22(5), 297–312 tic determination of minimum cost paths. IEEE Trans. Syst. Sci.
(2011). [Link] Cybern. 4(2), 100–107 (1968). [Link]
7. Yuan, X., Memon, A.M.: Iterative execution-feedback model- 300136
directed GUI testing. Inf. Softw. Technol. 52(5), 559–575 (2010). 26. Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den
[Link] Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam,
8. Moreira, R.O., Paiva, A., Nabuco, M., Memon, A.M.: Pattern-based V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbren-
GUI testing: bridging the gap between design and quality assur- ner, N., Sutskever, I., Lillicrap, T.P., Leach, M., Kavukcuoglu, K.,
ance. Softw. Test. Verif. Reliab. 27(3), e1629 (2017). [Link] Graepel, T., Hassabis, D.: Mastering the game of Go with deep neu-
org/10.1002/stvr.1629 ral networks and tree search. Nature 529(7587), 484–489 (2016).
9. Clarke, E.M., Klieber, W., Novacek, M., Zuliani, P.: Model check- [Link]
ing and the state explosion problem. In: Lecture Notes in Computer 27. Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets,
Science. Springer, pp. 1–30 (2012). [Link] A.S., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., Schrittwieser,
642-35746-6_1 J., Quan, J., Gaffney, S., Petersen, S., Simonyan, K., Schaul, T.,
10. Dobslaw, F., Feldt, R., Michaelsson, D., Haar, P., De Oliveira van Hasselt, H., Silver, D., Lillicrap, T., Calderone, K., Keet, P.,
Neto, F.G., Torkar, R.: Estimating return on investment for GUI Brunasso, A., Lawrence, D., Ekermo, A., Repp, J., Tsing, R.:
test automation frameworks (2019). arXiv. [Link] StarCraft II: a new challenge for reinforcement learning (2017).
issre.2019.00035 [Link]. [Link]
11. Garousi, V., Afzal, W., Çaglar, A., Isik, I.B., Baydan, B., Çaylak, S.,
Boyraz, A.Z., Yolaçan, B., Herkiloglu, K.: Comparing automated
visual GUI testing tools: an industrial case study (2017). https:// Publisher’s Note Springer Nature remains neutral with regard to juris-
[Link]/10.1145/3121245.3121250 dictional claims in published maps and institutional affiliations.
12. Wiklund, K., Eldh, S., Sundmark, D., Lundqvist, K.: Impediments
for software test automation: a systematic literature review. Softw.
Test. Verif. Reliab. 27(8), e1639 (2017). [Link]
stvr.1639
13. Nass, M., Alégroth, E., Feldt, R.: Why many challenges with GUI
test automation (will) remain. Inf. Softw. Technol. 138, 106625
(2021). [Link]
14. Yandrapally, R., Thummalapenta, S., Sinha, S., Chandra, S.: Robust
test automation using contextual clues (2014). [Link]
1145/2610384.2610390
15. Nass, M., Alégroth, E., Feldt, R., Leotta, M., Ricca, F.: Similarity-
based web element localization for robust test automation. ACM

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

 J. Spinak

Joshua Spinak is a data scien-

tist, developer, and independent
researcher based in Düsseldorf,
Germany. His research focuses on
enabling agents to reason and act
within complex visual environments.
He explores both specialized foun-
dational models and general-purpose
LLMs, leveraging model-based rep-
resentations to support robust data
generation, tool use, and dynamic
task execution. His work addresses
key bottlenecks in GUI automa-
tion, including process fragility,
semantic understanding, interface
generalization, and long-horizon planning. Joshua has degrees from
Yale University, MIT, and London Business School.

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Terms and Conditions
Springer Nature journal content, brought to you courtesy of Springer Nature Customer Service Center GmbH (“Springer Nature”).
Springer Nature supports a reasonable amount of sharing of research papers by authors, subscribers and authorised users (“Users”), for small-
scale personal, non-commercial use provided that all copyright, trade and service marks and other proprietary notices are maintained. By
accessing, sharing, receiving or otherwise using the Springer Nature journal content you agree to these terms of use (“Terms”). For these
purposes, Springer Nature considers academic use (by researchers and students) to be non-commercial.
These Terms are supplementary and will apply in addition to any applicable website terms and conditions, a relevant site licence or a personal
subscription. These Terms will prevail over any conflict or ambiguity with regards to the relevant terms, a site licence or a personal subscription
(to the extent of the conflict or ambiguity only). For Creative Commons-licensed articles, the terms of the Creative Commons license used will
apply.
We collect and use personal data to provide access to the Springer Nature journal content. We may also use these personal data internally within
ResearchGate and Springer Nature and as agreed share it, in an anonymised way, for purposes of tracking, analysis and reporting. We will not
otherwise disclose your personal data outside the ResearchGate or the Springer Nature group of companies unless we have your permission as
detailed in the Privacy Policy.
While Users may use the Springer Nature journal content for small scale, personal non-commercial use, it is important to note that Users may
not:

1. use such content for the purpose of providing other users with access on a regular or large scale basis or as a means to circumvent access
control;
2. use such content where to do so would be considered a criminal or statutory offence in any jurisdiction, or gives rise to civil liability, or is
otherwise unlawful;
3. falsely or misleadingly imply or suggest endorsement, approval , sponsorship, or association unless explicitly agreed to by Springer Nature in
writing;
4. use bots or other automated methods to access the content or redirect messages
5. override any security feature or exclusionary protocol; or
6. share the content in order to create substitute for Springer Nature products or services or a systematic database of Springer Nature journal
content.
In line with the restriction against commercial use, Springer Nature does not permit the creation of a product or service that creates revenue,
royalties, rent or income from our content or its inclusion as part of a paid for service or for other commercial gain. Springer Nature journal
content cannot be used for inter-library loans and librarians may not upload Springer Nature journal content on a large scale into their, or any
other, institutional repository.
These terms of use are reviewed regularly and may be amended at any time. Springer Nature is not obligated to publish any information or
content on this website and may remove it or features or functionality at our sole discretion, at any time with or without notice. Springer Nature
may revoke this licence to you at any time and remove access to any copies of the Springer Nature journal content which have been saved.
To the fullest extent permitted by law, Springer Nature makes no warranties, representations or guarantees to Users, either express or implied
with respect to the Springer nature journal content and all parties disclaim and waive any implied warranties or warranties imposed by law,
including merchantability or fitness for any particular purpose.
Please note that these rights do not automatically extend to content, data or other material published by Springer Nature that may be licensed
from third parties.
If you would like to use or distribute our Springer Nature journal content to a wider audience or on a regular basis or in any other manner not
expressly permitted by these Terms, please contact Springer Nature at

onlineservice@[Link]