Object Detection for Graphical User Interface: Old Fashioned or

Deep Learning or a Combination?

Jieshan Chen Mulong Xie Zhenchang Xing∗
[Link]@[Link] u6462764@[Link] [Link]@[Link]
Australian National University Australian National University Australian National University
Australia Australia Australia

Chunyang Chen† Xiwei Xu Liming Zhu‡

[Link]@[Link] [Link]@[Link] [Link]@[Link]
arXiv:2008.05132v2 [[Link]] 7 Sep 2020

Monash University Data61, CSIRO Data61, CSIRO

Australia Australia

Guoqiang Li
Li.G@[Link]
Shanghai Jiao Tong University
China
ABSTRACT CCS CONCEPTS
Detecting Graphical User Interface (GUI) elements in GUI images is • Software and its engineering → Software development tech-
a domain-specific object detection task. It supports many software niques; • Human-centered computing → Graphical user in-
engineering tasks, such as GUI animation and testing, GUI search terfaces.
and code generation. Existing studies for GUI element detection
directly borrow the mature methods from computer vision (CV) KEYWORDS
domain, including old fashioned ones that rely on traditional image Android, Object Detection, User Interface, Deep Learning, Com-
processing features (e.g., canny edge, contours), and deep learning puter Vision
models that learn to detect from large-scale GUI data. Unfortunately,
ACM Reference Format:
these CV methods are not originally designed with the awareness Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu,
of the unique characteristics of GUIs and GUI elements and the high Liming Zhu, and Guoqiang Li. 2020. Object Detection for Graphical User
localization accuracy of the GUI element detection task. We conduct Interface: Old Fashioned or Deep Learning or a Combination?. In Proceed-
the first large-scale empirical study of seven representative GUI ings of the 28th ACM Joint European Software Engineering Conference and
element detection methods on over 50k GUI images to understand Symposium on the Foundations of Software Engineering (ESEC/FSE ’20), No-
the capabilities, limitations and effective designs of these methods. vember 8–13, 2020, Virtual Event, USA. ACM, New York, NY, USA, 13 pages.
This study not only sheds the light on the technical challenges [Link]
to be addressed but also informs the design of new GUI element
detection methods. We accordingly design a new GUI-specific old- 1 INTRODUCTION
fashioned method for non-text GUI element detection which adopts GUI allows users to interact with software applications through
a novel top-down coarse-to-fine strategy, and incorporate it with the graphical elements such as widgets, images and text. Recognizing
mature deep learning model for GUI text detection. Our evaluation GUI elements in a GUI is the foundation of many software engineer-
on 25,000 GUI images shows that our method significantly advances ing tasks, such as GUI automation and testing [5, 36, 49, 52], support-
the start-of-the-art performance in GUI element detection. ing advanced GUI interactions [1, 17], GUI search [16, 40], and code
generation [11, 32, 34]. Recognizing GUI elements can be achieved
∗ Also by instrumentation-based or pixel-based methods. Instrumentation-
with Data61, CSIRO.
† Corresponding author. based methods [3, 28, 35] are intrusive and requires the support of
‡ Also with University of New South Wales. accessibility APIs [7, 24] or runtime infrastructures [23, 31] that
expose information about GUI elements within a GUI. In contrast,
pixel-based methods directly analyze the image of a GUI, and thus
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed are non-intrusive and generic. Due to the cross-platform character-
for profit or commercial advantage and that copies bear this notice and the full citation istics of pixel-based methods, they can be widely used for novel ap-
on the first page. Copyrights for components of this work owned by others than ACM plications such as robotic testing of touch-screen applications [36],
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a linting of GUI visual effects [54] in both Android and IOS.
fee. Request permissions from permissions@[Link]. Pixel-based recognition of GUI elements in a GUI image can be
ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA regarded as a domain-specific object detection task. Object detection
© 2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-7043-1/20/11. . . $15.00 is a computer-vision technology that detects instances of semantic
[Link] objects of a certain class (such as human, building, or car) in digital
ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA Chen, J., Xie, M., Xing, Z., Chen, C., Xu, X., Zhu, L. and Li, G.

images and videos. It involves two sub-tasks: region detection or Button Spinner Chronometer ImageView RatingBar ProgressBar
proposal - locate the bounding box (bbox for short) (i.e., the smallest
rectangle region) that contains an object, and region classification
- determine the class of the object in the bounding box. Existing VideoView ToggleButton RadioButton Switch Checkbox ImageButton
object-detection techniques adopt a bottom-up strategy: starts with
primitive shapes and regions (e.g., edges or contours) and aggregate
them progressively into objects. Old-fashioned techniques [32, 34, TextView EditText SeekBar
46] relies on image features and aggregation heuristics generated by
expert knowledge, while deep learning techniques [18, 39, 41] use
neural networks to learn to extract features and their aggregation Figure 1: Characteristics of GUI elements: large in-class vari-
rules from large image data. ance and high cross-class similarity
GUI elements can be broadly divided into text elements and
non-text elements (see Figure 1 for the examples of Android GUI of the GUI elements and the accuracy of the detected bounding
elements). Both old-fashioned techniques and deep learning models boxes.
have been applied for GUI element detection [10, 32, 34, 49, 53]. As It is surprising that anchor-box based models are robust to the
detailed in Section 2.1, considering the image characteristics of GUIs anchor-box settings, and merging the detection results by different
and GUI elements, the high accuracy requirement of GUI-element anchor-box settings can improve the final performance. Our study
region detection, and the design rationale of existing object detec- shows that detecting text and non-text GUI elements by a single
tion methods, we raise a set of research questions regarding the model performs much worse than by a dedicated text and non-
effectiveness features and models originally designed for generic text model respectively. GUI text should be treated as scene text
object detection on GUI elements, the region detection accuracy rather than document text, and the state-of-the-art deep learning
of statistical machine learning models, the impact of model ar- scene text model EAST [56] (pretrained without fine tuning) can
chitectures, hyperparameter settings and training data, and the accurately detect GUI text.
appropriate ways of detecting text and non-text elements. Inspired by these findings, we design a novel approach for GUI
These research questions have not been systematically studied. element detection. For non-text GUI element detection, we adopt
First, existing studies [32, 34, 49] evaluate the accuracy of GUI ele- the simple two-stage architecture: perform region detection and
ment detection by only a very small number (dozens to hundreds) region classification in a pipeline. For non-text region detection,
of GUIs. The only large-scale evaluation is GUI component design we prefer the simplicity and the bounding-box accuracy of old-
gallery [10], but it tests only the default anchor-box setting (i.e. a fashioned methods. By taking into account the unique boundary,
predefined set of bboxes) of Faster RCNN [41] (a two-stage model). shape, texture and layout characteristics of GUI elements, we de-
Second, none of existing studies (including [10]) have investigated sign a novel old-fashioned method with a top-down coarse-to-fine
the impact of training data size and anchor-box setting on the per- detection strategy, rather than the current bottom-up edge/contour
formance of deep learning object detection models. Furthermore, aggregation strategy in existing methods [34, 53]. For non-text
the latest development of anchor-free object detection has never region classification and GUI text detection, we adopt the ma-
been attempted. Third, no studies have compared the performance ture, easy-to-deploy ResNet50 image classifier [25] and the EAST
of different methods, for example old fashioned versus deep learn- scene text detector [56], respectively. By a synergy of our novel
ing, or different styles of deep learning (e.g., two stage versus one old-fashioned methods and existing mature deep learning models,
stage, anchor box or free). Fourth, GUI text is simply treated by our new method achieves 0.573 in F1 for all GUI elements, 0.523
Optical Character Recognition (OCR) techniques, despite the sig- in F1 for non-text GUI elements, and 0.516 in F1 for text elements
nificant difference between GUI text and document text that OCR in a large-scale evaluation with 25,000 GUI images, which signifi-
is designed for. cantly outperform existing old-fashioned methods, and outperform
To answer the raised research questions, we conduct the first the best deep learning model by 19.4% increase in F1 for non-text
large-scale, comprehensive empirical study of GUI element detec- elements and 47.7% increase in F1 for all GUI elements.
tion methods, involving a dataset of 50,524 GUI screenshots ex- This paper makes the following contributions:
tracted from 8,018 Android mobile applications (see Section 3.2.1), • We perform the first systematic analysis of the problem scope
and two representative old-fashioned methods (REMAUI [34] and and solution space of GUI element detection, and identify the key
Xianyu [53]) and three deep learning models (Faster RCNN [41], challenges to be addressed, the limitations of existing solutions,
YOLOv3 [39] and CenterNet [18]) that cover all major method styles and a set of unanswered research questions.
(see Section 3.2.2). Old-fashioned detection methods perform poorly • We conduct the first large-scale empirical study of seven repre-
(REMAUI F1=0.201 and Xianyu F1=0.154 at IoU>0.9) for non-text sentative GUI element detection methods, which systematically
GUI element detection. IoU is the intersection area over union area answers the unanswered questions. We identify the pros and cons
of the detected bounding box and the ground-truth box. Deep learn- of existing methods which informs the design of new methods
ing methods perform much better than old-fashioned methods, and for GUI element detection.
the two-stage anchor-box based Faster RCNN performs the best • We develop a novel approach that effectively incorporates the
(F1=0.438 at IoU>0.9), and demands less training data. However, advantages of different methods and achieves the state-of-the-art
even Faster RCNN cannot achieve a good balance of the coverage performance in GUI element detection.
Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA

Table 1: Existing solutions for non-text GUI element detection and their limitations
Style Method Region Detection Region Classification
• Detect primitive edges and/or regions, and merge them Heuristically distinguish image, text,
Edge/contour
Old into larger regions (windows or objects) list, container [34, 53]. Can be
aggregation [32,
Fashioned • Merge with text regions recognized by OCR enhanced by a CNN classification like
34, 53]
• Ineffective for artificial GUI elements (e.g., images) in [32]
• Depend on manual feature engineering (either sample images or abstract prototypes)
Template
• Match samples/prototypes to detect object bounding box and class at the same time
matching [2, 17,
• Only applicable to simple and standard GUI elements (e.g., button, checkbox)
36, 52]
• Hard to apply to GUI elements with large variance of visual features
• Must define anchor boxes
A CNN classifier for region
Deep Anchor-box, two • Pipeline region detection and region classification
classification, trained jointly with
Learning stage [10, 41] • Gallery D.C. [10] is the only work that tests the Faster
region proposal network
RCNN on large-scale real GUIs, but it uses default settings
Anchor-box, one
• YOLOv2 [38] and YOLOv3 [39] uses k-means to determine anchor boxes (k is user-defined)
stage [39, 49]
• Simultaneously region detection and region classification
• [49] uses YOLOv2; trains and tests on artificial desktop GUIs; only tests on 250 real GUIs
Anchor free [18] Never applied
2 PROBLEM SCOPE AND SOLUTION SPACE visual cues. Existing object detection tasks usually deal with physi-
In this section, we identify the unique characteristics of GUIs and cal objects with distinct features across classes, for example, horses,
GUI elements, which have been largely overlooked when designing trucks, persons and birds in the popular COCO2015 dataset [29].
or choosing GUI element detection methods (Section 2.1). We also High cross-class similarity affects not only region classification
summarize representative methods for GUI element detection and but also region detection by deep learning models, as these two
point out the challenges that the unique characteristics of GUIs and subtasks are jointly trained.
GUI elements pose to these methods (Section 2.2). Mix of heterogeneous objects: GUIs display widgets, images
and texts. Widgets are artificially rendered objects. As discussed
above, they have large in-class variance and high cross-class simi-
2.1 Problem Scope larity. ImageView has simple rectangle shape but can display any
Figure 1 and Figure 6 shows examples of GUI elements and GUIs contents and objects. For the GUI element detection task, we want
in our dataset. We observe two element-level characteristics: large to detect the ImageViews themselves, but not any objects in the
in-class variance and high cross-class similarity, and two GUI-level images. However, the use of visual features designed for physical
characteristics: packed scene and close-by elements, and mix of objects (e.g., canny edge [8], contour map [45]) contradicts this
heterogeneous objects. In face of these characteristics, GUI element goal. In Figure 6 and Figure 4, we can observe a key difference
detection must achieve high accuracy on region detection. between GUI texts and general document texts. That is, GUI texts
Large in-class variance: GUI elements are artificially designed, are often highly cluttered with the background and close to other
and their properties (e.g., height, width, aspect ratio and textures) GUI elements, which pose main challenge of accurate text detection.
depend on the content to display, the interaction to support and the These heterogeneous properties of GUI elements must be taken
overall GUI designs. For example, the width of Button or EditText into account when designing GUI element detection methods.
depends on the length of displayed texts. ProgressBar may have dif- Packed scene and close-by elements: As seen in Figure 6,
ferent styles (vertical, horizontal or circle). ImageView can display GUIs, especially those of mobile applications, are often packed with
images with any objects or contents. Furthermore, different design- many GUI elements, covering almost all the screen space. In our
ers may use different texts, colors, backgrounds and look-and-feel, dataset (see Section 3.2.1), 77% of GUIs contain more than seven GUI
even for the same GUI functionality. In contrast, physical-world elements. Furthermore, GUI elements are often placed close side by
objects, such as human, car or building, share many shape, appear- side and separated by only small padding in between. In contrast,
ance and physical constraints in common within one class. Large there are only an average of seven objects placed sparsely in an
in-class variance of GUI elements pose main challenge of accurate image in the popular COCO(2015) object detection challenge [29].
region detection of GUI elements. GUI images can be regarded as packed scenes. Detecting objects in
High cross-class similarity: GUI elements of different classes packed scenes is still a challenging task, because close-by objects
often have similar size, shape and visual features. For example, But- interfere the accurate detection of each object’s bounding box.
ton, Spinner and Chronometer all have rectangle shape with some High accuracy of region detection For generic object detec-
text in the middle. Both SeekBar and horizontal ProgressBar show tion, a typical correct detection is defined loosely, e.g., by an IoU>
a bar with two different portions. The visual differences to distin- 0.5 between the detected bounding box and its ground truth (e.g.,
guish different classes of GUI elements can be subtle. For example, the PASCAL VOC Challenge standard [20]), since people can recog-
the difference between Button and Spinner lies in a small triangle nize an object easily from major part of it. In contrast, GUI element
at the right side of Spinner, while a thin underline distinguishes detection has a much stricter requirement on the accuracy of region
EditText from TextView. Small widgets are differentiated by small
ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA Chen, J., Xie, M., Xing, Z., Chen, C., Xu, X., Zhu, L. and Li, G.

detection. Inaccurate region detection may not only result in inaccu- applied to GUI element detection. These two models rely on a set
rate region classification, but more importantly it also significantly of pre-defined anchor boxes. The number of anchor boxes and their
affects the downstream applications, for example, resulting in incor- height, width and aspect ratio are all the model hyperparameters,
rect layout of generated GUI code, or clicking on the background which are either determined heuristically [41] or by clustering the
in vain during GUI testing. However, the above GUI characteristics training images using k-means and then using the metrics of the
make the accurate region detection a challenging task. Note that centroid images [39] Considering large in-class variance of GUI
accurate region classification is also important, but the difficulty elements, how sensitive are these anchor-box based models to the
level of region classification relies largely on the downstream ap- definition of anchor boxes, when they are applied to GUI element
plications. It can be as simple as predicting if a region is tapable or detection? Furthermore, the recently proposed anchor-free model
editable for GUI testing, or if a region is a widget, image or text in (e.g., CenterNet [18]) removes the need of pre-defined anchor-boxes,
order to wireframe a GUI, or which of dozens of GUI framework but has never been applied to GUI element detection. Can anchor-
component(s) can be used to implement the region. free model better deal with large in-class variance of GUI elements?
Last but not least, the performance of deep learning models heavily
2.2 Solution Space depends on sufficient training data. How well these models perform
We summarize representative methods for GUI element detection, with different amount of training data?
and raise questions that have not been systematically answered. 2.2.2 Text Element Detection. Existing methods either do not de-
tect GUI texts or detect GUI texts separately from non-text GUI
2.2.1 Non-Text Element Detection. Table 1 summarizes existing
element detection. They simply use off-the-shelf OCR tools (e.g.,
methods for non-text GUI element detection. By contrasting these
Tesseract [43]) for GUI text detection. OCR tools are designed for
methods and the GUI characteristics in Section 2.1, we raise a series
recognizing texts in document images, but GUI texts are very dif-
questions for designing effective GUI element detection methods.
ferent from document texts. Is OCR really appropriate for detecting
We focus our discussion on region detection, which aims to distin-
GUI texts? Considering the cluttered background of GUI texts, would
guish GUI element regions from the background. Region classifica-
it better to consider GUI text as scene text? Can the deep learning
tion can be well supported by a CNN-based image classifier [32].
scene text model effectively detect GUI texts? Finally, considering the
The effectiveness of physical-world visual features. Old-
heterogeneity of GUI widgets, images and texts, can a single model
fashioned methods for non-text GUI element detection rely on
effectively detect text and non-text elements?
either edge/contour aggregation [32, 34, 53] or template match-
ing [2, 17, 36, 52]. Canny edge [8] and contour map [45] are primi- 3 EMPIRICAL STUDY
tive visual features of physical-world objects, which are designed
to capture fine-grained texture details of objects. However, they To answer the above unanswered questions, we conduct the first
do not intuitively correspond to the shape and composition of large-scale empirical study of using both old-fashioned and deep
GUI elements. It is error-prone to aggregate these fine-grained learning methods for GUI element detection. Our study is done
regions into GUI elements, especially when GUIs contain images on a dataset of 50,524 GUI screenshots from the Rico dataset [16],
with physical-world objects. Template matching methods improve which were extracted from 8,018 Android mobile applications from
over edge/contour aggregation by guiding the region detection and 27 application categories. Our study involves a systematic compar-
aggregation with high-quality sample images or abstract proto- ison of two old-fashioned methods, including the representative
types of GUI elements. But this improvement comes with the high method REMAUI [34] in the literature and the method Xianyu [53]
cost of manual feature engineering. As such, it is only applicable recently developed by the industry, and three popular deep learning
to simple and standard GUI widgets (e.g., button and checkbox of methods that cover all major model design styles, including two
desktop applications). It is hard to apply template-matching method anchor-box based methods - Faster RCNN [41] (two stage style)
to GUI elements of mobile applications which have large variance and YOLO V3 [39] (one stage style) and one one-stage anchor-free
of visual features. Deep learning models [10, 18, 39, 41, 49] remove model CenterNet [18]. For GUI text detection, we compare OCR
the need of manual feature engineering by learning GUI element tool Tesseract [43] and scene text detector EAST [56], and compare
features and their composition from large numbers of GUIs. How separate and unified detection of text and non-text GUI elements.
effective can deep learning models learn GUI element features and
their composition in face of the unique characteristics of GUIs and
3.1 Research Questions
GUI elements? As region classification can be well supported by a CNN-based
The accuracy of bounding box regression. Deep learning image classifier [32], the study focuses on three research questions
based object detection learns a statistical regression model to predict (RQs) on region detection in GUI element detection task:
the bounding box of an object. This regression model makes the • RQ1 Performance: How effective can different methods detect
prediction in the feature map of a high layer of the CNN, where the region of non-text GUI elements, in terms of the accuracy of
one pixel stands for a pixel block in the original image. Can such predicted bounding boxes and the coverage of GUI elements?
statistical regression satisfy the high-accuracy requirement of region • RQ2 Sensitivity: How sensitive are deep learning techniques to
detection, in face of large in-class variance of GUI element and packed anchor-box settings and amount of training data?
or close-by GUI elements? • RQ3 Text detection: Does scene text recognition fit better for
The impact of model architectures, hyperparameters and GUI text detection than OCR technique? Which option, separated
training data. Faster RCNN [41] and YOLOv2 [39]) have been versus unified text and non-text detection, is more appropriate?
Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA

the qualified neighbor points gradually, to identify the connected

regions and filters out the noise from the complex background.
Faster RCNN [41] is a two-stage anchor-box-based deep learn-
ing technique for object detection. It first generates a set of region
proposals by a region proposal network (RPN), also called as region
of interests (RoIs), which likely contain objects. RPN uses a fixed set
of user-defined boxes with different scales and aspect ratios (called
anchor boxes) and computes these anchor boxes in each point in the
Figure 2: GUI elements distribution in our dataset feature map. For each box, RPN then computes an objectness score
3.2 Experiment Setup to determine whether it contains an object or not, and regresses it
3.2.1 Dataset. We leverage Rico dataset [16] to construct our ex- to fit the actual bounding box of the contained object. The second
perimental dataset. In this study, we consider 15 types of commonly stage is a CNN-based image classifier that determines the object
used GUI elements in the Android Platform (see examples in Fig- class in the RoIs.
ure 1). The Rico dataset contains 66,261 GUIs, and we filter out YOLOv3 [39] is an one-stage anchor-box-based object detec-
15,737 GUIs. Among them, 5,327 GUIs do not belong to the app tion technique. Different from the manually-defined anchor box of
itself, they are captured outside the app, such as Android home Faster-RCNN, YOLOv3 uses k-means method to cluster the ground
screen, a redirection to the social media login page (e.g. Facebook) truth bounding boxes in the training dataset, and takes the box
or to a browser. We identify them by checking whether the package scale and aspect ratio of the k centroids as the anchor boxes. It also
name in the metadata for each GUI is different from the app’s true extracts image features using CNN, and for each grid of the feature
package name. 2,066 GUIs do not have useful metadata, which only map, it generates a set of bounding boxes. For each box, it computes
contain elements that describe the layout, or elements with invalid the objectness scores, regresses the box coordinates and classifies
bounds, or do not have visible leaf elements. 709 of them do not the object in the bounding box at the same time.
contain any of the 15 elements. The rest 7,635 GUIs are removed CenterNet [18] is an one-stage anchor-free object detection
because they only contain text elements or non-text elements. As a technique. Instead of generating bounding box based on the pre-
result, we obtain 50,524 GUI screenshots. These GUIs are from 8,018 defined anchor boxes, it predicts the position of the top-left and
Android mobile applications of 27 categories. These GUIs contain bottom-right corners and the center of an object, and then assem-
923,404 GUI elements, among which 426,404 are non-text elements bles them to get the bounding box of an object. It computes the
and 497,000 are text elements. We remove the standard OS status distance between each top-left corner and each bottom-right corner,
and navigation bars from all GUI screenshots as they are not part of and outputs the bounding box of the matched pair if the distance
application GUIs. We obtain the bounding-box and class of the GUI of them is smaller than a threshold and the center point of the
elements from the corresponding GUI metadata. Figure 2 shows bounding box has a certerness score higher than a threshold.
the distribution of GUIs per application and the number of GUI Tesseract [43] is an OCR tool for document texts. It consists of
elements per GUI. Compared with the number of objects per image two steps: text line detection and text recognition. Only the text
in COCO2015, our GUI images are much more packed. We split line detection is relevant to our study. The Tesseract’s text line
these 50,524 GUIs into train/validation/test dataset with a ratio of detection is old-fashioned. It first converts the image into binary
8:1:1 (40K:5K:5k). All GUIs of an application will be in only one split map, and then performs a connected component analysis to find
to avoid the bias of “seen samples” across training, validation and the outlines of the elements. These outlines are then grouped into
testing. We perform 5-fold cross-validation in all the experiments. blobs, which are further merged together. Finally, it merges text
lines that overlap at least half horizontally.
3.2.2 Baselines. The baseline methods used in this study include: EAST [56] is a deep learning technique to detect text in natural
REMAUI [34] detects text and non-text elements separately. scenes. An input image is first fed into a feature pyramid network.
For text elements, it uses the OCR tool Tesseract [43]. For non- EAST then computes six values for each point based on the final
text elements, it detects the structural edge of GUI elements using feature map, namely, an objectness score, top/left/bottom/right
Canny edge’s detection [8], by using Gaussian filter to smooth the offsets and a rotation angle. For this baseline, we directly use the
image and reduce noises, and then performing multi-stage filtering pre-trained model to detect texts in GUIs without any fine-tuning.
to identify the true edges in images. After that, REMAUI performs
edge merging, obtains the contours and obtains the bounding box 3.2.3 Model Training. For faster RCNN, YOLOv3 and CenterNet,
of the GUI elements by merging partial overlapping regions. We use we initialize their parameters using the corresponding pre-trained
the REMAUI tool [44] provided by its authors in our experiments. models of COCO object detection dataset and fine-tune all parame-
Xianyu [53] is a tool developed by the Alibaba to generate code ters using our GUI training dataset. We train each model for 160
from GUI images. We only use the element detection part of this iterations with a batch size of 8, and use Adam as the optimizer.
tool. Xianyu binarizes the image and performs horizontal/vertical Faster RCNN uses ResNet-101 [25] as the backbone. YOLOv3 uses
slicing, i.e., cutting the whole images horizontally/vertically in half, Darknet-53[39] as the backbone. CenterNet uses Hourglass-52 [51]
recursively to obtain the GUI elements. It uses the edge detection as the backbone. For Xianyu and REMAUI, we perform the parame-
method with Laplace filter, which highlights regions of rapid inten- ter tuning and use the best setting in all our evaluation. We perform
sity change, to detect the edges and get contours in the binarized non-maximum suppression (NMS) to remove highly-duplicated pre-
image. Then, it leverages the flood fill algorithm [48], which merge dictions in all experiments. It keeps the prediction with the highest
ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA Chen, J., Xie, M., Xing, Z., Chen, C., Xu, X., Zhu, L. and Li, G.

Table 2: Performance: non-text element detection (IoU>0.9)

Method #bbox Precision Recall F1
REMAUI 54,903 0.175 0.238 0.201
Xianyu 47,666 0.142 0.168 0.154
Faster-RCNN 39,995 0.440 0.437 0.438
YOLOv3 36,191 0.405 0.363 0.383
CenterNet 36,096 0.424 0.380 0.401
as that of deep learning models as the IoU threshold increases,
Figure 3: Performance at different IoU thresholds but their F1-scores are much lower than those of deep learning
objectness in the results and removes others if they have a IoU with models. This suggests that the detected element regions by these
the selected object over a certain value. We find the best object old-fashioned methods are mostly noise, but when they do locate
confidence threshold for each model using the validation dataset. real elements, the detected bounding boxes are fairly accurate.
All codes and models are released at our Github repository1 .
3.3.2 Performance Comparison. We observe that if the detected
3.2.4 Metrics. For region detection evaluation, we ignore the class bounding box has <0.9 IoU over the corresponding GUI element,
prediction results and only evaluate the ability of different methods not only does the box miss some portion of this element, but it
to detect the bounding box of GUI elements. We use precision, also includes some portions of adjacent elements due to the packed
recall and F1-score to measure the performance of region detection. characteristic of GUI design. Therefore, we use IoU> 0.9 as an
Precision is T P/(T P + F P) and recall is T P/(T P + F N ). True positive acceptable accuracy of bounding box prediction. Table 2 shows the
(TP) refers to a detected bounding box which matches a ground overall performance of the five methods at IoU>0.9 threshold for
truth box. False positive (FP) refers to a detected box which does not detecting non-text GUI elements.
match any ground truth boxes. False negative (FN) refers to a ground Xianyu performs the worst, with all metrics below 0.17. We ob-
truth bounding box which is not matched by any detected boxes. serve that Xianyu works fine for simple GUIs, containing some GUI
We compute F1-score as: F 1 = (2 × Precision × Recall)/(Precision + elements on a clear or gradient background (e.g., Xianyu-(c)/(d) in
Recall). TP is determined based on the Intersection over Union Figure 6). When the GUI elements are close-by or placed on a com-
(IoU) of the two boxes. IoU is calculated by dividing the intersection plex background image, Xianyu’s slicing method and its background
area I of the two boxes A and B by the union area of the two boxes, de-noising algorithms do not work well. For example, in Xianyu-
i.e., I /(A + B − I ). A detected box is considered as a TP if the highest (a)/(b) in Figure 6, it misses most of GUI elements. Xianyu performs
IoU of this box with any ground-truth boxes in the input GUI image slicing by the horizontal or vertical lines across the whole GUI. Such
is higher than a predefined IoU threshold. Each ground truth box lines often do not exist in GUIs, especially when they have complex
can only be matched at most once and NMS technique is used background images (Xianyu-(a)) or the GUI elements are very close-
to determine the optimal matching results. Considering the high by (Xianyu-(b)). This results in many under-segmentation of GUI
accuracy requirement of GUI element detection, we take the IoU images and the misses of many GUI elements. Furthermore, Xianyu
threshold 0.9 in most of our experiments. sometimes may over-segment the background image (Xianyu-(a)),
resulting in many noise non-GUI-element regions.
3.3 Results - RQ1 Performance REMAUI performs better than Xianyu, but it is still much worse
This section reports the performance of five methods for detecting than deep learning models. REMAUI suffers from similar problems
the regions of non-text GUI elements in a GUI. In this RQ, Faster as Xianyu, including ineffective background de-noising and over-
RCNN uses the customized anchor-box setting and YOLOv3 uses segmentation. It outperforms Xianyu because it merges close-by
k=9 (see Section 3.4.1). The models are trained using all 40k training edges to construct bounding boxes instead of the simple slicing
data and tested on 5k GUI images in 5-fold cross validation setting. method by horizontal/vertical lines. However, for GUIs with im-
age background, its edge merging heuristics often fail due to the
3.3.1 Trade off between bounding-box accuracy and GUI-element
noisy edges of physical-world objects in the images. As such, it
coverage. Figure 3 shows the performance of five methods at dif-
often reports some non-GUI-element regions of the image as ele-
ferent IoU thresholds. The F1-score of all deep learning models
ment regions, or erroneously merges close-by elements, as shown
drop significantly when the IoU threshold increases from 0.5 to
in Figure 6. Furthermore, REMAUI merges text and non-text re-
0.9, with the 31%, 45% and 28% decrease for Faster-RCNN, YOLOv3
gion heuristically, which are not very reliable either (see the text
and CenterNet respectively. The bounding box of a RoI is predicted
elements detected as non-text elements in REMAUI-(b)/(c)/(d)).
by statistical regression in the high-layer feature map of the CNN,
Deep learning models perform much better than old fashioned
where one pixel in this abstract feature map corresponds to a pixel
methods. In Figure 6, we see that they all locate some GUI elements
block in the original image. That is, a minor change of the predicted
accurately, even those overlaying on the background picture. These
coordinates in the abstract feature map will lead to a large change
models are trained with large-scale data, “see” many sophisticated
in the exact position in the original image. Therefore, deep learning
GUIs, and thus can locate GUI elements even in a noisy background.
models either detect more elements with loose bounding boxes
However, we also observe that the detected bounding boxes by
or detect less elements with accurate bounding boxes. In contrast,
deep learning models may not be very accurate, as they are esti-
the F1-score of REMAUI and Xianyu does not drop as significantly
mated by a statistical regression model. The two-stage model Faster
1 [Link] RCNN outperforms the other two one-stage models YOLOv3 and
Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA

Table 3: Impact of anchor-box settings (IoU>0.9) is drawn from the frequent scales and aspect ratios of the GUI
Setting Precision Recall F1 elements in our dataset. Considering the size of GUI elements, we
Faster RCNN Default 0.433 0.410 0.421 add two small scales 32 and 64. Furthermore, we add two more
Faster RCNN Customized 0.440 0.437 0.438 aspect ratios to accommodate the large variance of GUI elements.
Faster RCNN Union 0.394 0.469 0.428 For YOLOv3, we use two k settings: 5 and 9, which are commonly
Faster RCNN Intersection 0.452 0.460 0.456 used in the literature. YOLOv3 automatically derives anchor-box
YOLOv3 k=5 0.394 0.333 0.361 metrics from k clusters of GUI images in the dataset. All models
YOLOv3 k=9 0.405 0.363 0.383 are trained using 40k training data and tested on 5k GUI images.
YOLO Union 0.372 0.375 0.373 Table 3 shows the model performance (at IoU>0.9) of these differ-
YOLO Intersection 0.430 0.424 0.427 ent anchor-box settings. It is somehow surprising that there is only
Table 4: Impact of amount of training data (IoU>0.9) a small increase in F1 when we use more anchor-box scales and
aspect ratios. We further compare the TPs of different anchor-box
Method Size Precision Recall F1
settings. We find that 55% of TPs overlap between the two settings
2K 0.361 0.305 0.331
for Faster RCNN, and 67% of TPs overlap between the two settings
Faster-RCNN 10K 0.403 0.393 0.398
for YOLOv3. As the scales and aspect ratios of GUI elements follow
40K 0.440 0.437 0.438
standard distributions, using a smaller number of anchor boxes can
2K 0.303 0.235 0.265
still covers a large portion of the element distribution.
YOLOv3 10K 0.337 0.293 0.313
As different settings detect some different bounding boxes, we
40K 0.405 0.363 0.383
want to see if the differences may complement each other. To that
2K 0.319 0.313 0.316
end, we adopt two strategies to merge the detected boxes by the
CenterNet 10K 0.328 0.329 0.329
two settings: union strategy and intersection strategy. For two
40K 0.424 0.380 0.401
overlapped boxes, we take the maximum objectness of them, and
CenterNet. As discussed in Section 2.1, GUI elements have large then merge the two boxes by taking the union/intersection area for
in-class variance and high cross-class similarity. Two stage models union/interaction strategy. For the rest of the boxes, we directly
perform region detection and region classification in a pipeline so keep them. We find the best object confidence threshold for the
that the region detection and region classification are less mutually combined results using the validation dataset. The union strategy
interfered, compared with one-stage models that perform region does not significantly affect the F1, which means that making the
detection and region classification simultaneously. bounding boxes larger is not very useful. In fact, for the boxes
Between the two one-stage models, anchor-free CenterNet out- which are originally TPs by one setting, the enlarged box could
performs anchor-box-based YOLOv3 at IoU>0.9. However, YOLOv3 even become FPs. However, the intersection strategy can boost the
performs better than CenterNet at lower IoU thresholds (see Fig- performance of both Faster RCNN and YOLOv3, achieving 0.456
ure 3). Anchor-free model is flexible to handle the large in-class and 0.427 in F1 respectively. It is reasonable because the intersection
variance of GUI elements and GUI texts (see more experiments area is confirmed by the two settings, and thus more accurate.
on GUI text detection in Section 3.5). However, as shown in Fig-
ure 6, this flexibility is a double-blade, which may lead to less 3.4.2 Amount of Training Data. In this experiment, Faster RCNN
accurate bounding boxes, or bound several elements in one box uses the customized anchor-box setting and YOLOv3 uses k=9. We
(e.g., CenterNet-(a)/(d)). Because GUI elements are often close-by train the models with 2K, 10K, 40K training data separately, and test
or packed in a GUI, CenterNet very likely assembles the top-left the models on the same 5k GUI images. Each 2k- or 10k experiment
and bottom-right corners of different GUI elements together, which uses randomly selected 2k or 10k GUIs in the 40k training data.
leads to the wrong bounding boxes. As shown in Table 4, the performance of all models drops as the
training data decreases. This is reasonable because deep learning
Deep learning models significantly outperform old-fashioned de-
models cannot effectively learn the essential features of the GUI
tection methods. Two-stage anchor-box-based models perform the
elements without sufficient training data. The relative performance
best in non-text GUI element detection task. But it is challenging
of the three models is consistent at the three training data sizes,
for the deep learning models to achieve a good balance between
with YOLOv3 always being the worst. This indicates the difficulty
the accuracy of the detected bounding boxes and the detected GUI
in training one-stage anchor-box model. Faster RCNN with 2k (or
elements, especially for anchor-free models.
10k) training data achieves the comparable or higher F1 than that of
YOLOv3 and CenterNet with 10k (or 40k) training data. This result
3.4 Results - RQ2 Sensitivity further confirms that two-stage model fits better for GUI element
This section reports the sensitivity analysis of the deep learning detection tasks than one-stage model, and one-stage anchor-free
models for region detection from two aspects: anchor-box settings model performs better than one-stage anchor-box model.
and amount of training data. Anchor-box settings do not significantly affect the performance of
3.4.1 Anchor-Box Settings. For Faster RCNN, we use two settings: anchor-box-based models, because a small number of anchor boxes
the default setting (three anchor-box scales - 128, 256 and 512, and can cover the majority of GUI elements. Two-stage anchor-box-
three aspect ratios - 1:1, 1:2 and 2:1); and the customized setting based model is the easiest to train, which requires one magnitude
(five anchor-box scales - 32, 64, 128, 256 and 512, and four aspect less training data to achieve comparable performance as one-stage
(width:height) ratios - 1:1, 2:1, 4:1 and 8:1). This customized setting model. One stage anchor-box model is the most difficult to train.
ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA Chen, J., Xie, M., Xing, Z., Chen, C., Xu, X., Zhu, L. and Li, G.

Table 5: Text detection: separated versus unified processing Table 6: Text detection: OCR versus scene text
Method Element Precision Recall F1 Method Precision Recall F1
nontext-only 0.440 0.437 0.438 Tesseract 0.291 0.518 0.372
nontext 0.379 0.436 0.405 EAST 0.402 0.720 0.516
Faster-RCNN
mix text 0.275 0.250 0.262 REMAUI 0.297 0.489 0.369
both 0.351 0.359 0.355 Xianyu 0.272 0.481 0.348
nontext-only 0.405 0.363 0.383
YOLOv3
non-text 0.325 0.347 0.335 results in order to filter out false positives. But it does not signifi-
mix text 0.319 0.263 0.288 cantly change the performance of GUI text detection. As EAST is
both 0.355 0.332 0.343
nontext-only 0.424 0.380 0.401
specifically designed for scene text recognition, its performance is
non-text 0.321 0.397 0.355 significantly better than using generic object detection models for
CenterNet
mix text 0.416 0.319 0.361 GUI text detection (see Table 5). EAST detects almost all texts in
both 0.391 0.385 0.388 a GUI, including those on the GUI widgets (e.g., the button labels
3.5 Results - RQ3 Text Detection in Figure 4(c)). However, those texts on GUI widgets are consid-
ered as part of the widgets in our ground-truth data, rather than
3.5.1 Separated versus Unified Text/Non-Text Element Detection.
stand-alone texts. This affects the precision of EAST against our
All existing works detect GUI text separately from non-text ele-
ground-truth data, even though the detected texts are accurate.
ments. This is intuitive in that GUI text and non-text elements have
Figure 4 presents some detection results. Tesseract achieves the
very different visual features. However, we were wondering if this
comparable results as EAST only for the left side of Figure 4(d),
is a must or text and non-text elements can be reliably detected by
where text is shown on a white background just like in a document.
a single model. To answer this, we train Faster RCNN, YOLOv3 and
From all other detection results, we can observe the clear advantages
CenterNet to detect both text and non-text GUI elements. Faster
of treating GUI text as scene text than as document text. First, EAST
RCNN uses the customized anchor-box setting and YOLOv3 uses
can accurately detect text in background image (Figure 4(a)), while
k=9. The model is trained with 40k data and tested on 5k GUI im-
Tesseract outputs many inaccurate boxes in such images. Second,
ages. In this RQ, both non-text and text elements in GUIs are used
EAST can detect text in a low contrast background (Figure 4(b)),
for model training and testing.
while Tesseract often misses such texts. Third, EAST can ignore non-
Table 5 shows the results. When trained to detect text and non-
text elements (e.g., the bottom-right switch buttons in Figure 4(b),
text elements together, Faster RCNN still performs the best in terms
and the icons on the left side of Figure 4(d)), while Tesseract often
of detecting non-text elements. But the performance of all three
erroneously detects such non-text elements as text elements.
models for detecting non-text elements degrades, compared with
the models trained to detect non-text elements only. This indicates GUI text and non-text elements should be detected separately. Nei-
that mixing the learning of text and non-text element detection ther OCR techniques nor generic object detection models can reliably
together interfere with the learning of detecting non-text elements. detect GUI texts. As GUI texts have the characteristics of scene text,
CenterNet performs much better for detecting text elements than the deep learning scene text recognition model can be used (even
Faster RCNN and YOLOv3, which results in the best overall per- without fine-tuning) to accurately detect GUI texts.
formance for the mixed text and non-text detection. CenterNet
is anchor-free, which makes it flexible to handle large variance 4 A NOVEL APPROACH
of text patterns. So it has comparable performance for text and Based on the findings in our empirical study, we design a novel
non-text elements. In contrast, anchor-box-based Faster RCNN and approach for GUI element detection. Our approach combines the
YOLOv3 are too rigid to reliably detect text elements. However, the simplicity of old-fashioned computer vision methods for non-text-
performance of CenterNet in detecting text elements is still poor. element region detection, and the mature, easy-to-deploy deep
Text elements always have space between words and lines. Due learning models for region classification and GUI text detection (Sec-
to the presence of these spaces, CenterNet often detects a partial tion 4.1). This synergy achieves the state-of-the-art performance
text element or erroneously groups separate text elements as one for the GUI element detection task (Section 4.2).
element when assembling object corners.
4.1 Approach Design
3.5.2 OCR versus Scene Test Recognition. Since it is not feasible to
Our approach detects non-text GUI elements and GUI texts sepa-
detect text and non-text GUI elements within a single model, we
rately. For GUI text detection, we simply use the pre-trained state-
want to investigate what is the most appropriate method for GUI
of-the-art scene text detector EAST [56]. For non-text GUI element
text detection. All existing works (e.g., REMAUI, Xianyu) simply use
detection, we adopt the two-stage design, i.e, perform region detec-
OCR tool like Tesseract. We observe that GUI text is more similar
tion and region classification in a pipeline. For region detection, we
to scene text than to document text. Therefore, we adopt a deep
develop a novel old-fashioned method with a top-down coarse-to-
learning scene text recognition model EAST for GUI text detection,
fine strategy and a set of GUI-specific image processing algorithms.
and compare it with Tesseract. We directly use the pre-trained EAST
For region classification, we fine-tune the pre-trained ResNet50
model without any fine tuning on GUI text.
image classifier [25] with GUI element images.
As shown in Table 6, EAST achieves 0.402 in precision, 0.720 in
recall and 0.516 in F1, which is significantly higher than Tesseract 4.1.1 Region Detection for Non-Text GUI Elements. According to
(0.291 in precision, 0.518 in recall and 0.372 in F1). Both Xianyu the performance and sensitivity experiments results, we do not
and REMAUI perform some post-processing of the Tesseract’s OCR want to use generic deep learning object detection models [18,
Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA

Figure 4: Examples: OCR versus scene text

38, 41]. First, they demand sufficient training data, and different
model designs require different scale of training data to achieve
stable performance. Furthermore, the model performance is still
less optimal even with a large set of training data, and varies across
different model designs. Second, the nature of statistical regression
based region detection cannot satisfy the high accuracy requirement
of GUI element detection. Unlike generic object detection where Figure 5: Our method for non-text GUI element detection
a typical correct detection is defined loosely (e.g, IoU>0.5) [19]), [45] to compute the boundary of the block and produce a block
detecting GUI elements is a fine-grained recognition task which map. In Figure 5, we show the detected block in different colors
requires a correct detection that covers the full region of the GUI for the presentation clarity. Note that blocks usually contain some
elements as accurate as possible, but the region of non-GUI elements GUI elements, but some blocks may correspond to a particular GUI
and other close-by GUI elements as little as possible. Unfortunately, element.
neither anchor-box based nor anchor-free models can achieve this Next, our method generates a binary map of the input GUI,
objective, because they are either too strict or too flexible in face of and for each detected block, it segments the corresponding region
large in-class variance of element sizes and texture, high cross-class of the binary map. Binarization simplifies the input image into a
shape similarity, and the presence of close-by GUI elements. black-white image, on which the foreground GUI elements can be
Unlike deep learning models, old-fashioned methods [34, 53] do separated from the background. Existing methods [34, 53] perform
not require any training which makes them easy to deploy. Fur- binarization through Canny edge detection [8] and Sobel edge
thermore, when old fashioned methods locate some GUI elements, detection [21], which are designed to keep fine texture details in
the detected bounding boxes are usually accurate, which is desir- nature scene images. Unfortunately, this detail-keeping capability
able. Therefore, we adopt old-fashioned methods for non-text GUI- contradicts the goal of GUI element detection, which is to detect the
element region detection. However, existing old-fashioned methods shape of GUI elements, rather than their content and texture details.
use a bottom-up strategy which aggregates the fine details of the For example, we want to detect an ImageView element no matter
objects (e.g., edge or contour) into objects. This bottom-up strategy what objects are shown in the image (see Figure 6). We develop
performs poorly, especially affected by the complex background or a simple but effective binarization method based on the gradient
objects in the GUIs and GUI elements. As shown in Figure 5, our map [22] of the GUI images. A gradient map captures the change of
method adopts a completely different strategy: top-down coarse-to- gradient magnitude between neighboring pixels. If a pixel has small
fine. This design carefully considers the regularity of GUI layouts gradient with neighboring pixels, it becomes black on the binary
and GUI-element shapes and boundaries, as well as the significant map, otherwise white. As shown in Figure 5, the GUI elements
differences between the shapes and boundaries of artificial GUI stand out from the background in the binary map, either as white
elements and those of physical-world objects. region on the black background or black region with white edge.
Our region detection method first detects the layout blocks of a Our method uses the connected component labeling [42] to
GUI. The intuition is that GUIs organize GUI elements into distinct identify GUI element regions in each binary block segment. It takes
blocks, and these blocks generally have rectangle shape. Xianyu as input the binarized image and performs two-pass scanning to
also detects blocks, but it assumes the presence of clear horizontal label the connected pixels. As GUI elements can be any shape, it
and vertical lines. Our method does not make this naive assumption. identifies a smallest rectangle box that covers the detected regions
Instead, it first uses the flood-filling algorithm [48] over the grey- as the bounding boxes. Although our binarization method does
scale map of the input GUI to obtain the maximum regions with not keep many texture details of non-GUI objects, the shape of
similar colors, and then uses the shape recognition [37] to determine non-GUI objects (e.g., those buildings in the pictures) may still be
if a region is a rectangle. Each rectangle region is considered as present in the binary map. These noisy shapes interfere existing
a block. Finally, it uses the Suzuki’s Contour tracing algorithm bottom-up aggregation methods [8, 45] for GUI element detection,
ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA Chen, J., Xie, M., Xing, Z., Chen, C., Xu, X., Zhu, L. and Li, G.

Figure 6: Region detection results for non-text GUI element: our method versus five baselines
Table 7: Detection performance of our approach (IoU>0.9) is pre-trained with the ImageNet data. We fine-tune the pre-trained
Elements Precision Recall F1 model with 90,000 GUI elements (6,000 per element type) randomly
non-text 0.503 0.545 0.523 selected from the 40k GUIs in our training dataset.
text 0.589 0.547 0.516
both 0.539 0.612 0.573 4.1.3 GUI Text Detection. Section 3.5 shows that GUI text should
Table 8: Region classification results for TP regions be treated as scene text and be processed separately from non-
Non-text elements All elements text elements. Furthermore, scene text recognition model performs
Method #bbox Accuracy #bbox Accuracy much better than generic object detection models. Therefore, we
FasterRCNN 18,577 0.68 34,915 0.68 use the state-of-the-art deep-learning scene text detector EAST [56]
YOLOv3 15,428 0.64 32,225 0.65 to detect GUI text. As shown in Figure 4(c), EAST may detect texts
Centernet 16,072 0.68 36,803 0.66 that are part of non-image GUI widgets (e.g., the text on the buttons).
Our method 21,977 0.86 53,027 0.91 Therefore, if the detected GUI text is inside the region of a non-
Table 9: Overall results of object detection (IoU > 0.9) image GUI widgets, we discard this text.
Non-text elements All elements
Method Precision Recall F1 Precision Recall F1 4.2 Evaluation
Faster-RCNN 0.316 0.313 0.315 0.269 0.274 0.271 We evaluate our approach on the same testing data used in our
YOLOv3 0.274 0.246 0.260 0.258 0.242 0.249 empirical study by 5-fold cross-validation. Table 7 shows the region-
CenterNet 0.302 0.270 0.285 0.284 0.280 0.282 detection performance for non-text, text and both types of elements.
Xianyu 0.122 0.145 0.133 0.270 0.405 0.324
For non-text GUI elements, our approach performs better than
REMAUI 0.151 0.205 0.173 0.296 0.449 0.357
Our method 0.431 0.469 0.449 0.490 0.557 0.524 the best baseline Faster RCNN (0.523 versus 0.438 in F1). For text
elements, our approach is overall the same as EAST. It is better than
which results in over-segmentation of GUI elements. In contrast,
EAST in precision, because our approach discards some detected
our top-down detection strategy minimizes the influence of these
texts that are a part of GUI widgets. But this degrades the recall.
non-GUI objects, because it uses relaxed grey-scale map to detect
For text and non-text elements as a whole, our approach performs
large blocks and then uses strict binary map to detect GUI elements.
better than the best baseline CenterNet (0.573 versus 0.388 in F1).
If a block is classified as an image, our method will not further
Figure 6 shows the examples of the detection results by our
detect GUI elements in this block.
approach and the five baselines. Compared with REMAUI and Xi-
4.1.2 Region Classification for Non-Text GUI Elements. For each anyu, our method detects much more GUI elements and much less
detected GUI element region in the input GUI, we use a ResNet50 noisy non-GUI element regions, because of our robust top-down
image classifier to predict its element type. In this work, we consider coarse-to-fine strategy and GUI-specific image processing (e.g., con-
15 element types as shown in Figure 1. The Resnet50 image classifier nected component labeling rather than canny edge and contour).
Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA

Our method also detects more GUI elements than the three deep Table 1 and Section 3.2.2 summarizes the old-fashioned methods
learning models. Furthermore, it outputs more accurate bounding (e.g., REMAUI [34] and Xianyu [53]) designed for GUI element de-
boxes and less overlapping bounding boxes, because our method tection, the generic object detection models (Faster RCNN, YOLOv3
performs accurate pixel analysis rather than statistical regression in and CenterNet) applied for non-text GUI element detection, and
the high-layer of CNN. Note that deep learning models may detect the old-fashioned OCR tool Tesseract [43] and the state-of-the-art
objects in images as GUI elements, because there are GUI elements scene text detector EAST [56] for GUI text detection. Our empirical
of that size and with similar visual features. In contrast, our method study shows that old-fashioned methods perform poorly for both
detects large blocks that are images and treats such images as whole. text and non-text GUI element detection. Generic object detection
As such, our method suffers less over-segmentation problem. models perform better than old-fashioned ones, but they cannot
For failure analysis of our model, we conclude three main reasons satisfy the high accuracy requirement of GUI element detection.
when our model fails. First, same look and feel UI regions may Our method advances the state-of-the-art in GUI element detection
correspond to different types of widgets, such as text label versus by effectively assembling the effective designs of existing methods
text button without border. This is similar to the widget tappability and a novel GUI-specific old-fashioned region detection method.
issue studied in [47]. Second, the repetitive regions in a dense UI Besides, some researchers apply image captioning to generate
(e.g., Figure 6(b)) often have inconsistent detection results. Third, GUI code from GUI design [4, 11]. However, image captioning only
it is sometimes hard to determine whether a text region is a text predicts what elements are in a UI, but not the bounding box of
label or part of a widget containing text, for example, the spinner these elements. Therefore, pix2code assumes UIs use a small set
showing USA at the top of Figure 6(b). Note that these challenges of predefined fonts, shapes, sizes and layouts, to compensate for
affect all methods. We leave them as our future work. this shortage. However, due to this limitation, our experiments
Table 8 shows the region classification results of our CNN classi- showed that pix2code does not work on real-app UIs which use
fier and the three deep learning baselines. The results consider only much more diverse fonts/shapes/sizes/layouts. In comparison, in
true-positive bounding boxes, i.e., the classification performance this work, we study the object detection methods, which directly
given the accurate element regions. As text elements are outputted obtain the bounding box of the elements. We further propose a
by EAST directly, we show the results for non-text elements and all novel top-down coarse-to-fine method to detect elements.
elements. We can see that our method outputs more true-positive There are also some works [4, 34] which perform object detec-
GUI element regions, and achieves higher classification accuracy tion on other Platforms (e.g. iOS, website), and other kinds of design
(0.86 for non-text elements and 0.91 for all elements, and the other (e.g. sketch). In this work, we perform our experiments on Android
three deep models achieves about 0.68 accuracy). Our classification app UIs because of the availability of large-scale Rico dataset, while
accuracy is consistent with [32], which confirms that the effective- other datasets like REMAUI cannot effectively train and experiment
ness of a pipeline design for GUI element detection. deep learning models (see our results of training data size in Sec-
Table 9 shows the overall object detection results, i.e., the true- tion 3.4.2). However, since our model does not make any specific
positive bounding box with the correct region classification over all assumptions about Android UIs, we believe that our model could be
detected element regions. Among the three baseline models, Faster easily generalized to other platforms and other kinds of GUI design.
RCNN performs the best for non-text elements (0.315 in F1), but We leave them as future work because they demand significant
CenterNet, due to this model flexibility to handle GUI texts, achieves manual labelling effort. We release our tool to public.
the best performance for all elements (0.282 in F1). Compared with
these three baselines, our method achieves much better F1 for both 6 CONCLUSION
non-text elements (0.449) and all elements (0.524), due to its strong
This paper investigates the problem of GUI element detection. We
capability in both region detection and region classification.
identify four unique characteristics of GUIs and GUI elements, in-
cluding large in-class variance, high cross-class similarity, packed or
close-by elements and mix of heterogeneous objects. These charac-
5 RELATED WORK teristics make it a challenging task for existing methods (no matter
old fashioned or deep learning) to accurately detect GUI elements
GUI design, implementation and testing are important software
in GUI images. Our empirical study reveals the underperformance
engineering tasks, to name a few, GUI code generation [6, 11, 32],
of existing methods borrowed from computer vision domain and
GUI search [9, 10, 12, 26, 55], GUI design examination [33, 47, 54],
the underlying reasons, and identifies the effective designs of GUI
reverse-engineering GUI dataset [14, 16], GUI accessibility [13], GUI
element detection methods. Informed by our study findings, we de-
testing [5, 27, 30, 36, 49] and GUI security [15, 50]. Many of these
sign a new GUI element detection approach with both the effective
tasks require the detection of GUI elements. As an example, the RQ4
designs of existing methods and the GUI characteristics in mind.
in [49] shows exploiting exact widget locations by instrumentation
Our new method achieves the state-of-the-art performance on the
achieves significantly higher branch coverage than predicted loca-
largest-ever evaluation of GUI element detection methods.
tions in GUI testing, but widget detection (by YOLOv2) can interact
with widgets not detected by instrumentation. Our work focuses on
the foundational technique to improve widget detection accuracy, ACKNOWLEDGMENTS
which opens the door to keep the advantage of widget detection This research was partially supported by the Australian National
while achieving the benefits of instrumentation in downstream University Data61 Collaborative Research Project(CO19314) and
applications like GUI testing. Facebook gift funding.
ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA Chen, J., Xie, M., Xing, Z., Chen, C., Xu, X., Zhu, L. and Li, G.

REFERENCES [22] Rafael C. Gonzalez and Richard E. Woods. 2014. Digital image processing. Dorling
[1] Nikola Banovic, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2012. Kindersley.
Waken: reverse engineering usage information and interface structure from [23] Google. 2019. UI Automator. Retrieved March 2, 2020 from [Link]
software videos. In Proceedings of the 25th annual ACM symposium on User [Link]/training/testing/ui-automator
interface software and technology. 83–92. [24] Google. 2020. Build more accessible apps. Retrieved March 2, 2020 from https:
[2] Lingfeng Bao, Jing Li, Zhenchang Xing, Xinyu Wang, and Bo Zhou. 2015. scvRip- //[Link]/guide/topics/ui/accessibility
per: video scraping tool for modeling developers’ behavior using interaction data. [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual
In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, learning for image recognition. In Proceedings of the IEEE conference on computer
Vol. 2. IEEE, 673–676. vision and pattern recognition. 770–778.
[3] Lingfeng Bao, Deheng Ye, Zhenchang Xing, Xin Xia, and Xinyu Wang. 2015. [26] Forrest Huang, John F Canny, and Jeffrey Nichols. 2019. Swire: Sketch-based user
Activityspace: a remembrance framework to support interapplication information interface retrieval. In Proceedings of the 2019 CHI Conference on Human Factors in
needs. In 2015 30th IEEE/ACM International Conference on Automated Software Computing Systems. 1–10.
Engineering (ASE). IEEE, 864–869. [27] Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019. Humanoid: A
[4] Tony Beltramelli. 2018. pix2code: Generating code from a graphical user inter- Deep Learning-Based Approach to Automated Black-box Android App Testing. In
face screenshot. In Proceedings of the ACM SIGCHI Symposium on Engineering 2019 34th IEEE/ACM International Conference on Automated Software Engineering
Interactive Computing Systems. 1–6. (ASE). IEEE, 1070–1073.
[5] Carlos Bernal-Cardenas, Nathan Cooper, Kevin Moran, Oscar Chaparro, Andrian [28] Feng Lin, Chen Song, Xiaowei Xu, Lora Cavuoto, and Wenyao Xu. 2016. Sensing
Marcus, and Denys Poshyvanyk. 2020. Translating Video Recordings of Mo- from the bottom: Smart insole enabled patient handling activity recognition
bile App Usages into Replayable Scenarios. In 42nd International Conference on through manifold learning. In 2016 IEEE First International Conference on Con-
Software Engineering (ICSE ’20). ACM, New York, NY. nected Health: Applications, Systems and Engineering Technologies (CHASE). IEEE,
[6] Pavol Bielik, Marc Fischer, and Martin Vechev. 2018. Robust relational layout 254–263.
synthesis from examples for Android. Proceedings of the ACM on Programming [29] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Languages 2, OOPSLA (2018), 1–29. Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common
[7] Karl Bridge and Michael Satran. 2018. Windows Accessibility API overview. Re- objects in context. In European conference on computer vision. Springer, 740–755.
trieved March 2, 2020 from [Link] [30] Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang.
winauto/windows-automation-api-portal 2020. Owl Eyes: Spotting UI Display Issues via Visual Understanding. In Proceed-
[8] J. Canny. 1986. A Computational Approach to Edge Detection. IEEE Transactions ings of the 35th International Conference on Automated Software Engineering.
on Pattern Analysis and Machine Intelligence PAMI-8, 6 (Nov 1986), 679–698. [31] Microsoft. 2016. Introducing Spy++. Retrieved March 2, 2020
[Link] from [Link]
[9] Chunyang Chen, Sidong Feng, Zhengyang Liu, Zhenchang Xing, and Shengdong increment?view=vs-2019
Zhao. 2020. From Lost to Found: Discover Missing UI Design Semantics through [32] Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and
Recovering Missing Tags. Proceedings of the ACM on Human-Computer Interaction Denys Poshyvanyk. 2018. Machine learning-based prototyping of graphical user
4, CSCW (2020). interfaces for mobile apps. arXiv preprint arXiv:1802.02312 (2018).
[10] Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, [33] Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshy-
and Jinshui Wang. 2019. Gallery DC: Design Search and Knowledge Discovery vanyk. 2018. Automated reporting of GUI design violations for mobile apps. In
through Auto-created GUI Component Gallery. Proceedings of the ACM on Proceedings of the 40th International Conference on Software Engineering. 165–175.
Human-Computer Interaction 3, CSCW (2019), 1–22. [34] Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse engineering mobile
[11] Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. application user interfaces with remaui (t). In 2015 30th IEEE/ACM International
From UI design image to GUI skeleton: a neural machine translator to bootstrap Conference on Automated Software Engineering (ASE). IEEE, 248–259.
mobile GUI implementation. In Proceedings of the 40th International Conference [35] Suporn Pongnumkul, Mira Dontcheva, Wilmot Li, Jue Wang, Lubomir Bourdev,
on Software Engineering. 665–676. Shai Avidan, and Michael F Cohen. 2011. Pause-and-play: automatically linking
[12] Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xin Xia, Liming Zhu, John screencast video tutorials with applications. In Proceedings of the 24th annual
Grundy, and Jinshui Wang. 2020. Wireframe-based UI design search through ACM symposium on User interface software and technology. 135–144.
image autoencoder. ACM Transactions on Software Engineering and Methodology [36] Ju Qian, Zhengyu Shang, Shuoyan Yan, Yan Wang, and Lin Chen. 2020. RoScript:
(TOSEM) 29, 3 (2020), 1–31. A Visual Script Driven Truly Non-Intrusive Robotic Testing System for Touch
[13] Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guo- Screen Applications. In 42nd International Conference on Software Engineering
qiang Li, and Jinshui Wang. 2020. Unblind Your Apps: Predicting Natural- (ICSE ’20). ACM, New York, NY.
Language Labels for Mobile GUI Components by Deep Learning. In 42nd In- [37] Urs Ramer. 1972. An iterative procedure for the polygonal approximation of
ternational Conference on Software Engineering (ICSE ’20). ACM, New York, NY, plane curves. Computer Graphics and Image Processing 1, 3 (1972), 244âĂŞ256.
13 pages. [Link] [Link]
[14] Sen Chen, Lingling Fan, Chunyang Chen, Ting Su, Wenhe Li, Yang Liu, and Lihua [38] Joseph Redmon and Ali Farhadi. 2017. YOLO9000: better, faster, stronger. In
Xu. 2019. Storydroid: Automated generation of storyboard for Android apps. Proceedings of the IEEE conference on computer vision and pattern recognition.
In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 7263–7271.
IEEE, 596–607. [39] Joseph Redmon and Ali Farhadi. 2018. Yolov3: An incremental improvement.
[15] Sen Chen, Lingling Fan, Chunyang Chen, Minhui Xue, Yang Liu, and Lihua Xu. arXiv preprint arXiv:1804.02767 (2018).
2019. GUI-Squatting Attack: Automated Generation of Android Phishing Apps. [40] Steven P Reiss, Yun Miao, and Qi Xin. 2018. Seeking the user interface. Automated
IEEE Transactions on Dependable and Secure Computing (2019). Software Engineering 25, 1 (2018), 157–193.
[16] Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, [41] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn:
Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A mobile app dataset Towards real-time object detection with region proposal networks. In Advances
for building data-driven design applications. In Proceedings of the 30th Annual in neural information processing systems. 91–99.
ACM Symposium on User Interface Software and Technology. 845–854. [42] H. Samet and M. Tamminen. 1988. Efficient component labeling of images of
[17] Morgan Dixon and James Fogarty. 2010. Prefab: implementing advanced behav- arbitrary dimension represented by linear bintrees. IEEE Transactions on Pattern
iors using pixel-based reverse engineering of interface structure. In Proceedings Analysis and Machine Intelligence 10, 4 (1988), 579âĂŞ586. [Link]
of the SIGCHI Conference on Human Factors in Computing Systems. 1525–1534. 1109/34.3918
[18] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi [43] Ray Smith. 2007. An overview of the Tesseract OCR engine. In Ninth International
Tian. 2019. Centernet: Keypoint triplets for object detection. In Proceedings of Conference on Document Analysis and Recognition (ICDAR 2007), Vol. 2. IEEE, 629–
the IEEE International Conference on Computer Vision. 6569–6578. 633.
[19] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and [44] Mohian Soumik. 2019. pix2app. Retrieved March 2, 2020 from [Link]
Andrew Zisserman. 2009. The Pascal Visual Object Classes (VOC) Challenge. com/soumikmohianuta/pixtoapp
International Journal of Computer Vision 88, 2 (Sep 2009), 303âĂŞ338. https: [45] Satoshi Suzuki and KeiichiA be. 1985. Topological structural analysis of digi-
//[Link]/10.1007/s11263-009-0275-4 tized binary images by border following. Computer Vision, Graphics, and Image
[20] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Processing 30, 1 (1985), 32 – 46. [Link]
Andrew Zisserman. 2010. The pascal visual object classes (voc) challenge. Inter- [46] Amanda Swearngin, Mira Dontcheva, Wilmot Li, Joel Brandt, Morgan Dixon,
national journal of computer vision 88, 2 (2010), 303–338. and Andrew J Ko. 2018. Rewire: Interface design assistance from examples. In
[21] Rafael C. Gonzalez and Richard E. Woods. 1993. Digital image processing. Addison- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems.
Wesley. 1–12.
Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? ESEC/FSE ’20, November 8–13, 2020, Virtual Event, USA

[47] Amanda Swearngin and Yang Li. 2019. Modeling Mobile Interface Tappability Us- symposium on User interface software and technology. 183–192.
ing Crowdsourcing and Deep Learning. In Proceedings of the 2019 CHI Conference [53] Chen Yongxin, Zhang Tonghui, and Chen Jie. 2019. UI2code: How
on Human Factors in Computing Systems. 1–11. to Fine-tune Background and Foreground Analysis. Retrieved Feb 23,
[48] Shane Torbert. 2016. Applied computer science. Springer. 2020 from [Link]
[49] Thomas D White, Gordon Fraser, and Guy J Brown. 2019. Improving random foreground-analysis-2293652041/
GUI testing with image-based widget detection. In Proceedings of the 28th ACM [54] Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, Guoqiang
SIGSOFT International Symposium on Software Testing and Analysis. 307–317. Li, and Jinshui Wang. 2020. Seenomaly: Vision-Based Linting of GUI Animation
[50] Shengqu Xi, Shao Yang, Xusheng Xiao, Yuan Yao, Yayuan Xiong, Fengyuan Effects Against Design-DonâĂŹt Guidelines. In 42nd International Conference on
Xu, Haoyu Wang, Peng Gao, Zhuotao Liu, Feng Xu, et al. 2019. DeepIntent: Software Engineering (ICSE ’20). ACM, New York, NY, 12 pages. [Link]
Deep icon-behavior learning for detecting intention-behavior discrepancy in 10.1145/3377811.3380411
mobile apps. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and [55] Shuyu Zheng, Ziniu Hu, and Yun Ma. 2019. FaceOff: Assisting the Manifestation
Communications Security. 2421–2436. Design of Web Graphical User Interface. In Proceedings of the Twelfth ACM
[51] Jing Yang, Qingshan Liu, and Kaihua Zhang. 2017. Stacked hourglass network International Conference on Web Search and Data Mining. 774–777.
for robust facial landmark localisation. In Proceedings of the IEEE Conference on [56] Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He,
Computer Vision and Pattern Recognition Workshops. 79–87. and Jiajun Liang. 2017. EAST: an efficient and accurate scene text detector. In
[52] Tom Yeh, Tsung-Hsiang Chang, and Robert C Miller. 2009. Sikuli: using GUI Proceedings of the IEEE conference on Computer Vision and Pattern Recognition.
screenshots for search and automation. In Proceedings of the 22nd annual ACM 5551–5560.