Reverse Engineering Mobile Application

User Interfaces With REMAUI

Tuan Anh Nguyen, Christoph Csallner

Computer Science and Engineering Department
The University of Texas at Arlington
Arlington, TX 76019, USA
Email: tanguyen@[Link], csallner@[Link]

Abstract—When developing the user interface code of a mobile using Apple’s Xcode GUI builder introduced many bugs that
application, in practice a big gap exists between the digital later had to be corrected. Subjects produced these bugs even
conceptual drawings of graphic artists and working user interface though the study’s target layouts were much simpler than those
code. Currently, programmers bridge this gap manually, by commonly found in third-party mobile applications [56].
reimplementing the conceptual drawings in code, which is cum-
bersome and expensive. To bridge this gap, we introduce the first This challenge is compounded in practice. (1) First, custom
technique to automatically Reverse Engineer Mobile Application layouts are often desired but it is harder to create them with
User Interfaces (REMAUI). On a given input bitmap REMAUI a stock GUI builder. (2) Second, the conversion from user
identifies user interface elements such as images, texts, containers, interface concept drawing to user interface code is typically
and lists, via computer vision and optical character recognition performed many times during an application’s lifespan. The
(OCR) techniques. In our experiments on 488 screenshots of reason is that many development teams follow an iterative ap-
over 100 popular third-party Android and iOS applications,
REMAUI-generated user interfaces were similar to the originals,
proach, in which a user interface may undergo many revisions
both pixel-by-pixel and in terms of their runtime user interface during both initial software development and maintenance.
hierarchies. REMAUI’s average overall runtime on a standard This gap in the mobile application development process is
desktop computer was 9 seconds. significant as many mobile applications are being developed
and maintained. For example, In the USA over 90% of con-
I. I NTRODUCTION AND M OTIVATION sumers over 16 years of age use a mobile phone and more than
half of the mobile phones are smartphones, mostly running
Developing the user interface code of mobile applications
Android or iOS [52]. On these smartphones, people use mobile
is cumbersome and expensive in practice. Due to the early
applications to perform many tasks that have traditionally
consumer and entertainment focus of the two major platforms
been performed on desktop computers [28], [3], [52], [23].
Android and iOS and the high competitive pressure in the
Example tasks include reading and writing emails, listening to
mobile application market, users have come to expect mobile
music, watching movies, reading the news, and consuming and
user interfaces that are highly customized and optimized for
producing social media. To date, more than one million mobile
the task at hand [42], [45]. To satisfy this demand, mobile
applications have been released1 . Automating the conversion
user interfaces often deviate from their platforms’ standard
from user interface design drawings to working user interface
user interface (UI) components and provide their own novel or
code may therefore save a lot of time and money, which could
customized UI elements such as buttons, dividers, and custom
be put to better use.
element positioning and grouping.
Converting a conceptual drawing of a screen into good user
To create such optimized user interfaces, the development interface code is hard, as it is essentially a reverse engineering
process of mobile applications routinely incorporates non- task. As in other reverse engineering tasks, general principles
programmers. User experience (UX) designers and graphic have to be inferred from specific instances. For example, a
artists design, customize, and optimize each screen of the user suitable hierarchy of user interface elements has to be inferred
interface with a mix of prototyping techniques. Common pro- from a flat set of concrete pixels.
totyping techniques include paper-and-pencil and pixel-based
concept drawings created in Photoshop or similar graphic Compared to other reverse engineering tasks such as infer-
design tools [25], [17], [42], [30]. ring design documents from code [13], [21], [7], [15], [16],
an unusual additional challenge is that the input, i.e., the
Our key observation is that there is a gap in the produc- pixels, may originate from scanned handwriting and human
tion process, as user interface concept drawings have to be sketches with all their imperfections [53], [43], [6]. This
converted into working user interface code. Currently, these means that sets of pixels have to be grouped together and
conversions are done manually by programmers, which is recognized heuristically as images or text. Then groups of
cumbersome, error-prone, and expensive. While modern IDEs similar images and text have to be recognized heuristically as
such as Eclipse, Xcode, and Android Studio have powerful in- example elements of collections. And for the UI of innovative
teractive builders for graphical user interface (GUI) code [55], mobile applications, at each step the recognized elements may
[56], using such a GUI builder to re-create a complex user diverge significantly from the platform’s standard UI elements.
interface drawing is a complex task. For example, in an
evaluation of GUI builders on a set of small tasks, subjects 1 [Link]
 App  Screenshot   UI  View  Hierarchy   Source  Code  and  Resources  

Huong  
Reverse   Nguyen  
Export   You:  Hello,  
eng.  
how  are  
you  
Now  
Jing  Xu  
You:  Hi  hi  

Images   Text  

Source  Code  &  

Layout  DefiniDon  

Compile  
Deploy  
&  Run  

iOS   …  

Image   Text   Container   List  item  

Fig. 1. Example REMAUI use: The UI designer provides a conceptual UI drawing (left). REMAUI identifies UI elements such as lists of text and images and
arranges them in a suitable UI hierarchy. REMAUI then exports the inferred UI as source code and resource files, compiles them, and runs them on a phone.

For professional application development, one may wonder such as images, text, containers, and lists, via computer vision
if this reverse engineering step is artificial. That is, why are and optical character recognition (OCR) techniques. REMAUI
meaning and source code hierarchy of screen elements not further infers a suitable user interface hierarchy and exports
explicitly encoded in the conceptual design drawings if these the results as source code that is ready for compilation and
are done in digital tools such as Photoshop? One reason is that execution. The generated user interface closely mimics the user
some UX designers start with pencil on paper, so it would be interface of a corresponding real application. To summarize,
desirable to convert such drawings directly into working user the paper makes the following major contributions.
interface code.
• The paper describes REMAUI, the first technique for
More significantly, when UX designers create digital inferring mobile application user interface code from
bitmap images (typically by drawing them in Photoshop), the screenshots or conceptual drawings.
digital design tools do not capture the hierarchy information • To evaluate REMAUI, we implemented a prototype tool
that is needed by user interface code. More importantly, it is that generates the UI portion of Android applications.
not clear if UX designers and graphic artists want to think in This tool is freely available via the REMAUI web site.
terms of source code hierarchies. • In an evaluation on 488 screenshots of over 100 popular
While this gap is most apparent in forward engineering, third-party mobile applications, REMAUI-generated UIs
there may also exist a traditional reverse engineering scenario. were similar to the originals, pixel-by-pixel and in their
A developer may only have access to screenshots of a mobile runtime UI hierarchy.
application, maybe after losing all other software artifacts such
as the source code. In such a situation it would be desirable II. M OTIVATING E XAMPLE
to automatically infer from the screenshots the user interface
portion of the missing source code. As a motivating example, assume a UX designer has pro-
duced the screen design bitmap shown in the left of Figure 1.
This paper therefore identifies and addresses three prob- The top of the screen contains the user’s profile image and an
lems in mobile application development. In reverse engineer- icon. Below is a list, in which each entry has a person’s image
ing, we address the problem of inferring the user interface on the left, the person’s name and text message in the middle,
code of a mobile application from screenshots. In forward and the message date on the right. List entries are separated
engineering, we address the gap between scanned pencil-on- by horizontal bars. The bottom of the screen has four icons
paper UI sketches and code as well as the gap between pixel- and their labels.
based UI sketches and code. While these problems occur at
different times in the development process, they share the task REMAUI infers from this bitmap working UI code, by
of pixels-to-code inference. mimicking the steps a programmer would take. REMAUI thus
uses vision and character recognition techniques to reason
Specifically, this paper introduces the first technique to about the screen bitmap. REMAUI groups related pixels into
automatically Reverse Engineer Mobile Application User In- text or images, lines of text into text boxes, related items into
terfaces (REMAUI). REMAUI automatically infers the user in- containers, and repeated elements into list elements. REMAUI
terface portion of the source code of a mobile application from thus identifies non-standard user interface components such as
screenshots or conceptual drawings of the user interface. On a arbitrarily shaped items (e.g., the round images on the left) and
given input bitmap REMAUI identifies user interface elements non-standard lists (e.g., using the special horizontal separator).
<RelativeLayout <!−− List Entry ... −−> > A. GUI View Hierarchy & Declarative GUI Programming
<ImageView <!−− Horizontal Bar ... −−> />
<ImageView android:id=”@+id/ImageView 1” The graphical user interface (GUI) of many modern desk-
android:layout width=”59dip” top and mobile platforms is structured as a view hierarchy [37],
android:layout height=”61dip” [2]. Such a hierarchy has two types of nodes, leaf nodes
android:layout marginLeft=”5dip” (images, buttons, text, etc.) and container nodes. The root view
android:layout marginTop=”0dip” represents an application’s entire space on screen. The root can
android:src=”@drawable/img 9” have many transitive children. Each child typically occupies a
android:scaleType=”fitXY” rectangular sub-region of its parent. Each view can have its
android:layout alignParentTop=”true” own parameters such as height, width, background color, and
android:layout alignParentLeft=”true”/> position. A view can be positioned relative to the root or other
<RelativeLayout <!−− Nested: Text Block (center) ... −−> >
<TextView <!−− Sender name ... −−> />
views such as its parent or siblings.
<TextView <!−− Message ... −−> /> Mobile platforms such as Android and iOS render a parent
</RelativeLayout> view before its children on screen. A child view thus hides
<TextView <!−− Message date (right) ... −−> /> parts of its parent. Siblings are drawn in the order they are
</RelativeLayout>
defined. A best practice is to minimize rendering time waste
Listing 1. REMAUI-generated layout for each list entry of Figure 1. Details by keeping hierarchies flat and avoiding view overlap.
are only shown for the left part of a list entry.
Given the relatively small mobile phone screen size, mobile
platforms make it easy to hide their default screen elements
REMAUI generates several XML files to capture the
such as the iOS title bar or the Android navigation bar.
screen’s static properties. In our example, the main XML file
Applications often use this feature to maximize screen size.
declares and positions the elements of the top and bottom rows
including icons and their labels. This file also contains a list To define basic GUI aspects, modern platforms provide two
view for the bulk of the screen content. The layout of each alternatives. The traditional desktop approach is construction
list entry is defined by the Listing 1 XML file. For example, it through regular program code [37]. The now widely rec-
positions a contact’s image and aligns it with the top left of its ommended alternative is declarative [2], [39], [24], e.g., via
parent (alignParentTop, alignParentLeft). REMAUI recognizes XML layout definition files in Android. Advanced GUI aspects
aligned text blocks such as the sender’s name and message, are then defined programmatically, which typically leads to a
groups them into a (nested) layout container (Listing 1), and combination of code and layout declaration files.
exports the recognized text fragments as an Android resource
Building an appealing user interface is hard [36], [37].
file. At application runtime the list entries are added by the
Besides understanding user needs, the GUI facilities of mod-
also generated Listing 2 Java source code.
ern platforms are complex and offer many similar concepts
public class MainActivity extends Activity { to choose from. This challenge is especially significant for
//.. developers new to their target platform. While each platform
private void addListView0() { provides standard documentation and sample code, these sam-
ListView v = (ListView) findViewById([Link] 0); ples often produce unappealing results.
final ArrayList<ListI> values = new ArrayList<ListI>();
[Link](new ListI([Link] 4, [Link] 9, R. B. Example GUI Framework: Android
[Link] 0, [Link] 1, [Link] 2));
//.. The Android standard libraries define various GUI con-
} tainers (“layout containers”) and leaf nodes (“widgets”). Ac-
} cording to an August 2012 survey of the 400 most popular
//.. non-game applications in the Google Play app store [47], the
Listing 2. REMAUI-generated Android (i.e., Java) source code that populates following containers were used most frequently: LinearLay-
Listing 1 list entries at application runtime. out (130 uses per application on average) places its children in
a single row or column; RelativeLayout (47) positions children
The generated UI code and layout definitions can be relative to itself or each other; FrameLayout (15) typically has
compiled with standard Android development tools. Moreover, a single child; ScrollView (9) is a scrollable FrameLayout; and
the code is similar to how a professional developer would im- ListView (7) lays out children as a vertical scrollable list.
plement the screen. For example, the generated code uses the The following widgets were used most frequently:
appropriate kinds of layout container such as RelativeLayout TextView (141) is read-only text; ImageView (62) is a bitmap;
for the list entries. A RelativeLayout can eliminate the need Button (37) is a device-specific text button; View (17) is
for some nested containers and thus keep the layout hierar- a generic view; EditText (12) is editable text; and Image-
chy relatively flat, which improves rendering performance at Button (11) is a device-independent button that shows an
application runtime. image. Besides the above, the Android library documentation
currently lists some additional two dozen widgets and some
three dozen layout containers.
III. BACKGROUND
C. Optical Character Recognition (OCR)
This section contains necessary background information on
GUI programming, modern mobile phone GUIs, and computer To infer UI code that closely reproduces the input con-
vision and optical character recognition (OCR). ceptual drawing, REMAUI distinguishes text from images and
Fig. 2. Example OCR performance at various granularity levels. Left to right: UI drawing and Tesseract-detected words, lines, blocks, and paragraphs.

captures the text as precisely as possible. Decades of research Using computer vision REMAUI approximates the boundaries
into optical character recognition (OCR) have produced spe- of each screen element such as text and images.
cialized methods for recognizing various kinds of text such
In its final steps REMAUI merges OCR and computer
as text in different sizes, fonts, and orientation, as well as
vision results (step 3) and in the merged data identifies
handwritten text [53], [43]. Generally it is easier to recognize
structures such as lists (step 4). REMAUI then exports the
text online (while it is being written) than offline. Similarly, it
inferred user interface as a combination of layout declarations
is easier to recognize print than handwriting.
and program source code for the given target mobile platform
Existing OCR tools perform relatively well if the input (step 5), compiles this combination to binaries, and runs the
consists of mostly text. A good example is single-column text binaries on an unmodified smartphone (step 6).
with few images. Current OCR tools perform worse if the text Not shown in Figure 3 is a pre-processing step in which
density is lower and text is arranged more freely and combined REMAUI removes standard operating system title and navi-
with images [27]. A good representative OCR tool is the pow- gation bars, if they are present. Since these screen areas are
erful and widely used open-source OCR engine Tesseract [48], standardized it is relatively easy to detect and remove them.
[50], which, for instance, Mathematica 9 uses to recognize text.
In the closely related task of segmenting pages (for example, A. Optical Character Recognition (Step 1)
to distinguish images and individual text columns), Tesseract
performs on par with commercial tools [50], [1]. First, REMAUI applies on the given input bitmap off-the-
shelf OCR word detection. Since optical character recognition
However, the limitations of such a powerful OCR tool suffers from false positives, REMAUI post-processes OCR
on complex inputs become apparent when subjecting it to results to remove candidate words that likely do not reflect
screenshots or conceptual UI drawings. For example, Figure 2 true words in the input. Figure 4 visualizes this process on
shows from left to right a conceptual drawing and Tesseract’s the example bitmap from Figure 3. At word-level detection,
results when detecting text at various granularity levels, i.e., REMAUI’s OCR system classifies several UI elements as a
words, lines, blocks, and paragraphs. In this example Tesseract word that are not a word but an image or a part of an image.
found all words but also classified as words non-words such as
the contacts’ images. In general, for the domain of conceptual To remove likely false positive words, REMAUI encodes
screen drawings and screenshots Tesseract’s precision and knowledge about its mobile phone UI domain as heuristics,
recall are often both below one in all granularity levels. So summarized in Table I. As an example, rule 3 encodes that on
even a powerful OCR tool may miss some words and classify a phone screen a word is likely not cut off and thus does not
non-text as words. extend beyond the border of the screen. This rule is specific
to phone screens and does not apply in all the settings the off-
the-shelf OCR engine may be applied in outside REMAUI.
IV. REMAUI OVERVIEW AND D ESIGN
TABLE I. H EURISTICS FOR ELIMINATING LIKELY FALSE POSITIVE
CANDIDATE WORDS FROM THE OCR RESULTS .
Figure 3 shows REMAUI’s six main processing steps. At
the core is a powerful off-the-shelf optical character recogni- # Name Heuristic
tion (OCR) engine (step 1). Since OCR produces false positive 1 Zero h=0∨w =0
2 Long w/h < 0.05 ∨ h/w < 0.05
candidate words, REMAUI filters the OCR results with its 3 Cut off x<0∨y <0∨x+w >W ∨y+h>H
domain-specific heuristics. Both to further compensate for 4 Conf. c ≤ 0.4
|eh /ew −h/w| |a−e|
OCR’s limitations and to identify non-text elements such as 5 Content c ≤ 0.7 ∧ ( max(e > 0.5 ∨ max(a,e) > 0.8)
h /ew ,h/w)
images, REMAUI combines OCR with a powerful off-the-shelf 6 No-text [\p{C}\s]* ∨ [ˆ\\x00-\\x7F]*

computer vision system (step 2).

The heuristics are given in terms of the input data, the OCR
In two-dimensional images computer vision techniques can results, and heuristic values computed by REMAUI. Specifi-
quickly detect features such as corners and edges. Computer cally, from the input UI screen available are its width (W ) and
vision has therefore been applied to diverse tasks such as height (H). The OCR system produces for each of its candidate
recognizing faces in two-dimensional images of the world or words the word’s height (h), width (w), area (a = w ∗ h),
to allow self-driving cars to detect the edge of the road [51]. font family and size, upper left corner coordinates (x, y), text
 (2)   (3)   (4)   (6)  
Computer   Merge   IdenAfy   Compile,  
Vision       lists   Deploy  
&  Run  

(1)    
OCR  

(5)  
Concept:     Export  
 Drawing  or    
 Screenshot  or   Images,  Text,   Generated  app  with  
Pencil  on  Paper   Source  Code,   inferred  UI  hierarchy  
Layout   running  on  an  
Box:     Box:     DefiniAon  
Word   Line   unmodified  phone  

Fig. 3. Overview of REMAUI processing steps: (1) Locate and extract candidate words and lines with OCR; (2) locate and extract candidate UI elements as
a hierarchy of nested bounding boxes using computer vision; (3) merge the results to improve recognition quality; (4) identify repeated items and summarize
them as collections; (5) export the constructed UI as a mobile application for a given platform; (6) compile and execute.

content (t), and confidence level (c). The confidence level B. Computer Vision (Step 2)
is derived from the distance of the word’s characters from
In this step REMAUI infers a first candidate view hierarchy.
idealized characters [49].
Two important observations are that (1) many vastly different
view hierarchies can lead to very similar if not identical on-
screen appearances and (2) a programmer will likely find some
Remove of these view hierarchies more valuable than others. REMAUI
invalid therefore follows carefully chosen heuristics to produce desir-
OCR words able view hierarchies that balance the following two goals.
The first goal is a minimal hierarchy, i.e., having a min-
imum number of nodes. From the programmer’s perspective
this is important to prevent clutter in the generated code. More
importantly, drawing a large number of views slows down the
application. For example, a programmer would not want a
Fig. 4. Example results of the Table I heuristics: Input from Figure 3 (left),
container that contains one child view for each character of
candidate words from OCR (framed, middle), and candidates eliminated by every word displayed by the container.
our heuristics (solid rectangles, right).
However, a competing goal is maximum flexibility of
the inferred view hierarchy. Distinct UI elements should be
From the text content and font information produced by represented by distinct views to allow the generated UI to
OCR for a given word, REMAUI estimates the width (ew ), be well displayed on various combinations of screen size and
height (eh ), and area (e) the candidate word should occupy resolution. Thus, a programmer would, for instance, not want
given the font size and family. Rule 5 uses this information to to represent the four distinct buttons
11 of the Figure 3 bottom-
remove a word if, within bounds, the text area estimated by screen navigation bar as a single image. However, combining
REMAUI does not match the text area reported by OCR. This these four buttons into a single image and a single leaf view
rule removed all four candidate words that are removed in the would reduce the number of views.
right side of Figure 4. To infer a good candidate view hierarchy, REMAUI first
tries to identify all atomic visual elements in the input UI. By
The other rules exclude words OCR is not confident about
atomic we mean a visual element that reasonably should not
(rule 4), have a zero dimension (rule 1), or have an odd shape
be divided further. For example, an icon is atomic but so can
(rule 2). An odd shape likely does not capture an English-
be an entire text paragraph. For each identified atomic visual
language word, as they are long and narrow, vertically or
element REMAUI then computes its approximate view.
horizontally. Finally, rule 6 removes words that only contain
non-ASCII characters or only consist of control characters and To achieve these tasks, REMAUI leverages off-the-shelf
whitespace. computer vision. Figure 5 illustrates REMAUI’s key computer
vision steps on the Figure 3 example input bitmap. First we
The heuristics’ constants are derived through trial and error detect the edges of each image element via Canny’s widely
on a small set of third-party bitmaps. The resulting heuristics used algorithm [10], [51]. But these edges themselves are not
have held up reasonably well on the much larger set of third- good candidates for atomic elements as, for example, each
party bitmaps used in the evaluation (Section VI). character or even minor noise would become its own element.
 Original   Edges  (via  Canny)   Dilated  Edges   Contours   Container  Hierarchy  

Fig. 5. Computer vision processing steps from left to right: Original input bitmap; Edges detected via Canny’s algorithm as black and white; Dilated or
broadened edges to swallow noise and join adjacent elements; Contours of the joined elements; Output: Hierarchy of the contours’ bounding boxes.

TABLE II. H EURISTICS FOR ADDITIONAL ELIMINATIONS OF OCR

To merge close-by elements with each other and with sur- WORDS , BASED ON COMPUTER VISION RESULTS .
rounding noise and to close almost-closed contours REMAUI
dilates its detected edges. REMAUI uses a heuristic to, for # Description
1 Word aligns vertically & overlapped ≥ 70% with ≥ 2 vision boxes that 20   do not
example, allow a word’s characters to merge but keep words overlap each other
separate. REMAUI then computes the dilated edges’ contours. 2 Word aligns horizontally & overlapped ≥ 70% with ≥ 2 vision boxes, distance
between each pair of boxes > each box’s size
Each contour is a candidate atomic element. 3 Word contains a non-leaf vision box
4 Word contains only 1 vision box, box size < 0.2 word size
Figure 5 also illustrates the heuristic nature of this process. 5 Non-overlapped leaf vision box contains only 1 word, word size < 0.2 box size
The last list entry shown in the input screen is cut off by a 6 If leaf vision box’s words are > 50% invalidated, invalidate the rest
horizontal dividing line. Edge detection, dilation, and contour 7 If > 3 words are the same text and size, aligned left, right, top, or bottom, each
has < 0.9 confidence, and are non-dictionary words
thus all merge the last list item with the dividing line, reducing 8 Leaf vision box contains a word, M < 0.4 ∨ (M < 0.7 ∧ m < 0.4) ∨
REMAUI’s precision and recall of atomic visual elements. (M >= 0.7∧m < 0.2), with m = min( bw , h ), M = max( bw
w b
, h )
w b
h h

Finally, REMAUI computes the bounding box of each

candidate atomic element, to approximate the element’s view.
Recall from Section III-A that typically each view is rectangu- OCR lines often blend together into a single line unrelated text
lar and fully contained in its parent. Partially overlapping boxes that just happened to be printed on the same line. For example,
are thus merged into a new bounding box. A fully contained the Figure 3 contact names (left) appear on the same line as
box becomes the child view of the containing box. message dates (right). However they are conceptually separate.
REMAUI thus splits a line if the word-level OCR indicates that
C. Merging (Step 3) the distance between two words exceeds a heuristic threshold
(i.e., their height). Figure 6 shows this process for the Figure 3
In two sub-steps REMAUI merges the results of OCR and example. REMAUI adds the resulting text blocks to the view
computer vision to heuristically combine the best aspects of hierarchy and removes the vision boxes they overlap with.
both and to integrate the OCR-inferred text into the vision-
inferred candidate view hierarchy.
First, REMAUI removes OCR-detected words that conflict
with vision-inferred element bounding boxes. This step ad- OCR lines
dresses common OCR false positives such as classifying part
Text blocks
of an image as a text fragment, classifying bullet points as “o”
or a similar character, and merging lines of text that have too Valid words
little spacing. The resulting OCR-extracted text is not useful
and should instead be exported as an image. Fig. 6. Example: Merging Figure 3 OCR lines with processed OCR words.
Specifically, each OCR word is subjected to the Table II
heuristics. In addition to the OCR word’s width (w) and height
(h), we now also have the computer vision bounding box’s REMAUI aims at extracting text contents with high preci-
width (bw ) and height (bh ). For example, rule (1) checks sion. The employed OCR engine produces better text contents
if an OCR word overlaps with two vision boxes whose y- when treating its input as a single text line. This way the OCR
coordinates do not overlap. This happens if OCR merged two engine does not have to reason about which parts of the input
text lines whereas the vision results kept them separate. are text versus non-text. REMAUI thus invokes OCR on each
text block (in line mode), yielding text that resembles the text
REMAUI further removes OCR words that are not con- in the input relatively closely. Finally, REMAUI groups close-
tained by an OCR line (using the OCR lines from step 1). by text blocks into a container, if the vertical distance between
REMAUI then merges OCR words and lines into text blocks. text blocks is less than either of their heights.
D. Identify Lists (Step 4) installation wizard, which we approximate as up to one
minute.
In this step REMAUI identifies repeated items and sum-
marizes them as collections, for two reasons. First, the final • RQ2: Is a REMAUI-generated UI visually (pixel by
UI definition is more compact and efficient if each repeated pixel) similar to a given third-party input UI conceptual
resource is only represented once. Second, this step allows drawing?
REMAUI to generalize from a few instances to a generic ◦ E2: Given their wide variety, we do not expect RE-
collection. REMAUI can then supply the observed instances MAUI to work well for all applications.
as an example instantiation of the collection. ◦ H2: REMAUI produces a UI that is visually similar
to an input UI conceptual drawing, when running on
REMAUI identifies repeated instances by ordering the non-game mobile applications.
views by their relative location and searching them for identical • RQ3: Is the view hierarchy of a REMAUI-generated UI
sub-trees. A sub-tree consists of a view and a subset of its similar to the view hierarchy of a given third-party input
transitive children. Two sub-trees are identical if each of their application?
child nodes has a peer in the other sub-tree, such that both
◦ E3: Given their wide variety, we do not expect RE-
nodes have the same number of children and the same width,
MAUI to work well for all applications.
height, type (text or image), and matching location within its
◦ H3: REMAUI produces a UI whose view hierarchy is
parent (each within a threshold). Neither text contents nor
similar to the view hierarchy of a given handwritten
image bitmaps have to be identical, as a list item may, for
non-game mobile application.
example, contain the face of a user as in Figure 3.
If identical sub-trees are found, REMAUI creates a bound- VI. E VALUATION
ing box around each of them. Each box contained in such a
bounding box that is not part of the sub-tree belongs to the list To explore our research questions we implemented RE-
item anchored by the sub-tree. However, such an overlapping MAUI for Android. Our prototype generates Android code
box varies across list elements and will be exported as an and resource files that are ready to be compiled and executed.
optional element of the list entry. The properties of these Our prototype supports, among others, Android’s three most
optional elements are determined by overlaying all of them popular layout containers and three most popular widgets
and using the resulting bounding boxes. (Section III-B). For off-the-shelf OCR, REMAUI uses the
open source engine Tesseract [48] and Tesseract’s default
version 3.0.2 English language data trained model. This means
E. Export (Step 5) that REMAUI currently does not use Tesseract’s options
In this step REMAUI exports all results as an Android for training its classifiers. Step 1 uses Tesseract’s fastest
project directory, complete with relevant source code and mode2 with fully automatic page segmentation. For off-the-
resource files. This directory can be compiled with standard shelf computer vision, REMAUI uses the open source engine
Android IDEs. Specifically, REMAUI crops and extracts each OpenCV [6] in its default configuration, without training.
identified image from the input screenshot, only once for A REMAUI-generated application’s aspect ratio (between
repeated images. To provide a reasonable background color, output screen width and height) is the same as the one of its
REMAUI uses as the container’s background the dominant input screenshot. With this aspect ratio, a REMAUI-generated
color of each container after extracting all identified images. application supports many screen resolutions, via Android’s
REMAUI exports all detected text content and format to standard density-independent pixel (dp) scheme. The Android
Android [Link] and [Link] files. REMAUI exports runtime thereby scales a REMAUI-generated application’s dp
layout files to the Android layout directory, for the layout units based on a device’s actual screen density.
shared between list entries and for the main screen. Finally,
REMAUI generates Java code to fill lists with the identified The high-level workflow of our experiment is as follows.
entries at runtime. We first ran a subject Android or iOS application on a
corresponding phone. At some point we took a screenshot and
at the same time captured the current UI hierarchy. We (only)
V. R ESEARCH Q UESTIONS handed the captured screenshot to REMAUI and thus obtained
To evaluate REMAUI, we ask (a) if it is currently feasible a generated application. We then ran the generated application
to integrate REMAUI into a standard mobile application de- on an Android phone and, at the same time, took a screenshot
velopment setup and (b) if REMAUI-generated user interfaces and captured the runtime hierarchy. To clarify, no UI hierarchy
are useful in the sense that the generated UI is similar to the UI information was provided to REMAUI.
an expert developer would produce. We therefore investigate Obtaining the UI hierarchy at runtime required low-level
the following three research questions (RQ), expectations (E), OS access. We thus used a rooted Google Nexus 5 phone
and hypotheses (H). for Android (2 GB RAM, Android 4.4.4) and a jail-broken
iPhone 5 (1 GB RAM, iOS 7.1.2). To obtain the view hierarchy
• RQ1: What is REMAUI’s runtime in a standard develop-
on Android we used Android’s uiautomator3 via the Android
ment setup?
Debug Bridge [41] (adb shell uiautomator dump). For iOS,
◦ E1: Given its expensive OCR and computer vision tech- we used cycript [26] recursively starting from the root view.
niques, we do not expect REMAUI to run interactively.
◦ H1: REMAUI can run on a standard development 2 TESSERACT ONLY, PSM AUTO
machine in a similar amount of time as a software 3 [Link]
For the evaluation REMAUI ran on a 16 GB RAM 2.6 GHz
Core i7 MacBook Pro running OS X 10.10.2.

A. Subjects
Using existing third-party applications to explore our re-
search questions is a good fit for several reasons. (1) First,
it is straightforward to capture a screenshot of a running
application and hand such a screenshot to REMAUI. It is
also straightforward to compare such screenshots pixel by
pixel with REMAUI-generated screenshots (RQ2). (2) More
importantly, having a running application enables inspecting
the application’s UI hierarchy. We can then compare this hi-
erarchy with the corresponding UI hierarchy of the REMAUI- Fig. 7. Runtime of REMAUI’s seven main processing steps on the 488 sub-
generated application (RQ3). jects, shown by group, from left (A) to right (E).
Since our REMAUI prototype is implemented for Android,
our first group of subjects consists of third-party non-game view, which also takes time. On a modern desktop computer
Android applications. To sample popular applications, we total runtime was well within the one minute time frame, with a
downloaded the top-100 free Android applications from the 52 second maximum and an average total runtime of 9 seconds.
Google Play store as of November 9, 2014. From these we
excluded games, as most games do not provide GUIs through C. RQ2: Pixel-by-Pixel Similarity
a view hierarchy but through the native OpenGL library. This
left us with 46 top-100 applications, covering (except games)
all application categories present in the top-100, such as e-
commerce, email, maps, media players, productivity tools,
translation software, and social media. The REMAUI web
site lists name and version of each subject application used
in the evaluation. From each application, we captured the
application’s main screen (in the form it appears after starting
the application). We refer to these subjects as group C.
To broaden our set of subjects, and since many developers
first target iOS, we added iOS applications. We downloaded
Fig. 8. Normalized pixel-by-pixel screenshot similarity between REMAUI
on August 12, 2014 the top 100 free iOS applications from input and generated application on the 488 subjects, shown by group A–E
the Apple App Store. The resulting 66 non-game top-100 from left to right. Higher values are better.
applications cover a range of categories similar to group C.
We took a screenshot of every screen we could reach, yielding Since REMAUI currently removes all standard OS status
302 screenshots (group A). For each application we took and navigation bars from input screenshots, we do the same to
another screenshot showing the main screen with different data the screenshots of REMAUI-generated applications. To ensure
contents, yielding 66 subjects (group B). Since iOS 7 defined that input and generated screenshots have the same dimensions,
a new design language and Google and Apple are major we set the target application screen dimensions to account for
application developers, we included from them 58 screenshots subtracting the OS navigation bar.
of iOS 7 applications outside the top-100 (group D).
We used the open source Photohawk4 library to measure
There may also be a use case of manually drawing designs two widely used picture similarity metrics [51]. Specifically,
and scanning them. Since such third-party drawings are hard to following are the mean absolute error (MAE) and the mean
obtain, we created sketches of 16 screenshots (group E). These squared error (MSE) over a screenshot’s n pixels; ei,j is the
screenshots are our manual renderings of the main screen of delta of one of the three color channels RGB of a given pixel
the alphabetically first 16 of the top 100 iOS applications in in the original vs. the corresponding pixel in the REMAUI-
the Apple app store as of August 12, 2014. generated screenshot.

B. RQ1: REMAUI Runtime

n 3 n 3
Figure 7 shows the runtime of REMAUI’s seven major pro- 1 XX 1 XX 2
MAE = |ei,j | MSE = e
cessing steps, i.e., (1) OCR, (2) computer vision, (3a) merging 3n i=1 j=1 3n i=1 j=1 i,j
OCR text with vision boxes, (3b) splitting text lines, (3c) creat-
ing the view hierarchy, (4) identifying lists, (5) export, and total Figure 8 shows the normalized (to [0, 1]) similarity mea-
runtime. Each step shows the runtimes of groups A–E from left sures for our 488 subjects, arranged by our five groups. The
to right. Not surprisingly, steps 1, 2, and 3b took longest, as results indicate that REMAUI-generated applications achieved
these are the only steps that call REMAUI’s computer vision high average pixel-by-pixel similarity with the respective in-
and OCR engines. The cost of step 3b varied widely, as the puts on both metrics.
number of OCR calls depends on the number of identified text
blocks. Step 5 includes extracting a bitmap for each image 4 [Link]
D. RQ3: UI Hierarchy Similarity pixels t in a text view in the original application and in the
generated application (t0 ).
Achieving high pixel-by-pixel similarity is not sufficient, as
it is trivially achieved by a UI that consists of a single bitmap
(i.e., the input screenshot). So in this section we also evaluate |i ∩ i0 | |t ∩ t0 | |i ∩ i0 | |t ∩ t0 |
pi = pt = ri = rt =
how closely the generated view hierarchy resembles the view 0
|i | |t0 | |i| |t|
hierarchy that produced REMAUI’s input screenshot. |i ∩ i | + |t ∩ t0 |
0
|i ∩ i0 | + |t ∩ t0 |
p= r=
Evaluating the quality of a given UI hierarchy is hard. |i0 | + |t0 | |i| + |t|
First, there are often several different hierarchies of similar
overall quality. Not having the application’s specification, it Since in our setup this experiment required manual steps
is not clear which alternative REMAUI should target. Second, for capturing the UI hierarchies we restricted the scope of
unlike for RQ2, the input application’s UI hierarchy is often the experiment to our core group of Android subjects (group
not an obvious gold standard. Many of the subjects contained C) and the relatively small group of iOS subjects (group B).
redundant intermediate containers that have the same dimen- Figure 10 shows the experiment’s results. We found a moderate
sion as their immediate parent and do not seem to serve a to high structural match (in terms of the leaf nodes) between
clear purpose. Other container hierarchies could have been original and REMAUI-generated UI hierarchies.
refactored into fewer layers to speed up rendering.

Fig. 10. Image, text, and overall UI element precision (p) and recall (r) for
groups B (left) and C (right). Higher values are better.

The low recall in Figure 10 does not fully capture how RE-
Fig. 9. Part of a screenshot of Google Hangout (top), its UI hierarchy MAUI reproduced text or images. On the contrary, REMAUI’s
(middle), and the REMAUI-generated hierarchy (bottom). Each element is pixel-by-pixel similarity was high (Figure 8). We suspect a
annotated at its center with its level in the UI hierarchy, with root=1. Each culprit of low recall was white-space. REMAUI computes tight
number’s color matches the color of its element’s boundary. bounding boxes, but a corresponding original text or image
view may contain much additional white-space and fill its
Figure 9 shows an example of this challenge. The original parent container (as the much larger text boxes in Figure 9).
UI hierarchy and the REMAUI-generated one differ in several
To explore this issue on the example of text, we measured
aspects. For example, REMAUI puts the contact’s name and
the Levenshtein distance (edit distance) [33] of text box
message into two relatively small level-7 text boxes. The orig-
strings between original and generated applications. For the
inal application puts the same strings into much larger level-8
2.4k string pairs of group C, on average, an original text was
text boxes. Similarly, REMAUI groups name and message into
18.8 characters, a generated text 14.6, and the edit distance 4.7.
a level-6 container. The original application groups them with
So on average it took only 4.7 single-character additions,
the date into a level-7 container. This container is nested into a
removals, or substitutions to convert a string in the generated
level-6 container, which is nested into a level-5 container of the
application back to the corresponding string in the original
same dimensions. The latter container thus seems redundant.
application. For the 2.9k group B string pairs, on average, an
Despite these differences, screenshots of the two hierarchies
original text was 14.6 characters, a generated text 15.0, and
are very similar pixel-by-pixel.
the edit distance was only 2.9. These results indicate a higher
In our evaluation we side-step these challenges by compar- text recall than the pixel-based recall of Figure 10.
ing UI hierarchies at the leaf level. While this comparison does The following two trends emerged on manual inspection.
not capture the entire hierarchy, it still captures parts of the First, precision suffered if a subject contains a bitmap that con-
UI’s structure. For example, the boundary of each intermediate tained both text and non-text imagery. This is not surprising,
(container) node is represented by the leaf nodes it contains. as for OCR it is hard to distinguish if a given text is plain text
or belongs to a bitmap of text and other elements. REMAUI
Specifically, for this experiment we analyzed each pixel in
typically decomposed such bitmaps into text and image views.
a REMAUI-generated screenshot. If a pixel belongs to a text
The resulting UI hierarchy should be relatively easy to fix
box in both the original and the generated application, then
manually. A developer would just replace a generated container
we consider the pixel correct. Similarly, the pixel is correct
(containing both text and images) with a single bitmap. Overall
if it belongs to an image view in both the original and in
these incorrectly detected views were small. In Figure 10, their
the generated application. Given these criteria, we can define
average area was less than 0.25% of the input screen area.
precision p and recall r as follows, separately for images, text,
and overall, given the pixels i in an image view in the original The second observation is that low image recall occurred
application and in the generated application (i0 ) as well as the when images overlapped that were of similar color or where
the top image is somewhat transparent. These scenarios are assumes that the pixels that make up a particular widget are
challenging for edge detection. Similarly, text recall was low typically identical across applications. However, this is not
if the text color was similar to the background color. On the true for a mobile application UI. Mobile applications often
flip-side, with high contrast we observed high recall. have their own unique, non-standard identity, style, and theme.
For Prefab to work, all possible widget styles and themes of
VII. R ELATED W ORK millions of current and future mobile applications would need
to be modeled.
The gap between early prototyping and formal layout
definition also exists in the related domain of web site de- PAX [11] heavily relies on the system accessibility API
velopment. A study of 11 designers at 5 companies showed at program runtime. At runtime PAX queries the accessibility
that all designers started with sketching the layout, hierarchy, API to determine the location of text boxes. The accessibility
and flow of web pages with pencil on paper and in graphical API also gives PAX the text contents of the text box. PAX then
design tools such as Photoshop [40]. applies computer vision techniques to determine the location of
A similar design process has been reported for desktop words in the text. If a view does not provide accessibility, PAX
applications. At Apple, user interface sketches were first falls back to a basic template matching approach. PAX thus
created with a fat marker (to prevent premature focus on cannot reverse engineer the UI structure of mobile applications
details) and later scanned [54]. Separate studies of hundreds from their screenshots or application design images alone.
of professionals involved in UI design in various companies Recent work applies the ideas of SILK and DENIM to
indicated heavy use of paper-based sketches [31], [9]. One mobile applications [18], allowing the user to take a picture
of the reasons was that sketching on paper is familiar due to of a paper-and-pencil prototype. The tool allows the user to
designers’ graphic design background. place arbitrary rectangles on the scanned image and connect
Despite much progress in tools for creating user interfaces them with interaction events. This idea is also implemented by
that combine the unique talents of graphic designers and commercial applications such as Pop for iOS. As SILK and
programmers [38], [12], much conceptual user interface design DENIM, this approach is orthogonal to REMAUI.
work is still being done by graphic designers with pencil on
paper and digitally, e.g., in Photoshop. Previous work has VIII. C ONCLUSIONS AND F UTURE W ORK
produced fundamentally different approaches to inferring user
interface code, as it was based on different assumptions. Fol- When developing the UI code of a mobile application, a
lowing are the main changed assumptions for mobile applica- big gap exists between graphic artists’ conceptual drawings
tion UI development and reverse engineering that motivate our and working UI code. Programmers typically bridge this gap
work. (1) First, many UX designers and graphic artists do not manually, by reimplementing the conceptual drawings in code,
construct their conceptual drawings using a predefined visual which is cumbersome and expensive. To bridge this gap,
language we could parse [32], [8], [14], [46]. (2) Second, we introduced the first technique to automatically Reverse
while this was largely true for desktop development, mobile Engineer Mobile Application User Interfaces (REMAUI). On
application screens are not only composed of the platform’s a given input bitmap REMAUI identifies UI elements via
standard UI framework widgets [19], [20]. (3) Finally, we computer vision and OCR techniques. In our experiments on
cannot apply runtime inspection [11], [35] as REMAUI runs 488 screenshots of over 100 popular third-party applications,
early in the development cycle. REMAUI-generated UIs were similar to the originals, both
pixel-by-pixel and in terms of their runtime UI hierarchies.
Specifically, the closest related work is MobiDev [46],
which recognizes instances of a predefined visual language We plan to (1) generalize the export step to additional
of standard UI elements. For example, a crossed-out box is platforms such as iOS and cross-platform JavaScript-based
recognized as a text box. But unlike REMAUI, MobiDev does frameworks. (2) REMAUI currently converts each input screen
not integrate well with a professional mobile application de- to a separate application. We plan to provide a graphical nota-
velopment process. It would require UX designers and graphic tion to allow users to connect several input screens drawings,
artists to change the style of their paper and pencil prototypes, which REMAUI could use to generate a single application
for example, to replace real text with crossed-out boxes. Such with various screens and corresponding screen transitions.
changes may reduce the utility of the prototypes for other (3) We plan to integrate REMAUI with tools that generate
tasks such as eliciting feedback from project stakeholders. mobile application functionality either via keyword-based code
In a traditional reverse engineering setting, MobiDev cannot search [44] or from high-level models [34], [5], [22] or
convert screenshots into UI code. DSLs [29], [4]. (4) We plan to index a screenshot corpus by
SILK and similar systems bridge the gap between pen- running REMAUI on it and storing REMAUI’s intermediate
based GUI sketching and programming of desktop-based results. Exposing this index via a query interface would allow
GUIs [32], [8], [14]. Designers use a mouse or stylus to sketch a user to search for screenshots by their structure and features.
directly in the tool, which recognizes certain stroke gestures as The REMAUI prototype for Android used in the evaluation
UI elements. But these tools do not integrate well with current is freely available at: [Link]
professional development processes as they do not work on
paper-on-pencil scans or screenshots. These tools also do not
recognize handwritten text or arbitrary shapes. ACKNOWLEDGMENTS
UI reverse engineering techniques such as Prefab [19] This material is based upon work supported by the National
depend on a predefined model of UI components. The work Science Foundation under Grant No. 1117369.
 R EFERENCES [20] M. Dixon, D. Leventhal, and J. Fogarty, “Content and hierarchy in pixel-
based methods for reverse engineering interface structure,” in Proc.
[1] A. Antonacopoulos, S. Pletschacher, D. Bridson, and C. Papadopoulos, ACM SIGCHI Conference on Human Factors in Computing Systems
“ICDAR 2009 page segmentation competition,” in Proc. 10th Inter- (CHI). ACM, May 2011, pp. 969–978.
national Conference on Document Analysis and Recognition (ICDAR).
[21] M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin, “Dynamically
IEEE, Jul. 2009, pp. 1370–1374.
discovering likely program invariants to support program evolution,”
[2] Apple Inc., “View programming guide for iOS,” IEEE Transactions on Software Engineering (TSE), vol. 27, no. 2, pp.
[Link] 99–123, Feb. 2001.
conceptual/viewpg iphoneos/ViewPG [Link], Oct. 2013,
[22] J. Falb, T. Röck, and E. Arnautovic, “Using communicative acts in
accessed May 2015.
interaction design specifications for automated synthesis of user inter-
[3] P. Bao, J. S. Pierce, S. Whittaker, and S. Zhai, “Smart phone use by non- faces,” in Proc. 21st ACM/IEEE International Conference on Automated
mobile business users,” in Proc. 13th Conference on Human-Computer Software Engineering (ASE). IEEE, 2006, pp. 261–264.
Interaction with Mobile Devices and Services (Mobile HCI). ACM,
[23] L. Fortunati and S. Taipale, “The advanced use of mobile phones in five
Aug. 2011, pp. 445–454.
European countries,” The British Journal of Sociology, vol. 65, no. 2,
[4] S. Barnett, R. Vasa, and J. Grundy, “Bootstrapping mobile app develop- pp. 317–337, Jun. 2014.
ment,” in Proc. 37th ACM/IEEE International Conference on Software
[24] M. Gargenta and M. Nakamura, Learning Android: Develop Mobile
Engineering (ICSE). IEEE, May 2015.
Apps Using Java and Eclipse, 2nd ed. O’Reilly, Jan. 2014.
[5] M. Book and V. Gruhn, “Modeling web-based dialog flows for auto-
[25] Z. Hussain, M. Lechner, H. Milchrahm, S. Shahzad, W. Slany,
matic dialog control,” in Proc. 19th IEEE International Conference on
M. Umgeher, T. Vlk, and P. Wolkerstorfer, “User interface design for
Automated Software Engineering (ASE). IEEE, Sep. 2004, pp. 100–
a mobile multimedia application: An iterative approach,” in Proc. 1st
109.
International Conference on Advances in Computer-Human Interaction
[6] G. Bradski and A. Kaehler, Learning OpenCV: Computer Vision with (ACHI). IEEE, Feb. 2008, pp. 189–194.
the OpenCV Library, 1st ed. O’Reilly, Oct. 2008.
[26] Jay Freeman, “Cycript,” [Link] 2014, accessed May
[7] Y. Brun and M. D. Ernst, “Finding latent code errors via machine learn- 2015.
ing over program executions,” in Proc. 26th ACM/IEEE International
Conference on Software Engineering (ICSE). IEEE, May 2004, pp. [27] D. Karatzas, F. Shafait, S. Uchida, M. Iwamura, L. G. i Bigorda,
480–490. S. R. Mestre, J. Mas, D. F. Mota, J. Almazàn, and L. de las Heras,
“ICDAR 2013 robust reading competition,” in Proc. 12th International
[8] A. Caetano, N. Goulart, M. Fonseca, and J. Jorge, “JavaSketchIt: Conference on Document Analysis and Recognition (ICDAR). IEEE,
Issues in sketching the look of user interfaces,” in Proc. AAAI Spring Aug. 2013, pp. 1484–1493.
Symposium on Sketch Understanding. AAAI, Mar. 2002, pp. 9–14.
[28] A. K. Karlson, B. Meyers, A. Jacobs, P. Johns, and S. K. Kane,
[9] P. F. Campos and N. J. Nunes, “Practitioner tools and workstyles for “Working overtime: Patterns of smartphone and PC usage in the day
user-interface design.” IEEE Software, vol. 24, no. 1, pp. 73–80, Jan. of an information worker,” in Proc. 7th International Conference on
2007. Pervasive Computing (Pervasive). Springer, May 2009, pp. 398–405.
[10] J. Canny, “A computational approach to edge detection,” IEEE Trans- [29] A. Khambati, J. C. Grundy, J. Warren, and J. G. Hosking, “Model-
actions on Pattern Analysis and Machine Intelligence, vol. 8, no. 6, pp. driven development of mobile personal health care applications,” in
679–698, Nov. 1986. Proc. 23rd ACM/IEEE International Conference on Automated Software
[11] T.-H. Chang, T. Yeh, and R. C. Miller, “Associating the visual repre- Engineering (ASE). IEEE, Sep. 2008, pp. 467–470.
sentation of user interfaces with their internal structures and metadata,” [30] K. Kuusinen and T. Mikkonen, “Designing user experience for mobile
in Proc. 24th Annual ACM Symposium on User Interface Software and apps: Long-term product owner perspective,” in Proc. 20th Asia-Pacific
Technology (UIST). ACM, Oct. 2011, pp. 245–256. Software Engineering Conference (APSEC). IEEE, Dec. 2013, pp.
[12] S. Chatty, S. Sire, J.-L. Vinot, P. Lecoanet, A. Lemort, and C. P. 535–540.
Mertz, “Revisiting visual interface programming: creating GUI tools for [31] J. A. Landay and B. A. Myers, “Interactive sketching for the early
designers and programmers,” in Proc. 17th Annual ACM Symposium on stages of user interface design,” in Proc. ACM SIGCHI Conference on
User Interface Software and Technology (UIST). ACM, Oct. 2004, pp. Human Factors in Computing Systems (CHI). ACM, May 1995, pp.
267–276. 43–50.
[13] E. J. Chikofsky and J. H. C. II, “Reverse engineering and design [32] ——, “Sketching interfaces: Toward more human interface design,”
recovery: A taxonomy,” IEEE Software, vol. 7, no. 1, pp. 13–17, Jan. IEEE Computer, vol. 34, no. 3, pp. 56–64, Mar. 2001.
1990.
[33] V. I. Levenshtein, “Binary codes capable of correcting deletions, inser-
[14] A. Coyette, S. Kieffer, and J. Vanderdonckt, “Multi-fidelity prototyping tions and reversals,” Soviet Physics Doklady, vol. 10, no. 8, pp. 707–710,
of user interfaces.” in Proc. 11th IFIP TC 13 International Conference Feb. 1966.
on Human-Computer Interaction (INTERACT). Springer, Sep. 2007,
pp. 150–164. [34] A. Martı́nez, H. Estrada, J. Sánchez, and O. Pastor, “From early
requirements to user interface prototyping: A methodological approach,”
[15] C. Csallner and Y. Smaragdakis, “Dynamically discovering likely in- in Proc. 17th IEEE International Conference on Automated Software
terface invariants,” in Proc. 28th ACM/IEEE International Conference Engineering (ASE). IEEE, Sep. 2002, pp. 257–260.
on Software Engineering (ICSE), Emerging Results Track. ACM, May
2006, pp. 861–864. [35] X. Meng, S. Zhao, Y. Huang, Z. Zhang, J. Eagan, and R. Subramanian,
“WADE: simplified GUI add-on development for third-party software,”
[16] C. Csallner, N. Tillmann, and Y. Smaragdakis, “DySy: Dynamic in Proc. ACM SIGCHI Conference on Human Factors in Computing
symbolic execution for invariant inference,” in Proc. 30th ACM/IEEE Systems (CHI). ACM, Apr. 2014, pp. 2221–2230.
International Conference on Software Engineering (ICSE). ACM, May
2008, pp. 281–290. [36] B. A. Myers, “Challenges of HCI design and implementation,” Inter-
actions, vol. 1, no. 1, pp. 73–83, Jan. 1994.
[17] T. S. da Silva, A. Martin, F. Maurer, and M. S. Silveira, “User-
centered design and agile methods: A systematic review,” in Proc. Agile [37] ——, “Graphical user interface programming,” in Computer Science
Conference (AGILE). IEEE, Aug. 2011, pp. 77–86. Handbook, 2nd ed., A. B. Tucker, Ed. CRC Press, May 2012.
[18] M. de Sà, L. Carriço, L. Duarte, and T. Reis, “A mixed-fidelity [38] B. A. Myers, S. E. Hudson, and R. F. Pausch, “Past, present, and future
prototyping tool for mobile devices,” in Proc. Working Conference on of user interface software tools,” ACM Transactions on Computer-
Advanced Visual Interfaces (AVI). ACM, May 2008, pp. 225–232. Human Interaction (TOCHI), vol. 7, no. 1, pp. 3–28, Mar. 2000.
[19] M. Dixon and J. Fogarty, “Prefab: Implementing advanced behaviors [39] V. Nahavandipoor, iOS 7 Programming Cookbook, 1st ed. O’Reilly,
using pixel-based reverse engineering of interface structure,” in Proc. Nov. 2013.
ACM SIGCHI Conference on Human Factors in Computing Systems [40] M. W. Newman and J. A. Landay, “Sitemaps, storyboards, and speci-
(CHI). ACM, Apr. 2010, pp. 1525–1534. fications: A sketch of Web site design practice as manifested through
 artifacts,” EECS Department, University of California, Berkeley, Tech. 9th International Conference on Document Analysis and Recognition
Rep. UCB/CSD-99-1062, 1999. (ICDAR). IEEE, Sep. 2007, pp. 629–633.
[41] T. A. Nguyen, C. Csallner, and N. Tillmann, “GROPG: A graphical on- [49] ——, “An overview of the tesseract ocr engine,” in ICDAR ’07:
phone debugger,” in Proc. 35th ACM/IEEE International Conference on Proceedings of the Ninth International Conference on Document
Software Engineering (ICSE), New Ideas and Emerging Results (NIER) Analysis and Recognition. Washington, DC, USA: IEEE Computer
track. IEEE, May 2013, pp. 1189–1192. Society, 2007, pp. 629–633. [Online]. Available: [Link]
[42] G. Nudelman, Android Design Patterns: Interaction Design Solutions research/pubs/archive/[Link]
for Developers. Wiley, Mar. 2013. [50] R. W. Smith, “Hybrid page layout analysis via tab-stop detection.”
[43] R. Plamondon and S. Srihari, “Online and off-line handwriting recogni- in Proc. 10th International Conference on Document Analysis and
tion: A comprehensive survey,” IEEE Transactions on Pattern Analysis Recognition (ICDAR). IEEE, Jul. 2009, pp. 241–245.
and Machine Intelligence, vol. 22, no. 1, pp. 63–84, Jan. 2000. [51] R. Szeliski, Computer Vision: Algorithms and Applications. Springer,
[44] S. P. Reiss, “Seeking the user interface,” in Proc. 29th ACM/IEEE Nov. 2010.
International Conference on Automated Software Engineering (ASE).
[52] The Nielsen Company, “The mobile consumer: A global snapshot,”
ACM, Sep. 2014, pp. 103–114.
[Link]
[45] S. E. Salamati Taba, I. Keivanloo, Y. Zou, J. Ng, and T. Ng, “An [Link], Feb. 2013.
exploratory study on the relation between user interface complexity
and the perceived quality of Android applications,” in Proc. 14th [53] Ø. D. Trier, A. K. Jain, and T. Taxt, “Feature extraction methods for
International Conference on Web Engineering (ICWE). Springer, Jul. character recognition—a survey,” Pattern Recognition, vol. 29, no. 4,
2014. pp. 641–662, Apr. 1996.
[46] J. Seifert, B. Pfleging, E. del Carmen Valderrama Bahamóndez, M. Her- [54] Y. Y. Wong, “Rough and ready prototypes: Lessons from graphic
mes, E. Rukzio, and A. Schmidt, “Mobidev: A tool for creating apps design,” in Proc. ACM SIGCHI Conference on Human Factors in
on mobile phones,” in Proc. 13th Conference on Human-Computer Computing Systems (CHI), Posters and Short Talks. ACM, 1992, pp.
Interaction with Mobile Devices and Services (Mobile HCI). ACM, 83–84.
Aug. 2011, pp. 109–112. [55] C. Zeidler, C. Lutteroth, W. Stürzlinger, and G. Weber, “The Auckland
[47] A. S. Shirazi, N. Henze, A. Schmidt, R. Goldberg, B. Schmidt, and layout editor: An improved GUI layout specification process,” in
H. Schmauder, “Insights into layout patterns of mobile user interfaces Proc. 26th Annual ACM Symposium on User Interface Software and
by an automatic analysis of Android apps,” in Proc. ACM SIGCHI Technology (UIST). ACM, Oct. 2013, pp. 343–352.
Symposium on Engineering Interactive Computing Systems (EICS). [56] ——, “Evaluating direct manipulation operations for constraint-based
ACM, Jun. 2013, pp. 275–284. layout,” in Proc. 14th IFIP TC 13 International Conference on Human-
[48] R. Smith, “An overview of the Tesseract OCR engine,” in Proc. Computer Interaction (INTERACT). Springer, Sep. 2013, pp. 513–529.