Automatically Assessing Code Understandability:

How Far Are We?

Simone Scalabrino∗ , Gabriele Bavota† , Christopher Vendome‡ ,
Mario Linares-Vásquez§ , Denys Poshyvanyk‡ and Rocco Oliveto∗
∗ University of Molise, Italy — † Università della Svizzera italiana (USI), Switzerland
‡ TheCollege of William and Mary, USA — § Universidad de los Andes, Colombia

Abstract—Program understanding plays a pivotal role in soft- their correlation with the readability perceived by developers.
ware maintenance and evolution: a deep understanding of code This typically means that the developers participating in the
is the stepping stone for most software-related activities, such as evaluation are just asked to read the code and assess its
bug fixing or testing. Being able to measure the understandability
of a piece of code might help in estimating the effort required for readability on a given scale (e.g., from 1—low to 5—high
a maintenance activity, in comparing the quality of alternative readability). However, the perceived readability is something
implementations, or even in predicting bugs. Unfortunately, different from the actual understandability of the code; a
there are no existing metrics specifically designed to assess the developer could find a piece of code readable, but still difficult
understandability of a given code snippet. to understand, for example, due to unknown APIs used.
In this paper, we perform a first step in this direction, by
studying the extent to which several types of metrics computed
Let us consider the code fragment listed below:
on code, documentation, and developers correlate with code AsyncHttpClient client = new AsyncHttpClient () ;
String cookies = CookieManager . getInstance () . getCookie ( url );
understandability. To perform such an investigation we ran a Log . e ( TAG , cookies ) ;
study with 46 participants who were asked to understand eight client . addHeader ( SM . COOKIE , cookies ) ;
code snippets each. We collected a total of 324 evaluations aiming
All readability metrics/models existing in the literature [10],
at assessing the perceived understandability, the actual level of
understanding, and the time needed to understand a code snippet. [11], [12], [13], [14] would consider this snippet of code as
Our results demonstrate that none of the (existing and new) highly readable, since it is very short, it has a low complexity,
metrics we considered is able to capture code understandability, and uses meaningful and intuitive identifiers. Nevertheless, this
not even the ones assumed to assess quality attributes strongly snippet of code is not necessarily easy to understand for any
related with it, such as code readability and complexity. given developer, because the used APIs could be unknown
Index Terms—Software metrics, Code understandability, Em-
pirical study, Negative result to her and even poorly documented. In other words, even if
a developer can understand that getCookie(url) returns a
I. I NTRODUCTION cookie, she may not understand all the consequences of this
call if she does not know the specifications behind that API.
Code understanding is an important foundation to any code- While we have possible proxies for code understandabil-
related activity. Developing new features, fixing bugs, or ity, such as code complexity and readability, we (i) do not
refactoring code requires a deep understanding of the involved know whether these proxies actually correlate with the effort
code components and of their relationships. Unsurprisingly, required to understand a piece of code, and (ii) do not have a
recent studies showed that developers spend most of their time metric able to provide an estimation for code understandability.
(∼70%) understanding code [1]. Previous attempts to define a code understandability model
While the importance of code understandability is undis- [4], [15], [6] have not been empirically evaluated, consider
puted for maintenance-related activities [2], [3], [4], [5], [6], understandability as a factor in a quality model [2], [16], or
[7], there is still a lack of metrics to objectively assess measure understandability at the level of a whole system [17].
the understandability of a given piece of code. Indeed, our Having a metric to estimate the effort required to understand
knowledge of factors affecting (positively or negatively) code a piece of code would have a strong impact on several software
understandability is basically tied to common beliefs or is engineering tasks. For example, such a metric would be useful
focused on the cognitive process adopted when understanding to (i) improve the estimation of the time needed to fix a
code [8], [9]. For example, we commonly assume that code bug (the lower the understandability, the higher the time to
complexity has a direct impact on developers’ ability to fix the bug); (ii) create search-based refactoring recommender
understand the code, but we do not have strong empirical systems using code understandability as a fitness function; or
evidence supporting such a belief yet. (iii) assess the quality of code changes during code reviews.
Another aspect possibly related to the understandability of In this paper, we make the first step towards the definition
a piece of code is represented by its readability. In the last of a metric able to capture the understandability of a given
years, researchers have proposed several metrics and models piece of code by investigating to what extent a number
for assessing code readability [10], [11], [12], [13], [14]. of different (existing and new) metrics correlate with code
These metrics and models are generally evaluated by assessing understandability.
 We consider three types of metrics: A. Code Readability
1) Code-related metrics. We consider 105 metrics character- Identifiers and comments play a crucial role in program
izing the code to be understood: classic code metrics such comprehension, since developers express domain knowledge
as LOC and cyclomatic complexity, as well as metrics through the names that they assign to the code entities at
used to assess the readability of a code component, such different levels (i.e., packages, classes, methods, variables)
as text coherence [14] and the code indentation [10]; [18], [19], [20], [21], [22]. Thus, source code lexicon impacts
2) Documentation-related metrics. The availability of doc- the psychological complexity of a program [14], [23]. Another
umentation can clearly impact the understandability of aspect that also contributes to the readability (and potentially
a given piece of code. For this reason, we consider 11 understandability) of source code are structural aspects such as
documentation-related metrics (e.g., the availability of indentation, code entities length [10], [11], and visual/spatial
external documentation for the given code). Nine of them aspects such as syntax highlighting, code formatting, and
are introduced in this paper. visual areas covered by code entities [12].
3) Developer-related metrics. We consider five metrics re- All the aforementioned aspects have been used as features
lated to the developer’s experience. While developer- in binary classifiers able to predict the readability of code
related metrics could not be used to define an objective snippets [10], [11], [12], [13], [14]. In the model by Buse and
measure for code understandability, they are still useful Weimer [10], source code structural aspects (e.g., number of
in our study to assess to what extent the developer expe- branches, loops, operators, blank lines, comments) represent
rience and background play a role during comprehension. the underlying features in the classifier. The model was trained
We conducted a study involving 46 participants to assess a-priori on 100 small snippets; the snippets were tagged man-
the correlation of the aforementioned 121 (105 code-related ually as readable or non-readable by 120 human annotators.
+ 11 documentation-related + 5 developer-related) metrics The reported results provide evidence that readability can be
with the understandability of code snippets. Participants were estimated automatically.
required to understand eight code snippets each, for a total Posnett et al. [11] proposed a model based on a reduced set
of 324 data points. We adopted the following experimental of the features introduced by Buse and Weimer. An empirical
design to consider both the perceived and the actual code evaluation conducted on the same dataset used by Buse and
understandability. We asked participants to carefully read and Weimer [10] indicated that the model by Posnett et al. is more
to fully understand each snippet. Participants could, at any accurate than the one by Buse and Weimer.
moment, select the option “I understood the snippet” or “I Dorn introduced a readability model, which relies on a
cannot understand the snippet”. This provided us with a first larger set of features grouped in four categories: visual, spatial,
classification of snippets that are perceived by the participants alignment, and linguistic [12]. This larger set of features
as either “understandable” or “not understandable”. Also, since highlights the fact that structural aspects are not the only
we monitored the time spent before selecting one of the two ones that should be considered for code readability; aspects
options, we can also measure the understandability effort (i.e., representing and modeling how the code is read on the screen,
the time required to understand a code snippet). Finally, when such as syntax highlighting, variable naming standards, and
the participants clicked on “I understood the snippet”, we operators alignment, should be also considered. Dorn trained
verified their actual level of understanding by asking questions and validated the model on a new dataset, including programs
about the code snippet. The collected dataset, as opposed to in Java, Python and CUDA, for a total of 360 snippets. Such
the ones used for evaluating code readability metrics (see a model achieved a higher accuracy as compared to the one
[10], [14]), provides indications not only about the perceived by Buse and Weimer.
understandability, but also about the actual understandability. Scalabrino et al. [14] proposed and evaluated a set of
Our extensive statistical analysis provided us with what features based entirely on source code lexicon analysis (e.g.,
can be defined as a negative empirical result: none of the consistency between source code and comments, specificity of
considered metrics exhibit a significant correlation with the the identifiers, textual coherence, comments readability). The
understandability of code snippets. This holds for both the model was evaluated on the two datasets previously introduced
perceived and the actual understandability. This result was by Buse and Weimer [10] and Dorn [12] and on a new
quite surprising for us, especially considering that several dataset, composed by 200 Java snippets, manually evaluated
metrics (e.g., code complexity and readability) are supposed by nine developers. The results indicated that combining the
to influence the understandability of code. features (i.e., structural+textual) improves the accuracy of code
readability models.
II. BACKGROUND & R ELATED W ORK
In this section, we describe metrics and models that have B. Software/Code Understandability
been proposed to measure code readability. All these metrics While readable code might directly impact program compre-
have been included in our study. Afterwards, we briefly hension, code readability metrics are not sufficient to measure
describe related work presenting metrics for measuring un- to what extent the code allows developers to understand its
derstandability (at system level) as a single quality attribute purpose, relationships between code entities, and the latent
and as part of a quality model. semantics at the low-level (e.g., statements, beacons, motifs)
and high-level structures (e.g., packages, classes). Program Similarly, Chen et al. [7] investigated the COCOMO II
understanding is a non-trivial mental process that requires Software Understandability factors by conducting a study with
building high-level abstractions from code statements or vi- six graduate students asked to accomplish 44 maintenance
sualizations/models [9], [4]. There have been several metrics tasks, and found that higher quality structure, higher quality
designed to evaluate software understandability by focusing organization, and more self-descriptive code were all corre-
on complexity as well as source-level metrics. lated with less effort spent on the tasks, which leads to high
Lin et al. [4] proposed a model for assessing understand- maintainability.
ability by building an understandability matrix from fuzzy Bansiya and Davis [26] proposed a model where metrics
maximum membership estimation for population of fog index, are related to several quality attributes, including understand-
comment ratio, the number of components, CFS, Halstead ability. In terms of understandability, the model considers
Complexity, and DMSP. The authors then used PCA and factor encapsulation and cohesion to have positive influences, while
analysis to get the weights for the column vectors, which can abstraction, coupling, polymorphism, complexity, and design
be multiplied by the matrix to get the Synthesis Vector of size have a negative influence. The authors validated the model
Understandability. Finally, the understandability is calculated by analyzing several versions of two applications and found
by using the fuzzy integral. The authors did not empirically that understandability decreases as a system evolves with many
evaluate the proposed metric. new features. Additionally, 13 evaluators analyzed 14 versions
Misra and Akman [24] performed a comparative study of a project and the authors found a correlation between
between existing cognitive complexity measures and their the evaluators’ overall assessment of quality and the models
proposed measure: cognitive weight complexity measure assessment for 11 out of 13 evaluators.
(CWCM), which assigns weights to software components by It is worth noting that we do not consider the above
analyzing their control structures. The authors performed a discussed understandability metrics [4], [24], [3], [6], [17],
theoretical validation of these metrics based on the properties [16], [26] in our study since they are defined at system-
proposed by Weyuker [25]. They found that only one metric, level (i.e., they provide an overall indication of the system
Cognitive Information Complexity Measure (CICM), satisfied understandability), while we are interested in studying whether
all nine properties, while the others satisfied seven of the nine. it is possible to measure the understandability of a given code
snippet, as already done in the literature for code readability.
Thongmak et al. [3] considered aspect-oriented software Instead, we included in our study the metrics used by Kasto
dependence graphs to assess understandability of aspect- and Whalley [27] to study the understandability of code
oriented software, while Srinivasulu et al. [6] used rough sets snippets in an educational context. Specifically, Kasto and
and rough entropy (to filter outliers) when considering the Whalley analyzed the performance of 93 students in their
following metrics: fog index, comment ration, the number of final examination for the Java programming course and they
components, CFS, Halstead Complexity, and DMSC. These correlated their results with five metrics.
metrics are computed at system level for nine projects, and Several studies have explored software understandability
subsequently the rough entropy outlier factor was calculated and program comprehension with either students or practition-
for the metrics to identify the outliers, which correspond to ers. Shima et al. considered the understandability of a software
either highly understandable or not understandable software system by assessing the probability that a system can be
based on the metric values. correctly reconstructed from its components [28]. The authors
Capiluppi et al. [17] proposed a measure of understand- asked eight students to reconstruct a system and the results
ability that can be evaluated in an automated manner. The suggest that faults tend to occur in hard to understand files or
proposed measure considers: (i) the percentage of micro- very simple files. Roehm et al. performed an observational
modules (i.e., the numbers of files) that are within the macro- study with 28 developers to identify the steps developers
modules (i.e., the directories), and (ii) the relative size of the perform when understanding software and the artifacts they
micro-modules. The authors calculated the proposed measure investigate [5]. The authors found that developers are more in-
on the history of 19 open source projects, finding that un- clined towards relying upon source code as well as discussing
derstandability typically increased during the life-cycle of the with colleagues over utilizing the documentation. The authors
systems. Yet, no evaluation is provided for such a measure. also identified some behaviors that improve comprehension,
Understandability has also been a factor in quality models such as consistent naming conventions or meaningful names.
to assess software maintainability. Aggarwal et al. [2] inves- Understandability has been mostly analyzed from the per-
tigated the maintainability of software and proposed a fuzzy spective of (i) the quality attribute at the software level,
model, which is composed of three parts: (i) readability of i.e., understandability as the “The capability of the software
code, (ii) documentation quality, and (iii) understandability product to enable the user to understand whether the software
of the software. To quantify understandability, the authors is suitable, and how it can be used for particular tasks and
utilize a prior work that defines language of software as conditions of use" [29]; and (ii) the theories, challenges, and
the symbols used, excluding reserved words. The authors models for program understanding at cognitive levels [9], [8].
constructed rules based on the ranges of the three factors to However, as of today, we still lack models for assessing code
determine maintainability. understandability at snippet-level, similarly to code readability.
The only work we found that relates to a code understandabil- A. Code-related metrics
ity model is based on complexity and size source code level Most of the metrics considered in our study assess source
metrics [4], [15]. code properties. We include the five metrics used by Kasto
III. C ANDIDATE P REDICTORS FOR C ODE and Whalley [27]: cyclomatic complexity [30], which estimates
U NDERSTANDABILITY the number of linear independent paths of the snippet average
number of nested blocks, which measures the average code-
Understandability is a multifaceted property of source code
block nesting in the snippet, number of parameters, number of
and, as well as readability, is subjective in nature. In readabil-
statements and number of operands, i.e., number of identifiers.
ity, the subjectivity is represented by personal taste and habits,
We also include in this category all the code-related read-
while in understandability it lies in the previous knowledge of
ability metrics defined in the literature [10], [11], [12], [14].
a developer and in her mental models [8]. Consider a method
These include the ones by Buse and Weimer [10], assessing
of an Android activity in a mobile app; its understandability
properties for a single line of code (e.g., number of identifiers
might be high for an Android developer, while it could be
or line length) and then aggregated (with the maximum and/or
low for a Java developer with no experience in Android. In
the average) to work at the level of “code snippet”.
this section, we briefly discuss the 121 metrics considered
Lines of code (LOC), token entropy and Halstead’s volume
in our study aimed at assessing their ability to capture the
are used by Posnett et al. [11] in the context of readability
understandability of a given piece of code. Table I shows the
prediction. Dorn [12] presents a variation to the basic metrics
complete list of metrics: rows contain the basic metrics and
introduced by Buse and Weimer [10], measuring the band-
columns indicate how the metrics are aggregated (e.g., the
width of the Discrete Fourier Transform (DFT) of the metrics,
Identifiers length for a given code snippet is computed, as
the absolute and the relative area of characters belonging
suggested by previous work [10], as the average and as the
to different token categories (e.g., identifiers, keywords or
maximum length of the identifiers used in the snippet). We
comments), the alignment of characters through different lines,
report in boldface the new metrics introduced in this study. In
and the number of identifiers containing words belonging to an
the following subsections, we discuss the considered metrics
English dictionary. Note that the area-related metrics defined
grouped by their type.
by Dorn are computed both in an absolute way (e.g., total area
of comments) and in a relative way (e.g., area of comments
TABLE I
C ANDIDATE PREDICTORS FOR CODE UNDERSTANDABILITY. divided by area of strings). These variants are not reported
in Table I due to space constraints, but are considered in our
Metric Non-aggreg. Min Avg Max DFT Visual Area study and listed in our replication package [31].
Cyclomatic comp. [27]
#nested blocks [27]
Scalabrino et al. [14] define Narrow Meaning Identifiers
#parameters
#statements
[27]
[27]
(NMI), Number of Meanings (NM), Identifiers Terms In Dic-
#assignments [10] [12] tionary (ITID) and Textual Coherence (TC) to capture the
#blank lines [10]
#characters [10] readability of a code snippet. Such metrics are computed
#commas [10] [12]
#comments [10] [12] [12] [12] line-by-line (ITID), identifier-by-identifier (NMI and NM)
#comparisons [10] [12] or block-by-block (TC); the authors aggregate the measures
#conditionals [10] [12]
#identifiers [27] [10] [10] [12] [12] [12] using minimum, average and maximum, in order to have a
#keywords [10] [10] [12] [12] [12]
#literals [12] [12] single measure for the snippet. We also use code readability,
#loops [10] [12]
#numbers [10] [10] [12] [12] [12] as defined by Scalabrino et al. [14], as a separate metric,
#operators [10] [12] [12] [12]
combining together the previously listed metrics. We followed
Code

#parenthesis [10] [12]

#periods
#spaces
[10]
[10]
[12]
[12]
the steps described by Scalabrino et al. to define the readability
#strings [12] [12] model by using a logistic classifier that we train on the 420
#words [10]
Indentation length [10] [10] [12] Java snippets available in the literature [10], [12], [14].
Identifiers length [10] [10]
Line length [10] [10] [12] We also introduce a new code-related metric, the Invoked
#aligned blocks
Ext. of alig. blocks
[12]
[12]
Methods Signature Quality (IMSQ), which measures the qual-
Entropy [11] ity of the signature of the internal methods invoked by a given
LOC [11]
Volume [11] code snippet s (i.e., methods belonging to the same system of
NMI [14] [14] [14]
NM [14] [14] s) in terms of readability and representativeness. We define the
ITID [14] [14][12]
TC [14] [14] [14]
Method Signature Quality (MSQ) of an invoked method m as:
Readability [14]
IMSQ [new] [new] [new] 1 X
MSQ(m) = IQ(id)
CR [14] |IS (m)|
Docs

CIC [14] [14] id∈IS (m)

CICsyn [14] [14]
MIDQ
AEDQ
[new]
[new]
[new]
[new]
[new]
[new]
where IS(m) is the set of identifiers used in the m’s signature
EAP [new] [new] [new]
(i.e., method name and parameters) and IQ(id) is defined as:
Devs

PEgen [new] (
1
PEspec [new] (Rd (id) + Rp(id)), id is a method name
IQ(id) = 2
Rd (id ), id is a parameter name
 IQ(id) captures the quality of an identifier in terms of number of times each word from the identifier (expanded
its readability (Rd) and its representativeness (Rp). The idea or not) is contained in the dictionary extracted from the
behind the readability is that an identifier should be composed method body. We consider only names and verbs from the
of a (possibly small) set of meaningful words. To measure Rd identifiers, ignoring other parts of speech such as conjunctions,
for an identifier (id), we (i) split id into the words composing since they do not carry semantic information. Following the
it, (ii) expand each word to bring it in its original form printOnStdOut example, we check whether the method body
(e.g., ptr → pointer), (iii) create a new identifier idexp contains the words print, standard, std, output, and out.
composed by the expanded words separated by a “_”, and (iv) We measure the representativeness as the ratio between the
measure the Levenshtein distance between id and idexp . The number of words from the identifier (i.e., method name)
Levenshtein distance between two strings a and b measures contained in the method body, and the total number of words
the minimum number of single-character changes needed to in the identifier. If all the words from the identifier are used in
transform a into b. The conjecture behind IQ(id) is that m’s body, we assume that the method name is representative of
the higher the Levenshtein distance between id and idexp , m and thus, should ease the understanding of methods invoking
the higher the mental effort required for the developer to m. If, instead, words are not found in the method body, this
understand the meaning of the identifier by mentally splitting could hinder the understandability of the methods invoking m.
and expanding it during program comprehension. Note also In our study, we consider the minimum, the average, and the
that we separate the expanded terms in idexp by using “_” maximum values of the MSQ metric for a given code snippet
in order to penalize, by increasing the Levenshtein distance, (e.g., the average MSQ of all methods invoked in the code
identifiers composed by several words. For example, the snippet).
identifier printOnStdOut is first split into print, on, std,
out; then, each word is expanded, which has no effect on B. Documentation-related metrics
the first two words, but expands std into standard and out Scalabrino et al. [14] introduced three metrics to capture the
into output. Therefore, printOnStdOut is transformed in quality of the internal documentation of a snippet: Comments
print_on_standard_output. Readability (CR) measures the readability of the comments in
To have Rd(id) defined in [0, 1], we normalize the Leven- a snippet using the Flesch reading-ease test [32]; Comments
shtein distance (L) between id and idexp as follows: and Identifiers Consistency (CIC) measures the consistency
between comments and code; and CICsyn , a variant of CIC,
L(id , id exp )
Rd (id ) = 1 − which takes synonyms into account.
max(|id |, |id exp |)
We also introduce two new metrics aimed at capturing the
where max(|id |, |id exp |) represents the longest identifier quality of both the internal (MIDQ) and external (AEDQ)
among the two. When the distance equals zero, the readability documentation available for code components used in a given
of the identifier equals one, indicating no need for expansion/s- snippet. The Methods Internal Documentation Quality (MIDQ)
plitting (i.e., id is composed by a single expanded word). for a snippet s acts as a proxy for the internal documentation
Note that in the implementation of Rd(id), we used a semi- (i.e., Javadoc) available for the internal methods (the ones
automatic approach to split/expand identifiers. We first used belonging to the same project as s) invoked in s. Given m
a naive automatic splitting technique, based on camel case an internal invoked method, we compute MIDQ(m) using a
and underscores; then, we automatically checked the presence variation of the approach proposed by Schreck et al. [33]:
of each resulting word in an English dictionary. If the word
was not found, we manually expanded/further split the specific 1
MIDQ(m) = (DIR(m) + readability D (m))
word. For example, for the word “cmdline” there would not be 2
automatic split. Since the word “cmdline” does not exist in the where DIR(m) is the Documented Items Ratio computed as the
dictionary, we manually convert it to “command” and “line”. number of documented items in m divided by the number of
We save all the manual substitutions in order to minimize documentable items in m. We consider as documentable items
the human effort. In the literature, there are many automatic for m (i) its parameters, (ii) the exceptions it throws, and (iii)
approaches for identifier splitting/expansion, but we preferred its return value. Such items are considered as documented if
to implement a simpler and more effective strategy at this there is an explicit reference to them in the Javadoc through
stage, since the number of identifiers to split/expand was the tags @param, @throws and @returns. readabilityD (m)
limited and our goal was to assess the correlation of the defined represents, instead, the readability of the Javadoc comments
metrics with the understandability effort. Thus, we wanted to assessed using the Flesch reading-ease test [32]. The higher
be sure to avoid introducing imprecision while computing the MIDQ the higher the internal documentation quality for m. We
metrics. consider the minimum, the average, and the maximum values
When dealing with the identifier used to name a method, of the MSQ metric for a given code snippet.
we also verify whether it is representative of what the method Concerning the API External Documentation Quality
does (Rp). We compute the textual overlap between the terms (AEDQ), it tries to capture the amount of information about
used in the identifier and in the method body. We tokenize APIs used in the given snippet s that can be acquired from
the method body to define its dictionary. Then, we count the external sources of documentation, such as Q&A websites.
The conjecture is that if external documentation is available, it TABLE II
is more likely that developers are able to understand the usage S YSTEMS USED IN OUR STUDY
of an API in a code snippet s. We compute the availability of System Java KLOC Category Description
external documentation for each external class c used in s
ANTLR 178 Desktop Lexer-parser
via the AEDQ(c) metric. First, we identify all Stack Overflow Car-report 45 Mobile Car costs monitoring
discussions related to c by running the following query: Hibernate 948 Framework ORM framework
Jenkins 231 Web Continuous integration
“title:‘how to’ < c > hasaccepted:yes [java]” K9 mail 121 Mobile Mail client
In other words, we select all Stack Overflow discussions that MyExpenses 101 Mobile Budget monitoring
OpenCMS 1059 Web Content Management System
(i) contain “how to” and the class name in the title, (ii) have an Phoenix 352 Framework Relational database engine
accepted answer, and (iii) concern Java (since our study has Spring 197 Framework Generic application framework
Weka 657 Desktop Machine-learning toolkit
been performed on Java snippets). Then, we sum the votes
assigned by the Stack Overflow users to the question in each
retrieved discussion, in order to have a quantitative information
about the interest of the developers’ community in such a class. 100 times in E through the statement import bar.∗. The
We assume that higher interest in a given API class implies class Foo2, belonging to the same package, is imported 99
a higher availability of external sources of information (e.g., times with the statement import bar.Foo2. In this case, we
discussions, code examples, etc.). We consider in our study increase the number of occurrences of classes belonging to
the minimum, the average, and the maximum values of the imported package in a proportional way. In the presented
AEDQ metric for the external classes used in s. example, we add 1 to the number of Foo’s imports, and 99 to
the number of Foo2 imports. We found that imports of entire
C. Developer-related metrics packages represent only 2.6% of all the imports and, therefore,
Since understandability is a very subjective feature of code, their impact is very low. EAP(c) is defined as the number of
we introduced three developer-related metrics. We measure the c imports normalized over the number of imports of cmax ,
programming experience of the developer who is required to where cmax is the most imported class we found in E (i.e.,
understand a snippet (PEgen and PEspec ) and the popularity [Link]).
of the API used in the snippet (EAP).
The common wisdom is that the higher the programming
IV. E MPIRICAL S TUDY D ESIGN
experience of developers, the higher their capability of under-
standing code. PEgen measures the programming experience
The goal of our study is to assess the extent to which the
(in years) of a developer in general (i.e., in any programming
considered 121 metrics correlate with code understandabil-
language). PEspec assesses instead the programming experi-
ity. The perspective is of researchers interested in analyzing
ence (in years) of a developer in the programming language in
whether code-related, documentation-related, and developer-
which a given snippet s is implemented. The higher PEspec , the
related metrics can be used to assess the understandability
higher the developer’s knowledge about the libraries available
level of a given piece of code. This study aims at answering
for such a programming language.
the following major research question:
With External API Popularity (EAP), we aim at capturing
the popularity of the external APIs used in a given snippet. What is the correlation between the 121 considered metrics
The assumption is that the lower the popularity, the lower and the understandability level of a given developer for a
the probability that a typical developer knows the API. If the specific code snippet?
developer is not aware of the APIs used in a snippet, it is likely
that she has to look for its documentation or to inspect its Given the wide and heterogeneous set of considered metrics,
source code, thus spending more effort in code understanding. answering this research question would allow us and, more
We rely on an external base of Java classes E to estimate in general, the research community to understand how far we
the popularity of an external class. We chose as E a 10% are from defining a set of metrics capable of automatically
random sample of classes from Java/Android projects hosted and objectively assessing code understandability.
on GitHub in 2016, totaling ∼2M classes from ∼57K Java
projects. We used Google BigQuery to extract all the imports A. Data collection
of all the classes belonging to such projects using a regular
expression. Then, we counted the number of times each class The context of the study consists of 50 Java/Android meth-
imported in E occurred in the import statements. Note that ods extracted from ten popular systems listed in Table II
in Java it is possible to import entire packages (e.g., import (five methods from each system). We first extracted all the
[Link].∗). In this case, it is difficult to identify the actual methods having 50±20 ELOCs (i.e., Effective Lines Of Code,
classes imported from the package. For this reason, we applied excluding blank and comment lines) from the systems. The
the following strategy. Let us assume that a class, Foo, is choice of the methods’ size (i.e., 50 ± 20 ELOCs) was driven
imported only once with the statement import [Link], but by the decision of excluding methods that are too trivial or
it is part of a quite popular package, bar, that is imported too complex to understand.
 Afterwards, we computed all the metrics described in for the method’s understanding before clicking on one of
Section III for the selected methods1 . Then, we used a the two buttons. If the participant clicked on “I understood
greedy algorithm for center selection [34] to select the 50 the method”, the method was hidden and she was required
most representative methods based on the defined metrics. to answer three verification questions about the method she
Given a set of candidate methods M and a set of already just inspected. The provided answers were stored for future
selected centers C, such algorithm chooses, in each iteration, analysis. An example of verification question is: “What does
arg maxm∈M dist(C, m), i.e., the candidate method which is the invoked method X do?”. To answer such a question, the
the farthest possible (in terms of considered metrics) from the participant should have understood also all the consequences
already selected centers. The first center is randomly selected. of invoking a specific external method.
In order to select exactly five snippets from each system, we
used the set of candidate methods from a specific system as B. Analysis Method
M until the five methods for such a system were selected; In the context of our study, we measure the understandabil-
then, we changed M with the set of candidate methods from ity level of a given piece of code from different perspectives
another system, and so on, until |C| = 50. Note that (i) we and correlate it with 121 considered metrics. We defined the
did not empty the C set when changing the candidate methods following four different independent variables.
(i.e., when moving from one system to another) to always keep Perceived Binary Understandability (PBU). This binary
track of the methods selected up to that moment, thus avoiding metric gets value 0 if the participant clicked on the “I cannot
the risk of adding to C methods similar to the ones already understand the method” button, otherwise (i.e., the participant
in C; (ii) we did not run the algorithm on the union of all clicked on “I understood the method”) it gets value 1. This
candidate methods to ensure the selection of five methods per metric simply aims at discriminating between understandable
system (thus increasing the heterogeneity of the final sample). vs. non-understandable methods as perceived by participants.
After having selected the 50 methods and computed the Time Needed for Perceived Understandability (TNPU).
values of the metrics of interest for each of these 50 methods, Measured as time, in seconds, spent by the participant while
we needed to define a ground-truth reporting the understand- inspecting the method before clicking on “I understood the
ability of each method. To this aim, we invited 46 Java method”. This metric cannot be computed when the participant
developers and CS students to participate in a survey where clicked on “I cannot understand the method”.
they were required to understand the selected methods. The Actual Understandability (AU). It gets value 0 if the
survey was implemented in a Web application and featured participant clicked on the “I cannot understand the method”
the following steps. First, we collected demographic data button. Otherwise, it is computed as the percentage of correct
about participants: (i) years of experience in programming and answers the participant provided to the three verification
more specifically in Java, and (ii) current position (e.g., CS questions. The metric is defined in the range [0, 1], where
student, developer etc.). This information was used in part to 1 indicates high understandability.
compute the developer-related metrics. After this preliminary Timed Actual Understandability (TAU). It gets value 0 if
step, each participant was required to understand a subset the participant clicked on the “I cannot understand the method”
of eight methods randomly selected from the 50 methods. button. Otherwise, it is computed as:
The Web application was designed to automatically balance 
T N P Uno

the number of evaluations for each of the 50 methods (i.e., T AU = AU 1 −
max T N P Uno
the number of participants understanding each method was
roughly the same). In total, we collected 324 evaluations across where AU is the percentage of correct answers, TNPU is
the 50 methods (∼6.5 evaluations per method on average), the time needed to understand the method and TNPU no is
since not all participants completed the survey. a modified TNPU where outliers (detected using the Tukey’s
The eight methods were presented individually (i.e., each test [35], with k = 3) are substituted with the maximum
method in a different page) to participants, and the Web value of TPNU, which is not an outlier. The higher AU,
application allowed navigation of the method and access to the higher TAU, while the higher TNPU, the lower TAU.
the methods/classes invoked/used by it. Also, participants were Also, TAU is defined in [0, 1]. We considered the relative
T N P Uno
allowed to browse the Web to collect information about types, time ( max T N P Uno ) instead of the absolute time so that TAU
APIs, data structures, etc. used in the method. This was done gives the same importance to both the correctness achieved
to simulate the typical understanding process performed by (AU) and the time needed (TNPU). We removed outliers from
developers. We asked participants to carefully read and fully TNPU because the maximum value of such a variable is 1,649
understand each method. Participants could, at any moment, seconds, much greater than the third quartile (171 seconds).
click on the button “I understood the method” or the button Using such a value as the maximum would have flattened
“I cannot understand the method”. In both cases, the Web down all the relative times.
application stored the time spent, in seconds, by the developer We computed these four variables for each of the 324
evaluations performed by participants (i.e., for each method
1 Excluding the “Developer programming experience” and the “Developer each participant tried to understand). Before calculating the
Java experience”. correlation between each of the 121 metrics described in
Section III and PBU, TNPU, AU, and TAU, we excluded 2 3

of the 121 considered metrics (i.e., NMImin and ITIDmin ), 5

because the value of such metrics was 0 for all the snippets Position
Ph.D. student
taken into account. Then, we verified which metrics strongly 6 Professional developer
correlate among the 121. This was done to exclude from our Master's student
analysis redundant metrics capturing the same information Bachelor's student

in different ways. We compute the Kendall rank correlation 32

coefficient (i.e., Kendall’s τ ) [36] to determine whether there

are pairs exhibiting a strong correlation. We adopted the Fig. 1. Participants to the study
Kendall’s τ , since it does not assume the data to be normally
distributed nor the existence of a straight linear relationship
between the analyzed pairs of metrics. Cohen [37] provided any strong claims about the ability of these correlated metrics
a set of guidelines for the interpretation of the correlation to act as proxies for code understandability.
coefficient. It is assumed that there is no correlation when
0 ≤ |τ | < 0.1, small correlation when 0.1 ≤ |τ | < 0.3,
medium correlation when 0.3 ≤ |τ | < 0.7, and strong A. Perceived Binary Understandability (PBU)
correlation when 0.7 ≤ |τ | ≤ 1. The metric which has the highest correlation with PBU is
For each pair of metrics exhibiting a strong correlation “Max line length” (τ ≈ −0.17), one of the metrics introduced
(i.e., with a Kendall’s |τ | ≥ 0.7), we excluded the ones by Buse and Weimer [10] for readability prediction. Note that
which presented the highest number of missing values2 or, if Buse and Weimer also found that such a metric is the most
equals, one at random. This allowed us to reduce the number important one for readability prediction [10]. Therefore, this
of investigated metrics from 121 to 74. Note that such a is a further confirmation of the fact that, generally, developers
removal of metrics was only done for presentation purposes, tend to perceive code with long lines as less pleasant. We also
considering the high number of metrics taken into account found that there is a low correlation between PEspec and PBU:
in our study. Finally, we computed the Kendall correlation developers with more experience in the specific programming
between each of the remaining 74 metrics and PBU, TNPU, language tend to have a slightly higher confidence and they
AU, and TAU, to verify whether some of them are able to tend to perceive snippets of code as understandable more
capture the (actual and perceived) understandability of code. frequently than developers with less experience. There is also
C. Replication Package a low correlation between PBU and AIDQmin : if the internal
APIs are well documented, developers tend to answer that they
All the data used in our study is publicly available [31]. We
understood the snippet slightly more frequently.
provide the research community with our dataset reporting
the (perceived and actual) understandability achieved by the
46 participants, hoping that the availability of such a dataset B. Time Needed for Perceived Understandability (TNPU)
will help in fostering research on the definition of a metric to The time needed to answer “I understood the snippet” is
automatically assess code understandability. (weakly) correlated with the programming experience, both
V. E MPIRICAL S TUDY R ESULTS PEgen (τ ≈ 0.16), and PEspec (τ ≈ 0.15). Surprisingly, the
correlation is positive: the higher the experience, the higher the
Fig. 1 provides information about the participants involved
time needed to complete the comprehension step (and click
in our study. Most of them (∼ 70%) are bachelor’s students—
the button “I understood the snippet”). This result seems to
mixed in terms of years of programming experience. The
suggest that experienced developers tend to take more time
sample of participants also included six master’s students,
to understand the snippets, while less experienced developers
three Ph.D. students, and five professional developers.
tend to complete the comprehension process more quickly.
Fig. 2 reports a heat-map showing the Kendall’s τ between
However, it is worth remembering that TNPU assesses the
each metric and each understandability variable. We only show
time needed for the perceived understandability, and not the
in Fig. 2 correlations having |τ | > 0.1. As it is evident by
actual one (TAU).
looking at Fig. 2, very few metrics have a correlation with
understandability variables higher than |0.1|. Specifically
nine metrics have a weak correlation with PBU, 10 with C. Actual Understandability (AU)
TNPU, 8 with AU and 16 with TAU. 50 out of the 74 metrics The metric that has the highest correlation with AU is the
taken into account show no correlation at all. In the following, average Textual Coherence of the snippet (τ ≈ −0.16). The
we discuss the observed correlations by understandability fact that the correlation is negative is surprising, because we
metric. However, it must always be kept in mind that these are expected a higher Textual Coherence to imply a higher under-
all weak correlations and, as such, do not allow for making standability. Other examples of metrics correlated with AU are
2 Some metrics cannot be computed in some cases. For example, “Area of Number of parameters and DFT of conditionals (τ ≈ −0.15),
comments/literals” cannot be computed if the method does not contain literals. indicating that higher complexity reduces understandability.
 Color key

−1 −0.5 0 0.5 1
Correlation

0.16 0.11 PE (gen) #numbers (avg)

0.13 0.15 0.12 PE (spec) #operators (avg)

0.11 −0.15 Cyclomatic Comp. #parenthesis (avg)

IMSQ (min) −0.11 #periods (avg)

0.11 0.12 MIDQ (min) Identifiers length (max)

AEDQ (min) #keywords (max)

EAP (min) −0.17 −0.12 −0.13 Line length (max)

MIDQ (avg) #identifiers (max)

AEDQ (avg) #characters (max)

EAP (avg) #words (max)

IMSQ (max) Entropy

MIDQ (max) Volume

Readability LOC

ITID (avg) #assignments (dft)

NMI (avg) −0.12 #comparisons (dft)

0.11 NMI (max) 0.12 −0.15 −0.18 #conditionals (dft)

CIC (avg) Indentation length (dft)

CICsyn (avg) −0.11 Line length (dft)

CICsyn (max) #numbers (dft)

CR #operators (dft)

−0.11 NM (avg) −0.12 #comments (Visual X)

NM (max) −0.11 #identifiers (Visual X)

0.11 −0.16 −0.17 TC (avg) #numbers (Visual X)

0.1 TC (min) #strings (Visual X)

0.15 −0.11 −0.16 TC (max) #strings (Visual Y)

#assignments (avg) #literals (Visual X)

0.1 #blank lines (avg) #identifiers (area)

−0.11 #commas (avg) #keywords (area)

#comments (avg) 0.11 #strings (area)

0.1 −0.12 #comparisons (avg) #operators (area)

Identifiers length (avg) #keywords/#comments (area)

−0.11 −0.15 #conditionals (avg) #keywords/#identifiers (area)

Indentation length (avg) #operators/#identifiers (area)

#keywords (avg) #aligned blocks

Line length (avg) 0.12 −0.13 #nested blocks (avg)

0.11 −0.12 #loops (avg) −0.12 −0.15 −0.13 #parameters

#identifiers (avg) #statements

TAU
TNPU

AU

TNPU
TAU

PBU
PBU

AU

Fig. 2. Heat-map of the correlation among metrics and understandability proxies

D. Timed Actual Understandability (TAU) An important implication of this finding is that readability
The metric with the highest correlation with TAU is DFT and complexity metrics are not correlated with understand-
of conditionals (τ ≈ −0.18). Again, high complexity reduces ability, which is a result deserving additional empirical inves-
the understandability. It should be noticed that for TAU, i.e., tigations. For instance, the snippet in Fig. 3 from Weka was
actual understandability, we do not observe any correlation understood by all the participants who evaluated such a snippet
with the programming experience. during the study (i.e., mean PBU = 1). Note also that the mean
TAU = 0.72, i.e., the participants answered correctly two
E. Discussion
out of three the verification questions, on average. However,
Given the large and diverse set of metrics considered in the snippet is considered as unreadable (0.14) when using the
our study, we expected to observe some form of correlation readability model proposed by Scalabrino et al. [14].
allowing us to start designing an understandability metric/-
model. However, what we obtained is a clear and bold negative Another metric that has no correlation with understand-
result: none of the investigated metrics show a significant ability is LOC. Even if in our study we considered snippets
correlation with the snippets’ understandability. with a small variation of LOCs (maximum 40), it would have
private void addButtons () { metrics, we tested our implementation and, when needed (e.g.,
JButton okBut = new JButton ( " OK " ) ;
JButton cancelBut = new JButton ( " Cancel " ) ; for the IM SQ metric during the identifiers splitting/expan-
JPanel butHolder = new JPanel () ;
sion), relied on manual intervention to ensure the correctness
butHolder . setLayout ( new GridLayout ( 1 , 2 ) ) ; of the computed metrics. As for the independent variables,
butHolder . add ( okBut ) ;
butHolder . add ( cancelBut ) ; we tried to capture both the perceived and the actual code
add ( butHolder , BorderLayout . SOUTH ) ;
understandability. However, different results might be achieved
okBut . addActionListener ( new ActionListener () {
@Override by exploiting different variables.
public void actionPerformed ( ActionEvent e ) {
if ( m_modifyListener != null ) { Threats to internal validity concern external factors we
m_modifyListener . setModifiedStatus ( did not consider that could affect the variables and the rela-
ClassifierPerformanceEvaluatorCustomizer . this , true ) ;
} tions being investigated. Since two of the understandability
if ( m_evaluationMetrics . size () > 0) { proxies are time-related (i.e., they are based on the time
StringBuilder b = new StringBuilder () ;
for ( String s : m_evaluationMetrics ) { participants spent while understanding the code), it is possible
b. append (s ) . append ( " ," ) ;
} that some participants were interrupted by external events
String newList = b . substring (0 , b . length () - 1) ;
m_cpe . setEvaluationMetricsToOutput ( newList ) ; while performing the comprehension task. For this reason,
}
if ( m_parent != null ) {
we excluded outliers (i.e., participants requiring more than
m_parent . dispose () ; Q3 +(2.5×IQR) seconds to understand a code snippet, where
}
} Q3 is the third quartile and IQR is the Inter Quartile Range).
}) ;
Threats to conclusion validity concern the relation be-
cancelBut . addActionListener ( new ActionListener () {
@Override tween the treatment and the outcome. Although this is mainly
public void actionPerformed ( ActionEvent e ) {
an observational study, wherever possible we used an appro-
customizerClosing () ;
if ( m_parent != null ) {
priate support of statistical procedures.
}
m_parent . dispose () ; Threats to external validity concern the generalizability
} of our findings. Our study has been performed on a large,
}) ;
} but limited, set of metrics and by involving 46 participants
comprehending a subset of 50 methods extracted from 10
Fig. 3. Example of understandable snippet with low readability Java systems. Clearly, our findings hold for the considered
population of participants and for Java code. Larger studies
possibly involving more participants and code snippets written
been reasonable to expect at least a small correlation with in other languages should be performed to corroborate or
understandability. contradict our results.
Almost all of the metrics that we introduced in this study to
capture aspects of code reasonably related to understandability VII. C ONCLUSION AND F UTURE W ORK
do not correlate with any of the considered proxies. The only
exception is AIDQmin (API Internal Documentation Quality). We presented an empirical study investigating the correla-
Indeed, such a metric has a low correlation with both PBU tion between code understandability and 121 metrics related
and TAU. This result does not completely exclude the fact that to the code itself, to the documentation available for it, and
external documentation, quality of identifiers and popularity of to the developer understanding it. We asked 46 developers
external APIs could play a role in the automatic assessment of to understand 50 Java snippets, and we gathered a total of
code understandability, but it underlines that designing metrics 324 evaluations. We assessed the perceived and actual partic-
able to capture such an aspect is far from trivial and represents ipants’ understanding for each snippet they inspected and the
a big challenge for the research community. time they needed for the comprehension process. Our results
Finally, we should not exclude the possibility that code demonstrate that, in most of the cases, there is no correlation
understandability is simply too subjective to be captured by between the considered metrics and code understandability.
any metric. However, the research community has been able In the few cases where we observed a correlation, its mag-
to successfully define proxies for other highly subjective code nitude is very small. The most important (and surprising)
properties in the past, such as code readability. For this reason, conclusions of our study are that (i) there is no correlation
we believe that more research targeting the automatic assess- between understandability and readability, and (ii) metrics
ment of code understandability still represents an important generally used for effort estimation and commonly associated
and valuable research direction. with understandability, such as cyclomatic complexity, actually
have low or no correlation with understandability.
VI. T HREATS TO VALIDITY Our study lays the foundations for future research on new
Threats to construct validity, concerning the relation metrics actually able to capture facets of code understandabil-
between theory and observation, are mainly due to the mea- ity. For this reason, we publicly release our dataset to enable
surements we performed, both in terms of the 121 metrics the research community to investigate this direction further.
that we studied as well as when defining the four independent Acknowledgments. Bavota was supported in part by the
variables for the understandability level. Concerning the 121 SNF project JITRA, No. 172479.
 R EFERENCES [17] A. Capiluppi, M. Morisio, and P. Lago, “Evolution of understandability
in oss projects,” in 8th European Conference on Software Maintenance
[1] R. Minelli, A. Mocci, and M. Lanza, “I know what you did last summer and Reengineering, 2004, pp. 58–66.
- an investigation of how developers spend their time,” in 23rd IEEE [18] D. Lawrie, C. Morrell, H. Feild, and D. Binkley, “Effective identifier
International Conference on Program Comprehension, 2015, pp. 25–35. names for comprehension and memory.” Innovations in Systems and
[2] K. K. Aggarwal, Y. Singh, and J. K. Chhabra, “An integrated measure Software Engineering, vol. 3, no. 4, pp. 303–318, 2007.
of software maintainability,” in Annual Reliability and Maintainability [19] ——, “What’s in a name? a study of identifiers,” in 14th International
Symposium., 2002, pp. 235–241. Conference on Program Comprehension, 2006.
[3] M. Thongmak and P. Muenchaisri, Measuring Understandability of [20] B. Caprile and P. Tonella, “Restructuring program identifier names,” in
Aspect-Oriented Code. Berlin, Heidelberg: Springer Berlin Heidelberg, International Conference on Software Maintenance, 2000, pp. 97–107.
2011, pp. 43–54. [21] D. Lawrie, H. Feild, and D. Binkley, “Syntactic identifier conciseness
[4] J. c. Lin and K. c. Wu, “A model for measuring software understandabil- and consistency,” in 6th International Working Conference on Source
ity,” in 6th IEEE International Conference on Computer and Information Code Analysis and Manipulation, 2006, pp. 139–148.
Technology, 2006, pp. 192–192. [22] E. Enslen, E. Hill, L. L. Pollock, and K. Vijay-Shanker, “Mining source
[5] T. Roehm, R. Tiarks, R. Koschke, and W. Maalej, “How do professional code to automatically split identifiers for software analysis,” in 6th
developers comprehend software?” in 34th International Conference on Working Conference on Mining Software Repositories, 2009.
Software Engineering (ICSE), 2012, pp. 255–265. [23] V. Arnaoudova, M. Di Penta, and G. Antoniol, “Linguistic antipatterns:
[6] D. Srinivasulu, A. Sridhar, and D. P. Mohapatra, Evaluation of Software What they are and how developers perceive them,” Empirical Software
Understandability Using Rough Sets. New Delhi: Springer India, 2014, Engineering, vol. 21, no. 1, pp. 104–158, 2015.
pp. 939–946. [24] S. Misra and I. Akman, “Comparative study of cognitive complexity
[7] C. Chen, R. Alfayez, K. Srisopha, L. Shi, and B. Boehm, Evaluating measures,” in 23rd International Symposium on Computer and Informa-
Human-Assessed Software Maintainability Metrics. Singapore: Springer tion Sciences, Oct 2008, pp. 1–4.
Singapore, 2016, pp. 120–132. [25] E. J. Weyuker, “Evaluating software complexity measures,” IEEE Trans-
[8] M. A. Storey, “Theories, methods and tools in program comprehension: actions on Software Engineering, vol. 14, no. 9, pp. 1357–1365, 1988.
past, present and future,” in 13th International Workshop on Program [26] J. Bansiya and C. G. Davis, “A hierarchical model for object-oriented
Comprehension, 2005, pp. 181–191. design quality assessment,” IEEE Transactions on Software Engineering,
[9] M. A. D. Storey, K. Wong, and H. A. Muller, “How do program vol. 28, no. 1, pp. 4–17, Jan 2002.
understanding tools affect how programmers understand programs?” in [27] N. Kasto and J. Whalley, “Measuring the difficulty of code compre-
4th Working Conference on Reverse Engineering, 1997, pp. 12–21. hension tasks using software metrics,” in 15th Australasian Computing
[10] R. P. L. Buse and W. Weimer, “Learning a metric for code readability,” Education Conference. Australian Computer Society, Inc., 2013, pp.
IEEE Transactions on Software Engineering, vol. 36, no. 4, pp. 546–558, 59–65.
2010. [28] K. Shima, Y. Takemura, and K. Matsumoto, “An approach to experimen-
[11] D. Posnett, A. Hindle, and P. T. Devanbu, “A simpler model of tal evaluation of software understandability,” in International Symposium
software readability.” in 8th Working Conference on Mining Software on Empirical Software Engineering, 2002, pp. 48–55.
Repositories, 2011, pp. 73–82. [29] ISO/IEC. Iso/iec 9126 software engineering — product quality — part
[12] J. Dorn, “A general software readability model,” Master’s 1: Quality model.
thesis, University of Virginia, Department of Computer Science, [30] T. J. McCabe, “A complexity measure,” IEEE Transactions on software
[Link] Engineering, no. 4, pp. 308–320, 1976.
2012. [31] S. Scalabrino, G. Bavota, C. Vendome, M. Linares-Vásquez,
[13] E. Daka, J. Campos, G. Fraser, J. Dorn, and W. Weimer, “Modeling D. Poshyvanyk, and R. Oliveto, “Replication package.”
readability to improve unit tests,” in 10th Joint Meeting of the European [Link]
Software Engineering Conference and the ACM SIGSOFT Symposium [32] R. F. Flesch, How to write plain English: A book for lawyers and
on the Foundations of Software Engineering, 2015, pp. 107–118. consumers. Harpercollins, 1979.
[14] S. Scalabrino, M. Linares-Vásquez, D. Poshyvanyk, and R. Oliveto, [33] D. Schreck, V. Dallmeier, and T. Zimmermann, “How documentation
“Improving code readability models with textual features,” in 24th IEEE evolves over time,” in Ninth International Workshop on Principles of
International Conference on Program Comprehension, 2016. Software Evolution. ACM, 2007, pp. 4–10.
[15] J.-C. Lin and K.-C. Wu, “Evaluation of software understandability based [34] J. Kleinberg and É. Tardos, Algorithm design. Pearson Education India.
on fuzzy matrix,” in IEEE International Conference on Fuzzy Systems, [35] J. W. Tukey, “Exploratory data analysis,” 1977.
2008, pp. 887–892. [36] M. G. Kendall, “A new measure of rank correlation,” Biometrika, vol. 30,
[16] M. Bartsch and R. Harrison, “An exploratory study of the effect no. 1-2, p. 81, 1938.
of aspect-oriented programming on maintainability,” Software Quality [37] J. Cohen, Statistical power analysis for the behavioral sciences, 2nd ed.
Journal, vol. 16, no. 1, pp. 23–44, 2008. Lawrence Earlbaum Associates, 1988.