An Empirical Study of API Usability

Marco Piccioni, Carlo A. Furia, and Bertrand Meyer

ETH Zurich, Switzerland
Email: {[Link], [Link], [Link]}@[Link]

Abstract—Modern software development extensively involves consistency between different API functionalities [1]. In our
reusing library components accessed through their Application study, we collected and interpreted the feedback of participants
Programming Interfaces (APIs). Usability is therefore a funda- according to the cognitive dimensions. The second methodol-
mental goal of API design, but rigorous empirical studies of API ogy used in our study design is based on usability tokens,
usability are still relatively uncommon. In this paper, we present an original classification of the participant reactions (such as
the design of an API usability study which combines interview
questions, based on the cognitive dimensions framework, with
“surprise” or “incorrect choice”) as they try to understand
systematic observations of programmer behavior while solving and use API functionalities to perform the assigned tasks.
programming tasks, based on “tokens”. We also discuss the We tallied usability tokens when replaying the videos of
implementation of the study to assess the usability of a persistence the participants’ performances. As we discuss in Section VI,
library API (offering functionalities such as storing objects combining cognitive dimensions and usability tokens leads to
into relational databases). The study involved 25 programmers a rich characterization of API usability issues.
(including students, researchers, and professionals), and provided
additional evidence to some critical features evidenced by related The second paper contribution is an actual execution of
studies, such as the difficulty of finding good names for API the study to evaluate the usability of a persistence library
features and of discovering relations between API types. It also written in Eiffel. The study—which we detail in Section IV—
discovered new issues relevant to API design, such as the impact involved 25 participants, including both students and profes-
of flexibility, and confirmed the crucial importance of accurate sional developers. The participants solved five tasks requiring
documentation for usability. to access, modify, and query a relational database through the
services offered by the persistence library API; we recorded
I. I NTRODUCTION their performance to be able to analyze it postmortem. After
each session, we also conducted a structured interview with the
It is not by chance that software has become so complex, participants—about their experience and expectations—whose
but by building over the solid underpinnings of abstraction questions are characterized by the cognitive dimensions.
and modularity. Modern software is a complex composition
of library components, whose elementary functionalities are Section V analyzes the study results based on the analysis
combined to achieve domain-specific applicative goals. A of performance and interviews; and Section VI discusses the
component’s interface consists of features that clients can findings in a more general context, which include:
call; the combination of interface features and a description
of their usage protocol and semantics is what is normally • Finding descriptive, non-ambiguous names for API fea-
called API: Application Programming Interface. Using library tures is problematic given that programmers may be used
components solely based on their APIs makes it possible to to different terminologies.
abstract away implementation details and practice modularity; • Discovering relations between API types (classes) re-
therefore, learning and using APIs are everyday tasks for soft- quires significant effort; simple designs are beneficial,
ware developers. From a research perspective, it is interesting especially to less experienced programmers.
to understand what makes APIs easy or hard to use; this is the • Accurate and complete documentation is a crucial issue
overall goal of this paper. for API usability; all the major usability flaws discovered
in our study trace back to unsatisfactory documentation.1
Previous empirical studies of API usability—which we • Flexibility is a double-edged sword in API design: expe-
review in Section VIII—point to some critical issues such as rienced programmers can take advantage of it, but it may
the ease of discovering relationships between types offered by confuse those with less practice.
an API and how to instantiate new objects. With our work, we
want to find additional evidence to confirm or dispute these Finally, Section VII mentions possible threats to the valid-
issues, as well as to find out new criticalities not detected in ity of the study; and Section IX concludes.
previous work.
To this end, the first contribution of our paper is the II. R ESEARCH Q UESTIONS
design of an empirical study of API usability. The study’s We organize the study around four main research questions,
research questions—described in Section II—are based on the which characterize fundamental aspects of usability: under-
findings of previous studies with the goal of corroborating standability, abstraction level, reusability, and learnability. The
and extending them. Section III describes the specific study questions cover the critical aspects emerged in previous API
design, which combines two methodologies useful for assess- usability studies (see Section VIII) but are sufficiently general
ing usability. The first is the notion of cognitive dimensions:
elements that characterize the expectations of users and what 1 As documentation, we mainly consider: public method signatures, com-
an API actually provides, such as the abstraction level and the ments, and contracts of the class features.
to make room for new findings and specific aspects emerged Our empirical study follows the guidelines of the cognitive
in our study (see Sections V and VI). dimensions framework, and more specifically Clarke’s dimen-
sions of API usability [3]. It is based on the execution of
RQ1 What is the effort required to understand the se- programming tasks that require the participants to write client
mantics of API features based on their names and code combining API features to achieve certain functionalities;
documentation? the comparison between expectations and reality is based
RQ2 Does the API’s abstraction level cater to usability? on the participants’ performance on the programming tasks.
RQ3 Does the API’s design facilitate reuse and concise- With the goal of having a multidimensional assessment, which
ness in client code? helps reduce the impact of subjective perceptions, we collect
RQ4 Can API usage be learned easily and incrementally? data about the discrepancies between user expectations and
actual system features from two different sources, as described
RQ1 addresses the fundamental problems tackled by pro- hereafter.
grammers using the services offered by an API: what they have
to do to understand what each feature of the API represents After completing the tasks, we conducted a structured in-
and how it should be invoked. This question also encompasses terview with the participants involving a fixed list of questions
specific issues such as whether the API feature names are about their experience and expectations. This explicit feedback
descriptive, whether the relationships between API elements provided by programmers through the interview highlights
are clear and unambiguous, and how to access the features their perceived usability, the difficulties they experienced, and
through object creation, method call, or other means. the features that explicitly appreciated. Section III-A describes
the interview questions in some detail and discusses how they
RQ2 evaluates whether the level of abstraction is good articulate the research questions of Section II.
for usability. On the one hand, it should guarantee that
programmers can proficiently use the API without knowing Independent of the interview data, we also collected feed-
(or assuming) implementation details. On the other hand, the back given implicitly by the participants during their perfor-
abstractions should match the conventions and practices of mance. To this end, we asked them to follow the thinking-
programmers, without being elegantly abstract at the expense aloud protocol and recorded the video and audio of their
of understandability and other practical concerns. Summa- performances. The thinking-aloud protocol is based on Eric-
rizing with a slogan, this research question asks whether sson and Simon’s seminal work [4] and consists in having
the API “makes simple things simple, and complex things the participants declare in speech their mental process as
possible” [2]. it develops, including the doubts and questions they have,
the solution strategies they consider, and the reasons that
RQ3 determines to what extent the client code that can justify their decisions. This protocol is widely used in usability
be written using the API is concise, terse, maintainable, and testing [5], because it makes the external observer aware
extensible. Concretely, this question addresses problems such of the cognitive processes leading to a certain performance.
as: if we have an application using some API features and In our study, we perused postmortem the recordings of the
slightly vary or generalize its requirements, how hard is it to performances searching for episodes revealing the expectations
adapt the application to meet the extended requirements? of programmers and how they compared to the API features.
RQ4 targets the API learning process to see if it can be As we describe in Section III-B, we classify the episodes
incremental (that is, gradual and not requiring disproportionate in usability tokens, which express mismatch between user
efforts) and whether performing a certain programming task expectations and actual performance. The tokens make the
using the API has a positive impact on performing other, implicit data collection more uniform and aligned to the issues
related but different, tasks. This question is related to RQ1 targeted by the research questions.
and RQ3 but emphasizes the learning process rather than its The rest of this section details the structured interview and
practical outcomes. the usability tokens used in our study.

III. S TUDY D ESIGN A. Structured Interview

Usability is a multifaceted feature which is hard to assess Following Clarke’s guidelines [3], and reflecting the re-
reliably because it involves somewhat subjective aspects such search questions of Section II, the interview consists of twelve
the specific habits and styles of individual programmers. To ad- questions which characterize various aspects of usability as
dress such problems, the cognitive dimensions framework [1] assessed by users. The questions are as follows, roughly
suggests to evaluate usability operationally based on compar- grouped by the research question they target (although some
isons. First, we outline a number of aspects that are sufficient of them target multiple research questions).
to characterize the multiple facets of usability in our specific
domain. Then, we base the assessment on the comparison, for Questions regarding RQ1 (understandability):
each aspect, between the expectations of users and what the
system actually provides. For example, regarding incremental 1) Do you find that the API types map to the domain
learnability in APIs, users may expect to be able to execute concepts in the way you expected?
incomplete code to obtain feedback on API behavior, whereas 2) Do you feel you had to keep track of information not
the API might require specific complete call sequences before represented by the API to solve the tasks?
the code is executable without errors; this would signal an 3) Do you feel you had to learn many classes and depen-
imperfection in the API’s usability. dencies to solve the tasks?

2
Questions regarding RQ2 (abstraction): Token “missed”: the developer’s activity shows that she has
missed some important abstraction or feature of the API, which
4) Do you find the API abstraction level appropriate to the would have been useful to effectively solve the current task.
tasks? For example, the API may offer a feature to remove the first
5) Did you need to adapt the API (inheriting from API occurrence of a given element in the list, but the developer may
classes, overriding default behaviors, providing non-API miss it and implement the same functionality by first searching
types) to meet your needs? and then deleting.
6) Do you feel you had to understand the underlying imple-
mentation to be able to use the API? Token “incorrect”: the developer uses the API incorrectly,
in a way that introduces errors or implements an incorrect
Questions regarding RQ3 (reusability):2 functionality. For example, popping elements from a stack
without checking for emptiness may lead to errors when the
7) Does the amount of code required for each task seem stack is empty.
about right, too much, or too little for you?
8) How easy was it to evaluate your own progress (interme- Token “unexpected”: the developer uses the API in a way
diate results) while solving the tasks? undocumented or otherwise unforeseen by the API intended
9) Do you feel you had to choose one way (out of many) to design. Unlike the “surprise” token, this token adopts the point
solve a task in the scenario? of view of API designers. For example, the user may inherit
10) Do you feel you would have to change much in your from a stack class and override the push method to allow
code to access another kind of persistence store, or write insertion in the middle of the stack; this is not necessarily
another query? wrong, but violates the original design abstraction.
Questions regarding RQ4 (learnability): The usability tokens are largely orthogonal to the cognitive
dimensions (on which the interview questions are based), in
11) Once you performed the first two tasks, was it easier to that the same token may characterize events involving different
perform the remaining tasks? cognitive dimensions. For example, an element of surprise
12) Does the code required to solve the tasks match your may reveal inconsistent abstractions, poor understandability,
expectations? or both. This helps make the data from the interviews largely
independent of the usability token classification.
We use the cognitive dimensions framework to formulate
questions that may discover the existence of problematic issues
with the API, rather than aiming at “proving” its usability. IV. S TUDY S ETUP
This angle—reminiscent of the way testing cannot not prove
programs correct but discover the existence of errors—is We implemented the study design discussed in Section III
aligned to the best practices of using the cognitive dimensions to assess the API usability of a recently developed persistence
framework [6], [7]. library for Eiffel. In the following subsections, we give a few
details about: the library and its API; the programming tasks
involving the API which we submitted to the participants; the
B. Usability Tokens demographics of the participants; and the protocol used to
We classify the salient events occurring during participant carry out the experiments.
performances into five tokens. While the tokens describe
different dimensions, the same event may be associated to A. Persistence Library API
multiple tokens when it reveals aspects of different nature. For
each token, we make simple examples based on a hypothetical We evaluated the API of ABEL (A Better EiffelStore
data-structure library to make the description independent of Library), a library offering an object-oriented interface to
the specific application domain used in our study (namely, persistence-related functionalities like storing and retrieving
persistence). data using files (serialization) and relational databases. The
first author developed ABEL as part of his PhD work [8].
Token “surprise”: the developer discovers aspects of the
ABEL is written in Eiffel and is meant as an improvement over
API that clearly go against her expectations and original
the persistence services offered by Eiffel’s standard libraries.
intuitions. For example, the generic container classes may not
As it introduces features normally available in other mature
be usable with primitive types (e.g., integers) but only with
persistence frameworks such as Spring [9] and Hibernate [10],
object types; this requires a special treatment for certain types,
it should be usable by “standard” programmers, and hence it
which the user may find surprising.
is interesting to assess its usability.
Token “choice”: the developer is faced with a choice and
she must understand the alternatives to proceed in the right ABEL consists of 81 classes grouped into 11 clusters
way. For example, the element at a given position in a list (roughly equivalent to Java packages) for a total of roughly
may be accessible either with a method call [Link](i) or using 10500 lines of code, comments, and assertions. In Eiffel, it
the array notation s[i]; the user may have to understand if the is customary to use assertions in the form of contracts (pre-
two options are equivalent and choose the most appropriate and postconditions, and class invariants) to specify the essen-
one (in terms of correctness and code readability). tial requirements and behavior of methods. Therefore, when
browsing the API of ABEL, developers can see the signature,
2 Question 10 is specific to the domain of the API we evaluated, but it can comments, and contracts of its features, which constitute the
be easily generalized to different application domains. documentation of the API’s functionalities.

3
B. Tasks D. Protocol
The study participants solved five tasks involving accessing Each participant performed in an individual session tak-
a relational database using ABEL’s features. Solving the tasks ing place in an isolated office. The first author, henceforth
requires to store objects of a simple class PERSON, also the “proctor”, administered all the sessions according to the
given to the participants, and includes operations found to be following protocol.
critical in previous work [11], [12]. An outline of the five tasks
The proctor starts with a brief overview of the whole
follows.
process. It then administers a fifteen-minute tutorial showing
Task 1: initialize a REPOSITORY instance through a the basics of the Eiffel language and of the EiffelStudio
factory method; use the REPOSITORY instance through a IDE running on the laptop used for all the experiments. In
CRUD EXECUTOR to insert instances of PERSON into the particular, the tutorial highlights the IDE functionalities useful
repository. to browse library documentation (consisting of classes, method
signatures, header comments, and contracts) and to inspect the
Task 2: instantiate a QUERY class; use it through the inheritance relations among classes (such as listing all ances-
CRUD EXECUTOR to retrieve PERSON objects from the tors, descendants, clients, or suppliers of a given class). The
repository; and inspect the results. only documentation about ABEL available to the participant
during the study is accessed using these IDE features.
Task 3: change the state of the PERSON objects; update the
repository with the new objects using the CRUD EXECUTOR. After the tutorial, the proctor describes the thinking-aloud
protocol (Section III) and asks the participant to stick to it
Task 4: delete one of the PERSON objects and remove it during his or her performance. Then, the proctor opens a
from the repository using the CRUD EXECUTOR. project consisting of the PERSON class and a “main” client
class including a terse description of the five tasks as comments
Task 5: execute a complex query that requires selection (see Section IV-B). The project is set up with the “full void-
criteria. Developers can choose between using “predefined” safety” flag, which entails that the compiler statically checks
criteria (implemented using strings) and using function objects for possible dereferencing of void references (null references in
(called “agents” in Eiffel, and similar to C#’s delegates). Java); this helps avoid basic programming mistakes and lets the
participants focus on correctly using the API functionalities.
Finally, the proctor starts the audio/video recording of the
C. Participants session and invites the participant to begin.

We recruited 25 participants including 10 computer science During the experiment, the proctor sits in the same room
students at ETH (6 bachelor’s and 4 master’s), 7 researchers avoiding interactions with the participant. In a few cases, some
pursuing a PhD (2 in our group, 2 in other groups of the of the participants asked for the proctor’s instant help, mainly
computer science department of ETH, 2 in the computer sci- with using functionalities of the IDE or with the syntax of
ence department of other universities, and 1 from the robotics the Eiffel language. The proctor only answered requests that
group in the mechanical engineering department of ETH), 2 were independent of the specific tasks or the API function-
post-doctoral researchers also at ETH (1 from the computer alities, giving the minimal information necessary to proceed.
science department, and 1 from the mechanical engineering Section VII discusses this potential threat, to demonstrate that
department), and 6 professional programmers working for its impact on the soundness of the experiments is negligible.
various software companies mostly in the Zurich area. All The proctor lets the experiment continue until the partici-
the participants to the study were unpaid volunteers; the only pant completes all the five tasks. There is no time limit because
requirement on our side was that they had at least one year of the focus of the experiments is assessing usability, not measur-
experience with object-oriented programming. ing programming efficiency (something would have made not
The following table shows some statistics about their much sense anyway, given the heterogeneous experience of the
background in terms of: years of programming experience with participants). We still report the time taken by the participants
object-oriented programming languages; years of experience to show that it always was within a reasonable range: the fastest
with the Eiffel language; and years of experience as profes- participant finished in 32 minutes, the slowest in 118 minutes,
sional programmers. the median time was 70 minutes, and the standard deviation
22 minutes.
min median max stddev
After completion of the tasks, the proctor interviews the
Object-Oriented 1 5 22 4
participant asking the questions of Section III-A and recording
Eiffel 0 1 18 5
his or her answers. This concludes the experimental session.
Professional 0 2 21 5
The statistics show that the participant pool covers a V. R ESULTS
wide range of experiences; all participants, however, have a
programming background sufficient to make the experiment We present the results of the study in subsections corre-
meaningful. Furthermore, we ascertained that all the partici- sponding to the four research questions of Section II. For each
pants had at least some familiarity with the basic operations question, we discuss the data both from the interviews and
of a relational database, but none of them had used the ABEL from the usability tokens, which is summarized in Tables I
library before. and II.

4
 Table I classifies answers to each interview question however, this casts some doubts on the practicality of concrete
(Section III-A) into “yes”, “no”, and “sometimes”, giving factories, corroborating previous empirical findings on this
both the absolute number of replies in that category and design pattern [12].
the corresponding percentage. By recording and checking the
behavior of participants during interviews, we ascertained that The majority (84%) of the participants declared that the
the classification in three discrete categories is sufficiently API types map to domain concepts only “sometimes” (ques-
reliable: we found no case were the results of mapping the tion 1). More specifically, several usability tokens highlight
participant’s answer to one of the categories was ambiguous mismatches between names and underlying concepts. Token
or questionable. The few cases were the given answers were T3 corresponds to 44% of the participants not being familiar
considerably more articulate are mentioned in the analysis. with the acronym CRUD (Create, Read, Update, and Delete)
to represent the basic functions for databases access. As a
Table II lists the usability tokens occurring most frequently consequence, the API designer decided for the subsequent
during the experiments; for each token, the table gives an release an alternative, more general name such as EXECUTOR
identifier, its type (Surprise, Choice, Missed, Incorrect, or instead of CRUD EXECUTOR. Token T4 corresponds to 40%
Unexpected) a brief description, the research questions the of the participants expecting the REPOSITORY class to be
token addresses, and the number and percentage of participant named DATABASE instead, since all the tasks involved rela-
sessions where the token surfaced. tional databases. This is a valid point, even though the name
We focus the discussion on the most significant points; the REPOSITORY was preferred by the designer because the API
complete dataset is presented elsewhere [8, Chap. 6]. also supports serialization operations which do not fit the
database abstraction.
Q YES NO SOMETIMES
1 4 16% 0 0% 21 84% Finally, token T9 highlights a choice developers faced
2 4 16% 21 84% 0 0% when understanding how updates work (Task 3). The confusion
3 7 28% 18 72% 0 0%
4 20 80% 5 20% 0 0%
was due to the fact that querying is not necessary before
5 2 8% 23 92% 0 0% updating to solve Task 3 correctly, because the modified
6 12 48% 13 52% 0 0% PERSON objects are still in local memory from Task 1. In
7 22 88% 3 12% 0 0%
8 17 68% 8 32% 0 0% more general situations, however, querying is necessary, and
9 0 0% 3 12% 22 88% the API documentation reflects this more general scenario
10 4 16% 21 84% 0 0% which guarantees correctness.
11 23 92% 2 8% 0 0%
12 4 16% 1 4% 20 80%
Table I. S UMMARY OF ANSWERS GIVEN DURING THE INTERVIEWS .
B. RQ2: Abstraction
The abstraction level of the API was largely considered
A. RQ1: Understandability appropriate and the functionalities offered were found suitable
The effort required to understand the semantics of API to solve the tasks. Nearly half of the participants, however,
features was considered overall moderate, but a few class occasionally found it useful to peek at some implementation
and method names were found confusing and potentially details in order to more readily understand relations between
misleading. classes. This reveals some weak spots in the API abstraction
which are to be improved.
Over 70% of the participants did not have to become
familiar with many classes to use the API (question 3) nor to Over 90% of the participants did not have to modify the
process information not explicitly part of the API (question 2). library classes to solve the tasks (question 5). Three of the
This suggests that the API documentation is sufficiently self- participants slightly modified some classes by inheriting even
contained, and the API classes are acceptable for standard if that was not necessary. This usage was benign as it did
tasks. not break the abstraction of the API; it was done by some
of the more experienced Eiffel programmers, probably out of
Three of the seven participants who claimed to have learned habit since multiple inheritance is used extensively in Eiffel.
“many classes and dependencies” missed a concrete factory These were the only cases of “unexpected” usability tokens,
class [13] named REPOSITORY FACTORY, which was quite which do not feature in Table II, which only reports frequently
useful to set-up repositories with only few operations. This occurring tokens. This suggests that the API can be largely
is connected to a more general problem of using factories used as intended by the designer.
instead of ordinary constructors to create objects and initialize
them. REPOSITORY FACTORY is also the source of token The majority (80%) of the participants also found the
T6, where 36% of the participants were confused by not API abstraction level appropriate to the tasks (question 4).
having to use class CONNECTION to establish a database Seven participants, however, were surprised that method
connection; in fact, the factory took care of establishing the execute query does not return the query result (token T11).
connection. The most frequently occurring usability token T1 The reason is that it is customary to practice command/query
corresponds to participants who missed the other concrete separation [14] in Eiffel libraries: each method should either
factory CRITERION FACTORY to create criteria (a form of be a function (returning a component of the object state
query useful in Tasks 2 and 5). Since it is still possible to without modifying it) or a command (a procedure modifying
instantiate criteria without using the corresponding factory, all the object state without directly returning a result). Method
participants could successfully complete Tasks 2 and 5 even execute query is a command, which should be followed by
if they had problems with using factories. For API designers, a function call to retrieve its result. Non-Eiffel programmers

5
 ID TOKEN DESCRIPTION RQs # %
T1 M criterion factory 1 14 56%
T2 C predefined criterion 2, 3 13 52%
T3 S what does CRUD mean? 1, 4 11 44%
T4 S expecting database, not repository 1, 2 10 40%
T5 S REPOSITORY cannot execute CRUD operations 1, 2 10 40%
T6 S a connection class cannot be used 1, 2 9 36%
T7 C agent criterion 2, 3 9 36%
T8 C which strings are valid operators? 1, 4 9 36%
T9 C read required before update? 1 8 32%
T10 M default query retrieves all objects 1, 4 7 28%
T11 S expecting method execute query to return result 1, 2 7 28%
T12 S QUERY also contains query result 2, 4 6 24%
T13 I reusing the same QUERY object without resetting 2, 4 6 24%
Table II. U SABILITY TOKENS OCCURRING MOST FREQUENTLY.

may not be familiar with this design principle, which reveals the API has a certain flexibility, and that certain tasks can be
a trade-off between design clarity and practical usability. solved very concisely (using the defaults) or less concisely but
with more control on the individual steps. This was particularly
The fact that 48% of the participants had to understand
true for the query mechanism, which offers some simple
some implementation details to use the API (question 6)
defaults but also more flexible features based on CRITERION
reveals a few significant deficiencies in some abstractions of
classes. On the negative side, choice may also be confusing
the API. Specifically, tokens T5 and T12 point to two critical
or slow down programmers, as evidenced in a couple of the
aspects for several participants. Token T5 refers to the fact that
usability tokens. Token T2 reveals that 52% of the participants
ten participants expected class REPOSITORY to also directly
pondered whether a predefined criterion or a default query was
offer features to access the database, whereas such operations
better suited for Task 2. Token T7 concerns a similar choice
were offered by the CRUD EXECUTOR class. The problem
of how to use criteria with agents (Eiffel’s function objects),
was worsened because REPOSITORY does not mention class
useful in Task 5 but unnecessary in the other tasks. Neither
CRUD EXECUTOR explicitly in the method signatures, and
token highlights flaws in the API, but both suggest possible
hence some participants decided to inspect its implementation
elements of design simplification.
to find references to the other class. Token T12 is connected
to token T11, and prompted a few participants to peek into the
implementation of QUERY to understand how to access result D. RQ4: Learnability
objects. Such difficulties in understanding relations between The learning curve for the API is initially steep, as it
types and in discovering new classes corroborate previous requires to become familiar with a few non-trivial abstractions.
empirical observations along the same lines [11]. After the initial learning phase, however, solving more ad-
vanced tasks becomes relatively simpler, as the learning curve
C. RQ3: Reusability flattens.
The participants agreed that they managed to write concise Over 90% of the participants agreed that they became
client code in an incremental fashion, and that their solutions more efficient after completing the first two “exploratory” tasks
were reusable to solve variants of the problems. They also (question 11). The two participants who disagreed were slowed
agreed that the API offers different ways of implementing down by the choice offered by the CRITERION classes, already
certain functionalities. discussed in relation to RQ3 in Section V-C.
Over 90% of the participants thought that it would have
Whether the code written matches the expectations, and
been easy to modify their code to access different database or
hence developers have a positive feedback that reinforces
to perform different queries (question 10). This is an important
learning, largely depended on the individual backgrounds.
goal which the API seems to achieve satisfactorily. Nearly 90%
Participants not familiar with Eiffel conventions—or famil-
of the participants also found “about right” the amount of code
iar but unappreciative—suggested a simplified design (for
they had to write (question 7). Even the few in disagreement
example, not using command/query separation as discussed
mainly found the keywords and names a bit verbose, but did
in Section V-B). Participants used to work with relational
not express a strong criticism about succinctness.
databases at a lower level of abstraction expected to have
Nearly 70% of the participants positively answered ques- to implement the object-to-relational mapping themselves, and
tion 8, which asks whether it was easy to keep track of progress hence were hesitant to use some of the features of the API that
while solving the tasks. The recurring problem that the other transparently took care of the mapping. These are significant
30% experienced was that they had no simple programmatic examples of the trade-offs between abstraction and efficiency
way to clean up the database after failed attempts and restore which populate the rich design space of persistence APIs.
it to the original state to try again. This was more a deficiency
Independent of this, three usability tokens point instead
of the experimental setup than a flaw of the API, even if
to usability issues which negatively impact learnability and
quick trials are something that could often be useful to support
other aspects. The three tokens originate in incomplete or
programmers learning an API.
inaccurate documentation (in terms of comment and contracts,
The answers to question 9 show that the API provides see Section IV-A). Token T10 shows that seven participants
alternatives to solve certain tasks. This carries both a positive missed the fact that a QUERY object returns all objects of its
and a negative connotation. On the positive side, it shows that generic parameter type “by default”, that is when created with

6
the argumentless constructor. Indeed, this is not clear from the on the relationships between types; it is thus confusing when
constructor’s comments and could only be surmised indirectly these are not crystal-clear from reading the documentation.
or by looking at the constructor’s implementation. Token T8 Instantiating classes using constructors is also almost second-
points to a deficiency in the documentation of a constructor nature to object-oriented programmers; when factories should
of the CRITERION class: while its precondition imposes a be used instead, the API design must be tailored to emphasize
constraint on its string argument, the semantic of the constraint this exceptional usage.3 None of the experienced developers,
is obscure as it involves how strings represent operators (e.g., however, reported difficulties in discovering relationships be-
“and” rather than “&&” for logical conjunction). Finally, token tween types. Since the same group also declared that they
T13 reveals that six participants incorrectly used the same hardly needed to look at the implementation to find their way
QUERY object multiple times without calling a reset method through the API, this suggests that experienced programmers
after each usage. This is indeed something not adequately have enough flexibility to interpret non-plain choices of API
documented, and a good example of the kinds of issues design, whereas the novices need significantly more support.
usability testing may find.
Determining the outcome of method calls is another issue
mentioned in other studies of API usability [21]. Developers
VI. D ISCUSSION
tend to rely on a method’s return type to access its result;
We now summarize the overall findings of our study in when the method is a procedure not returning anything, they
more general terms than Section V. We organize the discussion incur a cognitive overhead. As we discussed in Section V-B,
in three parts: Section VI-A targets issues with API usability the Eiffel design style involves the command/query separation,
that were discovered in previous work and replicated in our which caused issues with determining the outcome of method
study; Section VI-B targets issues that emerged in previous calls in our study. This shows a conflict between design clarity
work but were not critical in our study; Section VI-C discusses and practical usability.
new findings and lessons that specifically emerged in our study.
In the following discussion, we also report on the usability
issues that are significantly affected by the previous experience B. Potential Issues Not Critical
of developers. We partitioned the interview answers into two
groups: 11 “experienced” participants, whose years of expe- A positive note of our study is the infrequent occurrence of
rience with object-oriented programming is greater than the usability tokens characterizing incorrect or unexpected usage
median; and the other 14 “novice” participants with below- of the API (see Section III-B and Table II). This suggests that,
median experience. In the reminder, we point out the few even if programmers may have been slowed down by other
questions whose answers look qualitatively different in the deficiencies of the API, the design normally avoids at least the
experienced and novice groups. most obviously incorrect usages and makes it hardly necessary
to override the designer’s intentions. This is one way to resolve
the friction between correctness and practical usability.
A. Confirmation of Previous Issues
A more specific issue emerged regarding argumentless
Associating API feature names to functionalities is a prob-
constructors. Argumentless constructors are used with (but not
lematic issue because it is hard to select names that conform to
limited to) a create-set-call style, whereas constructors with
the heterogeneous jargon of programmers and are descriptive
arguments are used with a create-call style. Previous work
but not verbose. Even the basic object-oriented terminology
suggests [22] that argumentless constructors are preferred, that
may vary from language to language: for example, Eiffel calls
is considered easier to use, to constructors with arguments.
“routine” [14] what normally is a “method” in Java [15] and
In our study, however, the participants had no particular
a “member function” in C++ [16]. Another difference is in
difficulties with using constructors with arguments, which are
the naming convention for features, such as using the prefix
in fact extensively present in the ABEL API. On the contrary,
“is” for functions returning Booleans—such as in is empty.
we had one case of argumentless constructor (of class QUERY
Specific to the persistence domain targeted in our study, we
, discussed at the end of Section V-D) which was found
discussed the problem with the class name REPOSITORY
confusing because its documentation did not spell out what
vs. the more specific one DATABASE, and the unfamiliarity
the default behavior was. More generally, choosing argument-
of several programmers with the acronym CRUD which is,
less vs. with-argument constructors also exercises a friction
however, popular in the database community [17]. Our study
between correctness and practical usability. Eiffel classes use
reinforced the lessons learned by others [18], [19], [20]: name
invariants to characterize valid object states; constructors must
API features consistently and use common names that are still
return objects satisfying their invariants, which then every
descriptive and not vague.
method must preserve. To take advantage of this guard against
Another issue which was found problematic by others [11], incorrect behavior (particularly useful for consistent object
as well as in our study, is discovering relationships between storage [8]), constructors may require arguments to correctly
types. This issue typically emerged in connection to known initialize objects without relying on successive method calls
usability difficulties [12] with the concrete factory pattern to which programmers may forget. Hence, such a stricter design
create objects without calling constructors. Specifically, we style comes at a cost but also brings tangible benefits.
often detected a clash between the expectations of developers
and the usage intended by the API developer. Given the central 3 Note that this issue applies only to the concrete factory pattern; abstract
role of classes and types in object-oriented programming, factories export abstract types, which cannot be directly instantiated using
developers are used to ground their understanding of the API constructors [13].

7
C. Other Lessons as former students, former colleagues, or their acquaintances);
none of them, however, knew the author’s work on ABEL
A recurring lesson emerging from our study—and an un- or participated in any way to the design and preparation of
surprising one at that—is the critical importance of having ac- the study. The participants’ programming background was
curate, unambiguous, and self-contained documentation [23]. sufficiently heterogeneous (Section IV-C) to guarantee that the
While previous work suggests that API documentation should study is representative of programmers with quite different
ideally include code snippets and tutorials [24], the participants experience. We do not believe that the lack of complete
of our study faltered whenever the API documentation given strangers has influenced the outcome of the study in any
to them was imprecise or incomplete. The lesson should be significant way; it may have even been conducive to reducing
familiar, but is worth emphasizing: bad documentation is a performance stress, and hence removed a potential factor of
nonstarter. disturbance.
Somewhat related to the problem of documentation is
The performance of the study participants might have
another dimension of the API design space which surfaced
been different, and possibly better representative of usability
in relation to the usability token T8 (discussed at the end of
in standard conditions, if they had been allowed to access
Section V-D). Developers had trouble understanding how to in-
external documentation (for example, about Eiffel or the IDE)
stantiate an argument of string type with a valid representation
and to search for code snippets on the Internet. However,
of logic operators. While this is also an instance of incomplete
we introduced this restriction to focus the evaluation on the
documentation, the operators passed as argument could have
features of the API and its official documentation; evaluating
been represented as types instead of constant strings. This
the effects of browsing the web on programmer performance
would have required some additional effort to discover the
is a problem in its own right [21], whose consequences would
new types, but would also have removed the ambiguity and
have been difficult to control for.
made it possible to check the actual argument at compile time
(as a type constraint) rather than relying on the weaker checks Collecting answers to questions through an interview car-
done at runtime. ries a risk of “interviewer effect”, where the interviewer biases
the answers by giving involuntary subconscious clues [25]. A
A final lesson emerged from our study is the role of
similar threat is involved in the proctor sitting through the
choice, also discussed in Section V-C. When the API provides
programming sessions, where he occasionally answered simple
different ways to solve a task, programmers have more flex-
questions by the participants (Section IV-D). We minimized
ibility but also more difficulties to fully understand the API
the impact of these two threats as follows. First, the proctor
design. When answering questions during the interview, the
only answered generic questions about the Eiffel language or
more experienced developers tended to emphasize the positive
the IDE used during the study, giving succinct verbal answers
connotation of choice, whereas the less experienced ones often
and avoiding as much as possible any reference to the specific
considered choice as a negative feature. It is clear, however,
functionalities of the API evaluated in the study, or to the
that, even for experienced developers, choice is positive only
tasks to be solved. Second, the interviews was structured
when the options are really complementary, rather than being
and consisted only of twelve predefined questions, which the
just unnecessary complications of the design.
interviewer read from a printout. Third, all sessions were
recorded; after the experiments we replayed the recordings
VII. T HREATS TO VALIDITY checking that the intended protocol was followed. During the
We discuss the main threats to validity and what measures replaying, we noted down the interactions between the proctor
we deployed to minimize their impact. and the participants. As required by the protocol, all inter-
actions were only about Eiffel syntax details or EiffelStudio
IDE functionalities. To better identify potential sources of such
A. Construct Validity
requests, we measured correlations between the participants’
A multifaceted feature such as usability can elude attempts background data (Section IV-C) and the number of requests for
to get objective measures. In our study, we addressed such po- clarification, using Kendall’s τ . We found a significant negative
tential threat to construct validity with a careful design which correlation with Eiffel experience (p ' 0.003, τ ' −0.59),
follows well-established protocols (the cognitive dimension and a significant positive correlation with time to complete the
framework and the thinking-aloud protocol for data collection). tasks (p ' 0.0002, τ ' 0.44). This gives independent support
As we mention in Sections IV and VIII, these protocols to the claim that this threat did not have a significant impact:
have already been successfully used for usability studies, the more Eiffel experience a participant has—and the faster
which vouches for their soundness. Additionally, we tried to the participant is—the fewer clarifications he or she requests.
complement the inherent limitations of structured interviews
by also analyzing the recordings of the experiments according C. External Validity
to a few “usability tokens” (Section III-B). Combining the
explicit answers given during interviews with the implicit The main threats to the generalizability of the findings of
tokens emerged during the study gives a richer set of data our study come from the fact that it targeted a single API
which help reduce the impact of inaccurate measurements. and a single programming language. In fact, some of the
issues emerging during the study are somewhat specific to the
B. Internal Validity application domain or to the Eiffel design style. However, the
discussion in Section VI also shows that the study confirmed
A possible bias exists in the selection of participants to several issues that emerged in other API usability studies tar-
the study, most of whom were known to the first author (such geting different languages and different domains. This suggests

8
that the gist of our findings are also applicable to different query, where almost one fourth of the participants to our study
contexts. expected the method to also explicitly return the result of the
query itself.
VIII. R ELATED W ORK Another issue raised in [21] as well as in ours and in other
APIs are a fundamental interface for the interactions studies regards the fact that programmers rely on feature names
between programmers and computers [26]. Thus, API us- to make educated guessues about the features’ semantics,
ability impacts software development quality [27]: usable especially when the documentation is difficult to access or
APIs are more intuitive, require less documentation browsing, incomplete. The task of selecting the appropriate abstractions
and encourage reuse, thus increasing developers’ productivity. creates then potential selection barriers [27]. For example, an
Conversely, APIs that are hard to use reduce programmer empirical study of the names used in Java API class names and
productivity and quality of the final product, as shown, for JavaDoc documentation [33] shows that the most frequently
example, by measuring requests for technical support [28]. used words are “Exception”, “UI”, “Helper”, “Type”, “Event”,
Despite the critical impact that API usability seems to have, and “Factory”. Programmers who become familiar with such
there are only a few rigorous studies of API usability in the a terminology may then experience a worse usability when
literature. This section briefly reviews them and highlights the working with frameworks sticking to different jargon, as we
connections with the rest of the paper. discuss in Section VI-A.
An early attempt at investigating API usability analyzed Yet another issue raised in [21] and as well [11] and
the role of examples to help design understandable APIs [29]. confirmed in our study regards the recurring difficulties of
According to the study, APIs designed around examples are discovering relationships between API types. [11] points out
simple to use for programming tasks that follow closely the that types are difficult to discover when they are not mentioned
original examples. Conversely, when developers need to use as attributes, local variables, arguments, or even in comments.
the APIs in scenarios significantly different from the original In our experiments, the issue of discovering types emerged
examples, they may prove hard to use or inadequately de- mostly in connection with another usability issues reported in
signed. In our empirical study, we did not specifically consider related work [12]: effectively using the concrete factory design
the role of examples but focused on the evaluation of the pattern [13]. Combining evidence from different sources, it is
usability of a self-documented API solely based on its features’ remarkable that the word “Factory” is frequently used to label
signatures, comments, and contracts. API features [33] and the concrete factory pattern is one of
the most used; and yet it is a source of common usability
Some of the difficult issues developers face when program- problems.
ming using APIs are understanding the rationale of design
decisions and answering questions about APIs that are not cov- The thinking-aloud protocol [4], used in our experiments
ered by documentation or tool support [30]. These observations for data collection, is a widely used method in usability testing.
suggested to restrict our study participants to access only the We applied it in a less strict variant [5] suitable for usability
official API documentation plus take advantage of IDE support, tasks where not only the human subjects but also the “product”
so as to focus the evaluation on the actual API design rather being used (in our case, an API) should be inspected and
than on external factors such as generic documentation and evaluated.
other programmers’ suggestions. Methods to improve API usability are a natural complement
Software development practices for API design are typi- to techniques to assess the usability (the object of the discus-
cally specific to one programming language, such as C# [18], sion so far). One approach [34] consists of extracting additional
Java [19], or C++ [20]. [31] suggests to generalize the tech- information from the API documentation—such as usage rules
niques used to evaluate specific APIs in order to investigate and special cases—and displaying such information within the
the impact on usability of different design choices. To this IDE whenever the information is relevant to the current activ-
end, [31] uses APIs created ad hoc to compare the usability ity. Another approach [35] suggests that documentation writers
of specific features and to evaluate the client code developers should be involved early on during API design stages; applying
would write in an abstract setting. Related work [32] attempted text analysis and other documentation writing techniques can
to reconstruct the relations between dimensions in the space help to write APIs with improved usability.
of API design and their impact on usability, and in particular
outlined the involved trade-offs. Our study also evidenced IX. C ONCLUSIONS
some of these trade-offs, but based on an “in vivo” empirical
study based on an actual API. This paper presented the design of an empirical study to
assess the usability of APIs and, more specifically, to find
A recent study about the performance of developers facing issues that may hamper it. The design is based on the idea
unfamiliar APIs [21] highlighted several important issues that of comparing the expectations of programmers to their actual
were also confirmed in our study. In particular, develop- performance on programming tasks requiring to write client
ers often have difficulties associating API feature names to applications using API features. To provide a richer charac-
functionalities, discovering relationships between API types terization of their behavior, the study collected programmers’
(i.e., classes), and determining the outcome of method calls feedback both directly—through structured interviews with
for methods that do not explicitly return values. Specifically, questions based on the cognitive dimensions framework—
developers tend to expect methods to return values that notify and indirectly—by observing recordings of the performances
about the success or failure of a method call. We observed and classifying episodes into “tokens” revealing of usability
a similar problem in the case of a method that executes a issues. We executed the study with 25 programmers (students,

9
researchers, and professionals) working on a persistence library [21] E. Duala-Ekoko and M. P. Robillard, “Asking and answering questions
written in Eiffel. The study confirmed usability issues that about unfamiliar APIs: An exploratory study,” in ICSE, 2012, pp. 266–
emerged in related studies, such as the difficulty of assigning 276.
names to API features and of discovering relations between [22] J. Stylos and S. Clarke, “Usability implications of requiring parameters
in objects’ constructors,” in 29th International Conference in Software
types (i.e., classes) of the API. It also found how several Engineering, 2007, pp. 529–539.
usability flaws are ultimately due to incomplete or unclear doc- [23] D. L. Parnas, “Precise documentation: The key to better software,” in
umentation; and revealed that flexible features are appreciated The Future of Software Engineering. Springer, 2011, pp. 125–148.
by experienced programmers but may disorient novices. [24] M. P. Robillard, “What makes APIs hard to learn? Answers from
developers,” IEEE Software, vol. 26, no. 6, pp. 26–34, 2009.
ACKNOWLEDGMENTS [25] F. J. Fowler and T. W. Mangione, Standardized Survey Interviewing:
Minimizing Interviewer-Related Error. Sage Publications Inc., 1989.
We would like to thank Martin Robillard for his useful [26] K. Arnold, “Programmers are people, too,” Queue, vol. 3, no. 5, pp.
comments on preliminary versions of this content. Many 54–59, Jun. 2005.
thanks also to the participants to the usability study. [27] A. Ko, B. Myers, and H. H. Aung, “Six learning barriers in end-user
programming systems,” in Proceedings of Visual Languages and Human
Centric Computing, 2004, pp. 199–206.
R EFERENCES [28] M. Henning, “API design matters,” Commun. ACM, vol. 52, no. 5, pp.
[1] A. F. Blackwell, C. Britton, A. L. Cox, T. R. G. Green, C. A. Gurr, 46–56, May 2009.
G. F. Kadoda, M. Kutar, M. Loomes, C. L. Nehaniv, M. Petre, C. Roast, [29] S. McLellan, A. Roesler, J. Tempest, and C. Spinuzzi, “Building more
C. Roe, A. Wong, and R. M. Young, “Cognitive dimensions of notations: usable APIs,” Software, IEEE, vol. 15, no. 3, pp. 78–86, May/Jun.
Design tools for cognitive technology,” in Cognitive Technology, 2001, [30] A. J. Ko, R. DeLine, and G. Venolia, “Information needs in collocated
pp. 325–341. software development teams,” in 29th International Conference on
[2] A. C. Kay, “Quoted in: The wiki way: Quick collaboration on the web Software Engineering, 2007, pp. 344–353.
by B. Leuf and W. Cunningham,” [Link] [31] J. Stylos, “Informing API design through usability studies of API design
Kay, 2001. choices: A research abstract,” in Visual Languages and Human-Centric
[3] S. Clarke, “Measuring API usability,” [Link] Computing, VL/HCC 2006. IEEE Symposium on, 2006, pp. 246–247.
windows/measuring-api-usability/184405654, 2004. [32] J. Stylos and B. A. Myers, “Mapping the space of API design decisions,”
[4] K. A. Ericsson and H. A. Simon, Protocol Analysis: Verbal Reports as in VL/HCC, 2007, pp. 50–60.
Data. MIT Press, 1984. [33] C. Anslow, J. Noble, S. Marshall, and E. Tempero, “Visualizing the
[5] M. T. Boren and J. Ramey, “Thinking aloud: reconciling theory and word structure of Java class names,” in Companion to OOPSLA ’08.
practice,” IEEE Transactions on Professional Communication, vol. 43, ACM, 2008, pp. 777–778.
no. 3, pp. 261–278, 2000. [34] U. Dekel and J. D. Herbsleb, “Improving API documentation usability
[6] S. Clarke, “Evaluating a new programming language,” in 13th Workshop with knowledge pushing,” in Proceedings of the 31st International
of the Psychology of Programming Interest Group, 2001, pp. 275–289. Conference on Software Engineering, ser. ICSE. IEEE Computer
[7] J. Dagit, J. Lawrance, C. Neumann, M. M. Burnett, R. A. Metoyer, and Society, 2009, pp. 320–330.
S. Adams, “Using cognitive dimensions: Advice from the trenches,” J. [35] R. B. Watson, “Improving software API usability through text analysis:
Vis. Lang. Comput., vol. 17, no. 4, pp. 302–327, 2006. A case study,” in IEEE IPCC, 2009.
[8] M. Piccioni, “A seamless framework for object-oriented persistence in
presence of class schema evolution,” Ph.D. dissertation, ETH Zurich,
2012.
[9] “Spring framework data access,” [Link]
data-access, last visited: 15.03.2013.
[10] “Hibernate relational persistence for Java and .NET,” [Link]
[Link], last visited: 15.03.2013.
[11] J. Stylos and B. A. Myers, “The implications of method placement on
API learnability,” in 16th International Symposium on Foundations of
Software Engineering, 2008, pp. 105–112.
[12] B. Ellis, J. Stylos, and B. A. Myers, “The factory pattern in API design:
A usability evaluation,” in ICSE, 2007, pp. 302–312.
[13] E. Gamma, R. Helm, R. Johnson, and J. Vlissides, Design Patterns:
Elements of Reusable Object-Oriented Software. Addison-Wesley,
1994.
[14] B. Meyer, Object Oriented Software Construction, 2nd ed. Prentice
Hall PTR, 1997.
[15] J. Gosling, B. Joy, and G. L. S. J. and Gilad Bracha and Alex Buckley,
The Java Language Specification, Java SE 7 Edition (Java Series).
Addison-Wesley Professional, 2013.
[16] B. Stroustroup, Programming: Principles and Practice Using C++.
Addison-Wesley Professional, 2008.
[17] J. Martin, Managing the Data-base Environment. Pearson Education
Canada, 1983.
[18] K. Cwalina and B. Abrams, Framework Design Guidelines: Conven-
tions, Idioms, and Patterns for Reusable .NET Libraries (2nd Edition).
Addison-Wesley Professional, 2008.
[19] J. Tulach, Practical API Design: Confessions of a Java Framework
Architect. Apress, 2012.
[20] M. Reddy, API Design for C++. Morgan Kaufmann, 2011.

10