1

Understanding and Improving

Regression Test Selection in Continuous Integration
August Shi, Peiyuan Zhao, Darko Marinov
Department of Computer Science
University of Illinois at Urbana-Champaign, USA
{awshi2,pzhao12,marinov}@[Link]

Abstract—Developers rely on regression testing in their contin- technique tracks the dependencies among the tests and code
uous integration (CI) environment to find changes that introduce entities (e.g., modules, classes, or methods) and selects to run
regression faults. While regression testing is widely practiced, it only the tests whose (direct or indirect) dependencies changed.
can be costly. Regression test selection (RTS) reduces the cost of
regression testing by not running the tests that are unaffected by RTS aims to run fewer tests, speeding up regression testing.
the changes. Industry has adopted module-level RTS for their CI Researchers have proposed many RTS techniques that perform
environment, while researchers have proposed class-level RTS. selection at different granularity levels of dependencies [14],
In this paper, we compare module- and class-level RTS tech- [15], [18], [23], [26], [27], [28], [32], [34], [38]. Researchers
niques in a cloud-based CI environment, Travis. We also develop have also reported RTS to be effective in experiments, mea-
and evaluate a hybrid RTS technique that combines aspects of
the module- and class-level RTS techniques. We evaluate all sured by the percentage of tests selected and testing time
the techniques on real Travis builds. We find that the RTS saved. Large companies have adopted RTS to speed up their
techniques do save testing time compared to running all tests regression testing [12], [13], [17], [19], [25]. For industry,
(RetestAll), but the percentage of time for a full build using important metrics are the overall build time and the quality
RTS (76.0%) is not as low as found in previous work, due to of the test outcomes (i.e., if test failures reveal real faults).
the extra overhead in a cloud-based CI environment. Moreover,
we inspect test failures from RetestAll builds, and although we While both industry and research use and study RTS tech-
find that RTS techniques can miss to select failed tests, these test niques, there is a gap in the granularity level of RTS they
failures are almost all flaky test failures. As such, RTS techniques use. A typical (object-oriented) software project is organized
provide additional value in helping developers avoid wasting time hierarchically into modules that contain classes that contain
debugging failures not related to the recent code changes. Overall,
methods that contain statements; there can be also dependen-
our results show that RTS can be beneficial for the developers
in the CI environment, and RTS not only saves time but also cies among projects. In industry, RTS has progressed from
avoids misleading developers by flaky test failures. coarser- to finer-grain dependencies, from running all tests
Index Terms—regression test selection, continuous integration,
(RetestAll) to tracking dependencies among project modules1 .
flaky tests Modern systems in industry [12], [13], [17], [19] commonly
use module-level dependencies, track changes made to project
modules, select a subset of modules that are affected by
I. I NTRODUCTION the changes, and then run all tests within the selected mod-
Developers rely on regression testing to quickly detect ules [13], [25]. In research, the progress has been from finer-
regression faults introduced by their code changes. Nowadays, to coarser-grain dependencies, from traditional work using
regression testing is commonly performed in continuous inte- statements [34] to using methods [38] to the most recent work
gration (CI): after every push to the repository, a CI server, reporting class-level dependencies to be more effective than
typically in the cloud, builds and tests the code [20]. While finer-grain dependencies [15], [23], [37].
regression testing is important and widely-practiced, it has two We aim to understand how module- and class-level RTS
major problems. First, it can be time-consuming due to running techniques compare in a real CI environment: should everyone
many tests after every change and having frequent changes, adopt module-level RTS used by large companies, should
e.g., as reported by Google [25]. Also, running regression they adopt class-level RTS proposed by researchers, or is the
testing on CI servers in the cloud incurs a monetary cost ideal trade-off in the middle? While module-level RTS has a
for the cloud resources, e.g., as reported by Microsoft [19]. very small overhead to analyze what modules are affected by
Second, regression testing suffers from flaky tests [7], [24], the changes, the coarse-grained dependency tracking and test
[36], which can pass or fail non-deterministically regardless selection (all tests within affected modules) can select more
of the code changes, e.g., as reported by Facebook [17]. With tests than class-level RTS selects (only the affected test classes,
flaky tests, a developer cannot trust a new test failure to not all tests, within affected modules). While class-level RTS
indicate a regression fault in the recent code changes that the can select fewer tests, it has two issues stemming from tracking
developer should debug. dependencies (only) on classes: (1) it can potentially miss to
Regression test selection (RTS) can reduce the costs of
regression testing. RTS runs only a subset of the regression test 1 We use the term “module” following the Maven build system for Java, but
suite—the tests that are affected by the changes [34]. An RTS other regression testing, CI, or build systems use other terms, e.g., “target”.

©2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing
this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this
work in other works.
 2

select affected tests, e.g., due to changes to non-source code and systematically inspect failures. Some test failures during
files like configuration files, whereas module-level RTS finds regression testing, namely those from flaky tests, are undesir-
such changes to affect entire modules and then selects all tests able [7], [17], [24], [36]. Because RTS may miss test failures
within such modules; (2) it requires extra analysis time to from RetestAll (in our experiments, it missed 39 failures), we
determine the affected test classes, compared to the time to analyzed whether RTS missed desirable or undesirable test
determine only the affected modules, so the overall time for failures. We find that almost all the test failures (38 out of 39)
class-level RTS can be higher despite selecting fewer tests. from RetestAll that any RTS technique missed are undesirable,
To evaluate module- and class-level RTS techniques in a CI flaky test failures.
environment, we would ideally compare techniques in general While RTS techniques are not explicitly designed to avoid
and not specific tools. Some metrics, such as the number of flaky test failures, we empirically find that RTS is highly
selected tests, are mostly determined by the technique, but a beneficial for avoiding flaky test failures, a benefit of RTS
key metric that developers care about—the total build time— not previously reported [14], [15], [18], [23], [26], [27], [28],
is determined by the tool. We thus compare specific tools and [32], [34], [38]. We also apply the RTS techniques on 19 failed
carefully analyze results to draw general conclusions about pull request jobs that are not flaky (confirmed through reruns)
techniques. We start with GIB [2], a module-level RTS tool, from five of our projects. We find that the RTS techniques do
and Ekstazi [15], a class-level RTS tool. not miss any failed test in these 19 jobs, even though class-
We also implement a new RTS tool, GIBstazi, that simply level RTS could miss to select affected tests due to non-source
combines both module- and class-level RTS. GIBstazi first code changes.
uses GIB to quickly select what modules are affected by the In summary, the contributions of this paper are:
changes, and then only on those modules applies Ekstazi to • Hybrid RTS Technique: We develop a simple, hybrid
select affected tests. If a change is in a non-source-code file module- and class-level RTS technique.
that is not tracked by class-level RTS, GIBstazi defaults to GIB • Empirical Evaluation: We empirically evaluate the
behavior and selects all the tests within the affected modules. module-level, class-level, and hybrid RTS techniques in
Moreover, after our preliminary experiments show that GIB CI; no prior work compared such RTS techniques in CI.
out-of-the-box would almost always selects all modules based • Failure Analysis: We are the first to analyze test failures
on the changes, we make enhancements to better filter changes from RetestAll runs in the context of RTS. We find almost
that tests should not be affected by. Our subsequent evaluation all failures are due to flaky tests, showing how RTS
of GIB uses these enhancements as the default configuration. techniques by chance mitigate the flaky test problem.
By combining GIB and Ekstazi, GIBstazi aims to select and From our results, we recommend developers to use the
run fewer tests than GIB, leading to faster testing, but not hybrid GIBstazi RTS technique, because it provides the best
necessarily faster than Ekstazi. However, because GIBstazi trade-off: it works faster than GIB and only somewhat slower
defaults to GIB behavior due to non-source-code changes, than Ekstazi, but it is safer than Ekstazi. Moreover, our
GIBstazi can be safer (i.e., not miss to select some affected analysis of failed tests reveals that RTS techniques provide
tests) than Ekstazi. additional, although not directly targeted, benefits in helping
To evaluate real build times that developers would see developers avoid flaky test failures.
in practice for the three RTS techniques, we utilize Travis
CI [5], the most popular cloud-based CI service for open-
source projects [20]. We evaluate the techniques on a diverse II. C ONTINUOUS I NTEGRATION (CI) BACKGROUND
set of open-source Java projects already configured to use We describe how developers use CI systems to build and test
Travis; whenever the developer pushes some change, Travis their code, with an emphasis on CI systems in the cloud. We
triggers one (or more) build job(s). We replay the build jobs use Travis [5] as an example to help illustrate the concepts.
of each project by running all the tests (RetestAll) and using Travis is widely used [21] and integrates well with projects
each of the three RTS techniques. For each technique run on hosted on GitHub, the most popular platform for open-source
each job, we collect three metrics relative to RetestAll: the projects [6], [9], [10], [11]. We use Travis in our evaluation
percentage of tests selected, the percentage of time to run the and introduce some Travis terminology here.
selected tests, and the percentage of time overall to build the When a developer pushes a commit to a repository, the push
job. On the 22 projects and the 935 build jobs we replayed, triggers a build on the CI servers to compile and test the code
we find that all three RTS techniques on average save time for that commit. Developers can specify the exact commands
over RetestAll on Travis: GIB, Ekstazi, and GIBstazi take to be run on the CI server; for Travis, these commands are
79.7% 76.0%, and 77.4%, respectively, of the total RetestAll in the .[Link] file. Travis also allows developers to
build time. These percentages for total build time in CI are configure the build to run multiple jobs. Each job is configured
higher than previously reported (60%–70%) for local, non- with different environment variables or even different build
CI environments [15], [23], [37]. Moreover, these percentages commands; all jobs from one build run for the same commit.
are much higher than suggested by just the percentage of tests Each job is scheduled on its own clean virtual machine in the
selected, e.g., 30.6% for Ekstazi. cloud. Hence, we focus our evaluation per job.
To understand the effectiveness of RTS techniques with When a job starts on a remote CI server in the cloud,
respect to test failures that happen during RetestAll, we collect artifacts from prior jobs will not be on the machine. As such,
the test outcomes (passes and failures) for each technique CI needs to either recompile the project from scratch before
 3

running any tests or rely on some way of saving compiled for incremental building, but as long as the Maven command
artifacts from prior jobs in a persistent datastore and copying includes some testing, GIB effectively performs module-level
them over for the new job. Most Travis setups recompile the RTS by running all the tests within only affected modules. As
code from scratch. A Travis job is divided into multiple phases module-level RTS is conceptually rather simple, we believe
that are executed sequentially. If a phase fails, the job fails GIB is a representative tool.
early, and the later phases are not run. The install phase Enhancing GIB In our preliminary experiments, we use GIB
typically compiles the project code but does not run the tests. out-of-the-box on 423 commits from open-source projects
For example, the default command in the install phase for from our later evaluation. We run GIB using the command mvn
building a Maven project is mvn install -DskipTests=true validate, which just checks the structure of the modules in
-[Link]=true -B -V, which purposely skips the Maven project without compiling or running tests, to check
tests, but installs all the compiled artifacts. On Travis, it how many modules GIB selects based on the changes. We find
is possible to save certain artifacts across jobs, specified in that in 65% of these commits GIB selects all the modules in
the cache section of the .[Link]. Such artifacts are the project. Overall, GIB selects over 70% of all modules in
copied onto the virtual machine before the job starts, and all the commits. As such, we add two key enhancements to
they are uploaded onto a separate, persistent server after the default GIB to improve it for RTS. Our subsequent evaluation
before_cache phase, which occurs near the end of the job. uses GIB with these enhancements as the default.
After code has been properly compiled, CI can start running Our enhancements (1) configure GIB in a reasonable way
tests. In Travis, tests usually run in the script phase, which for RTS and (2) extend GIB with a new feature. GIB uses
is executed after (but not immediately after) the install Git to determine what files changed, but not all changed files
phase. The default command in the script phase for Maven affect Java test outcomes. For example, if the only change is
projects is mvn test -B. Since the script phase occurs after to the top-level README file, GIB computes that a change
the install phase that should compile all necessary artifacts, to a file in the root of the project affects all modules, thereby
the script phase commands need not recompile code. running all tests, even though changing a README file is
Finally, CI gives feedback to the developer about the status unlikely to actually affect a test outcome. To avoid such over-
of each job after it completes. Travis assigns a status to each selection of tests, we configure GIB to exclude certain files,
completed job: pass means all phases ran successfully, fail using the following regex based on file names we have seen
means the script phase ran unsuccessfully, and error means commonly changing but are unlikely to affect test outcomes:
some other phase ran unsuccessfully (usually the install \.apt$|\.txt$|\.md$|\.html$|\.rst$|\.scss$|\.css$
phase, suggesting compilation failed). |\.png$|\.py$|\.jpg$|\.jpeg$|\.git.*|NOTICE$
|README$|README\.|[Link]|[Link]|[Link]
III. R EGRESSION T EST S ELECTION (RTS) T ECHNIQUES
We describe the three RTS techniques that we evaluate. Our regex works well generally, but developers using GIB
They track dependencies and perform selection at different should tailor the regex further for their projects.
granularity levels: module-level, class-level, and a hybrid While our regex filters out many cases where file changes
module- and class-level. Prior work found RTS at the class- would lead to unnecessary test runs, our initial experiments
level to outperform finer granularity such as method-level [15], found many other cases where changes to the Maven [Link]
[23], [37], so we do not evaluate finer-grained RTS techniques. files lead to a large number of test runs. In general, changes to
[Link] can affect tests, e.g., by changing a project’s library
dependencies [16]. However, we observed many changes to
A. Module-level RTS [Link] files that did not change the dependencies, e.g., some
Developers use modules to group related project parts, and changes simply update the project’s own version number,
module-level RTS detects changes at the level of modules. which should not affect test outcome. We extended GIB to
When module-level RTS detects some changed module(s), it check if any dependency of a module within the project
finds all the affected modules by computing the transitive changed between runs. Our enhancement stores in a separate
closure of the changed modules in the module dependency file, classpathfile, the names of all the dependencies for
graph specified by the developers. Module-level RTS then runs each module (but no version number if a dependency is a mod-
all the tests within all affected modules. ule in the current project). Before each run, if classpathfile
We use Gitflow Incremental Builder (GIB) as a module- exists, GIB compares the contents of the file with the depen-
level RTS tool [2]. GIB is a Maven extension that can dencies of each module to see if any dependencies changed.
perform module-level RTS. GIB relies on Git to determine If not, GIB ignores changes to [Link].
the code changes. Given two Git commits, GIB diffs the two
commits to determine what files changed. GIB then maps the
changed files back to the Maven modules to determine changed B. Class-level RTS
modules. GIB finds the affected modules through the module Class-level RTS tracks dependencies at the class level. First,
dependency graph obtained from parsing the [Link] Maven it maps each test class to the classes that the test depends on.
build files provided by the developers. GIB then runs the Then, if a class changes, class-level RTS selects all test classes
provided Maven command (e.g., mvn install or mvn test) that depend on the changed class. The dependencies of each
only on the affected modules. GIB was originally intended test class can be computed dynamically [15] or statically [23].
 4

We use Ekstazi [14], [15] as a class-level RTS tool. Ekstazi TABLE I: Filtering of projects for our evaluation
is a Maven plugin that performs dynamic class-level RTS.
Total starting Maven projects from GitHub 1000
Ekstazi instruments the code under test to obtain which classes Multi-module Maven projects on Travis 105
each test2 depends on. Ekstazi also tracks checksum values Projects whose build takes longer than 10 minutes 46
for each .class file (compiled from a source Java file). After Projects with tests and replayable with RTS tools 22
a project change, Ekstazi first waits for Maven to compile
source files to .class files, then uses the stored checksums to
determine which classes actually changed, and finally selects the top 1000 popular Java projects ranked by stars, and then
the tests that depend on the changed classes as per the we filter to obtain only multi-module Maven projects. Finally,
stored dependency mapping. Both the mapping from tests we filter for projects that use Travis, resulting in 105 projects.
to dependencies and the class checksums are stored within Further, we want to evaluate on projects whose builds are
.ekstazi directories, one for each module. sufficiently long-running such that a developer may want to
use RTS in the first place. For each of the 105 projects, we
C. Hybrid Module- and Class-level RTS query Travis for the latest 20 builds, average the build times,
and select projects that took on average longer than 10 minutes
Module-level RTS can select many more tests than class-
to build, resulting in 46 projects. The build times reported by
level RTS, because module-level RTS selects all tests within
Travis represent the overall time the project takes to build
all affected modules, even if many such tests may not be
and not just the times for testing. In particular, some of these
affected by the changes. We propose a hybrid module- and
projects only compile code on Travis and intentionally skip
class-level RTS that simply combines elements of both. It first
tests. Since we are evaluating RTS techniques, we want the
uses module-level analysis to determine the affected modules,
projects that run at least some tests during the script phase on
and then uses class-level analysis on the affected modules to
Travis. From the 46 projects, we keep the projects that run tests
select individual tests. However, if a change is to a non-source-
on Travis, and we further keep only the projects that can run
code file (not specified in the exclude regex), e.g., a .json file
with all three RTS tools (e.g., GIB requires Java 8), resulting
that may be a test input, the hybrid technique defaults back to
finally in 22 projects. Some of the projects have testing time
module-level RTS and selects all tests in the affected modules,
shorter than 10 minutes, and the average testing time per job
being safer than class-level RTS (which does not track changes
is 9.9 minutes (Section V-A). Table I summarizes the filtering.
to non-source-code files).
We collect revisions for each of the 22 projects for rerunning
We implement our hybrid technique in a tool called GIB-
on Travis. We collect these revisions from actual prior Travis
stazi. GIBstazi builds upon GIB to determine affected mod-
builds. In contrast, recent work on RTS [15], [23] selected
ules, and for each such module, GIBstazi applies Ekstazi to
the revisions as sequential commits from the master branch
select tests within the module. If any change is to non-source-
in the GitHub repository of each project. However a single
code files, GIBstazi defaults back to GIB and selects all the
Travis build corresponds to a push from the developer, and the
tests within the affected modules. For each module, GIBstazi
code changes between two pushes can correspond to several
selects either (1) no tests (if the module is not affected), (2) all
commits in the repository.
tests (if some non-source-code file changed), or (3) the same
tests as Ekstazi. We expect GIBstazi to select fewer tests than For each project, we collect from Travis the commit SHAs
GIB but more than Ekstazi; the time savings from GIBstazi associated with the latest 20 push builds on the master branch.
should also be between GIB and Ekstazi. GIBstazi is a fork We collect these SHAs in the order in which they actually hap-
of GIB, publicly available on GitHub [3]. pened on Travis such that replaying these historical commits
later on gives the same code changes between each build as
observed by the developers when using Travis for these builds.
IV. E XPERIMENTAL S ETUP
We describe how we select the projects for our experiments
and the commits for each project. We then describe how we B. Replaying with RTS
configure to run the different RTS techniques for each project’s We replay the commits collected for each project on Travis
commits on Travis. Finally, we describe how we collect the for all RTS techniques, including RetestAll. For each tech-
job results for our evaluation. The collected job logs and our nique, we create a new GitHub account and fork the projects
results are publicly available [30]. into the account, and then for each commit of a project, do
the following four steps: (1) checkout the commit (specifi-
A. Projects cally with “git checkout $sha .” using ‘.’ to not create a
detached branch); (2) modify the [Link] and .[Link]
Since the RTS tools we use are for the Maven build sys-
files to use a specific RTS technique on the project when run on
tem [4], our evaluation requires Maven projects. In addition,
Travis; (3) modify the [Link] and .[Link] files further
given that GIB and GIBstazi operate at the module level, we
for our experimental purposes, to count tests run and measure
need Maven projects that are multi-module. Moreover, we
time for running, with these modifications being the same for
need these projects to build on Travis. We query GitHub to get
all techniques, including RetestAll; and (4) recommit the files
2 By “test” we mean “test class”. Ekstazi selects test classes that each can after the modifications as a new, fresh commit and push it to
have several test methods. We count tests at the level of test classes as well. our forked repository on GitHub, triggering the build, and thus
 5

one or more jobs, on Travis. (The very first commit that we TABLE II: Basic statistics about projects used in evaluation,
recommit for each of the three RTS techniques selects all tests including distribution of pass/fail/error statuses for RetestAll
as in RetestAll, but the later commits use RTS.) We describe
ID Project # Jobs Pass Fail Error
next the specific modifications for each RTS technique for P1 SonarSource/sonarqube 38 19 19 0
step (2) and the general modifications for evaluation purposes P2 elasticjob/elastic-job-lite 19 19 0 0
P3 apache/rocketmq 19 0 19 0
for step (3). We aim for smallest necessary modifications to P4 alibaba/dubbo 18 18 0 0
minimize risk of affecting the build process in each project. P5 aws/aws-sdk-java 19 18 1 0
P6 brianfrankcooper/YCSB 19 18 1 0
1) GIB: We modify the project’s top-level [Link] file to P7 apache/incubator-skywalking 19 19 0 0
include the GIB Maven extension. We configure the extension P8 antlr/antlr4 170 169 1 0
P9 vavr-io/vavr 5 5 0 0
to compare the differences between two Git commit SHAs, P10 Graylog2/graylog2-server 1 1 0 0
where the first is the commit SHA of the previous build and the P11 javaparser/javaparser 19 19 0 0
P12 languagetool-org/languagetool 18 17 1 0
second is the current SHA. The Travis environment variable P13 druid-io/druid 83 69 14 0
TRAVIS_COMMIT_RANGE provides these two commit SHAs. P14 killbill/killbill 47 0 47 0
P15 apache/storm 84 76 8 0
We modify the .[Link] file’s cache section to save the P16 iluwatar/java-design-patterns 19 14 5 0
classpathfile generated (Section III-A). The cache is needed P18 google/guava 35 35 0 0
P17 javaee-samples/javaee7-samples 335 334 1 0
to share data between jobs because Travis runs each job on a P19 prestodb/presto 172 171 1 0
fresh virtual machine. We also configure .[Link] to not P20 apache/incubator-pulsar 13 7 4 2
P21 apache/flink 211 209 2 0
use GIB during the install phase, as the entire project must P22 Tencent/angel 14 7 7 0
build from scratch, and using GIB in this phase could prevent SUM 1377 1244 131 2
certain modules from being compiled. We disable GIB in the
before_install phase (which occurs right before the install
phase) and then enable it in the before_script phase (which install and script phases, and because we remove this phase
occurs right before the script phase). for all techniques, including RetestAll, our timing comparison
2) Ekstazi: We modify the project’s top-level [Link] is consistent as well. Finally, we modify .[Link] to not
file to include the Ekstazi Maven plugin; we use version run any jobs with Java versions below Java 8, because GIB
4.6.3 in our evaluation. We modify the .[Link]’s cache requires Java 8.
section to save in between jobs one combined .ekstazi
directory with metadata for all modules. We further add
in the before_script phase the commands to copy the C. Collecting Job Logs
cached .ekstazi directories to each module in the project Replaying each commit starts jobs on Travis. After each
for the script phase to use for testing, and we add in the job finishes, we download its log from Travis for analysis.
before_cache phase the commands to combine the updated We further consider only the jobs where we can successfully
.ekstazi directories after the tests finish. These .ekstazi parse from the logs the number of tests run and the test time
directories can be much bigger than the classpathfile cached in the script phase. Jobs may not finish properly for several
by GIB, and caching these directories is a necessary extra reasons, such as compilation errors (so tests are not even run)
overhead to use Ekstazi in a cloud-based CI environment. or strict timeouts maintained by Travis. Moreover, we consider
3) GIBstazi: We modify the [Link] and .[Link] only the jobs after the first commit for each project, because
files the same way as necessary for both GIB and Ekstazi for the first commit all RTS techniques select all tests (there is
individually, i.e., configuring to add the GIBstazi extension, no change yet), and we want to measure the effectiveness of
and configuring .[Link] to cache between jobs both the RTS in the steady state, after changes have happened. Finally,
classpathfile and .ekstazi directories while also copying we do not analyze any project where any of the RTS tools we
them appropriately. use consistently crashes for all the jobs due to internal tool
4) Modifications for all techniques for experiments: For our errors. In total we collect 1377 jobs across 22 projects.
evaluation, we need extra modifications to report tests selected
and time taken. At the end of the script phase, we add
commands to report how many tests are run by counting the D. Statistics of Jobs
number of Surefire report files generated that each represent Table II shows the distribution of the jobs that we collected
a test run. We also add commands in the script phase to across the 22 projects from our evaluation. We label each
report how much time the script phase takes to run. Timing project with an ID that we use later and show the project’s slug
the script phase, where testing is meant to be performed, from GitHub. We also show the number of jobs we analyze for
we can simulate running RTS “locally”, without including each project, classified as pass, fail, or error based on the job
the times for compiling code from scratch or downloading status reported by Travis for RetestAll. The overall number
dependencies; we refer to the time measured in this phase of jobs with status pass, fail, and error are 1244 (90.3%),
as test time. We next remove from .[Link] the entire 131 (9.5%), and 2 (0.2%), respectively. For the two jobs with
notifications phase, which is used to notify developers of the error status, we find it due to an unsuccessful phase that
the job status; we do not want to spuriously notify developers occurs after the script phase, i.e., after tests have run, so
concerning our replaying of their jobs. Removing this phase even in such cases, we can still collect information about the
does not disrupt the compile and testing process in the previous tests selected to run and the time for testing.
 6

Recall that jobs having status fail in Travis does not neces- We see from the final row that Ekstazi now outperforms
sarily mean that tests failed but that the script phase failed, both GIB and GIBstazi in terms of selecting the fewest tests
which may not be due to test failures. For example, project P3 and having the shortest test time and total time. GIBstazi,
has all of its jobs with status fail, but we find that the reason on the other hand, outperforms GIB in terms of all three
is due to the script phase including a step that tries to deploy metrics. Performing a series of Wilcoxon paired signed-rank
artifacts to another server, which we cannot access. Tests pass tests for the tests selected, test time, and total time among all
before this step, but because the deploy step is in the script pairs of the three techniques, we find statistically significant
phase, Travis marks the entire job as fail. differences (p < 0.01) for the percentage of tests selected and
test time, but no such differences for the total time.
V. R ESULTS Overall, the trend between GIB, Ekstazi, and GIBstazi in
We aim to answer the following two research questions: terms of tests selected, test time, and total time now matches
our initial expectations. However, there are jobs where the
RQ1: How do different RTS techniques compare in terms of
trend does not hold, so we inspect them in more detail.
tests selected, test time, and total build time in CI?
RQ2: How well does RTS select failing tests in CI? 1) Ekstazi Selects More Tests: We find 64 jobs distributed
across eight projects where Ekstazi runs more tests than GIB
or GIBstazi. We sample a job from each of these projects, as
A. RQ1: Tests Selected, Test Time, and Total Time it is likely a characteristic of the project that leads to Ekstazi
We first evaluate RTS techniques for all Travis jobs in our running more tests. We examine the job logs, the diffs between
experiments, regardless of the job status. For each job, we the job’s commits, and the job configurations. Overall, we find
compute the percentage of tests selected, test time, and total four different causes.
time of each RTS technique relative to RetestAll. We also Non-Default Runners. In P6, P15, and P16, we find jobs
compute the arithmetic mean of these percentages for all jobs where GIB skips modules where Ekstazi runs tests. These
in each project, and finally we compute the overall arithmetic projects have tests that do not use the default JUnit4 runner:
mean of these averages per project. Overall, GIB, Ekstazi, and TestNG, JUnit Enclosed runner, or JUnit Jupiter (new in
GIBstazi, respectively, select 59.1%, 35.2%, and 42.8% of the JUnit5). Ekstazi incorrectly runs all tests using these runners,
tests, take 86.6%, 65.5%, and 59.4% of the test time, and take regardless of changes3 . GIB and GIBstazi (correctly) find that
77.3%, 77.9%, and 72.2% of the total time. a module is unaffected and do not run any test.
Surprisingly, GIBstazi appears to be the fastest technique, Non-Deterministic Compilation. In P4 and P17, we find that
unlike our initial expectations. However, we find many jobs compiling even the same commit twice in a row results in
passing for some technique with the corresponding jobs failing different compiled .class files. P4 uses cobertura, which
for RetestAll. As a result, some jobs even exceed 100% as creates instrumented classes on which Ekstazi finds test de-
the percentage of tests run by the RTS techniques relative to pendencies; cobertura’s instrumentation is non-deterministic
RetestAll, appearing as if RTS runs more tests than available! and does not always create the same final .class file for the
The reason for this anomaly is that test failures occur in the same source file. P17 automatically generates some source
middle of job execution. (Many of these failures are flaky tests, files as part of the build process, but the generation is non-
as we discuss in Section V-B.) By default, when a test fails deterministic. Specifically, the order of the methods in the
in a multi-module Maven project, Maven stops early, skipping generated source files can differ between runs, which in turn
all modules that come after the module with the failed test(s). results in different compiled .class files; Ekstazi relies on
As such, the remaining tests that should have been run are the .class files to not change if the developer makes no
not actually run. Our tooling counts the number of tests that changes to source code, arguing that comparing .class files is
are actually run, so it ends up not counting all the tests the more robust than comparing source files [14]. In this scenario,
technique would have run had there been no test failure. While Ekstazi finds spurious changes and runs too many tests.
these numbers reflect what a developer would actually observe Incompatible with GIB. In P8 and P13, we notice that the
on Travis, they do not allow us to properly answer RQ1. job configuration for the jobs where Ekstazi runs more tests
To provide a fairer comparison of the RTS techniques and is set to navigate into a specific module to run only its tests
RetestAll, we focus on only the jobs where RetestAll and all (effectively cd module; mvn test). GIB assumes the root of
three RTS techniques have status pass, i.e., jobs where all tests the project starts from the current module and is unaware that
that should be run are actually run. Table III shows the results the current module is part of a larger Maven project. Thus,
for these 935 passing jobs, with an average of 51.9 jobs per GIB does not determine that the current module is affected
project for 18 projects. We do not show the four projects that by changes from the other modules, and it skips the current
have no jobs where RetestAll and all three RTS techniques module altogether. Therefore, GIB is running too few tests in
pass. The columns under “RetestAll” show the number of tests, these cases. Developers using GIB (and GIBstazi) need more
the test time, and the total job time, all averaged across all jobs in-depth changes to their specific job configurations for GIB
for each project, and then across the projects in the final row. to work correctly in these cases; specifically, they should not
The columns under “GIB”, “Ekstazi”, and “GIBstazi” show navigate into a module and instead run from the root using
the average percentage of each metric relative to RetestAll the Maven’s -pl option to specify the module.
for each respective RTS technique. The “AVG” row is the
arithmetic mean of the values in each column. 3 We confirmed via private communication with the developers of Ekstazi.
 7

TABLE III: Tests selected and time savings from using RTS across only passed jobs

ID # Jobs RetestAll GIB Ekstazi GIBstazi

Tests Test Total Tests Test Total Tests Test Total Tests Test Total
(#) Time (m) Time (m) (%) Time (%) Time (%) (%) Time (%) Time (%) (%) Time (%) Time (%)
P1 19 2.0 3.9 5.1 100.0 102.8 100.9 100.0 101.1 97.0 100.0 99.2 97.2
P2 19 158.4 1.4 6.5 24.4 39.1 78.3 7.1 58.9 97.8 24.4 40.4 70.3
P4 18 161.9 5.6 8.7 57.3 61.4 74.4 11.6 94.6 121.7 29.1 55.7 80.9
P5 15 179.0 7.4 9.3 100.0 103.6 103.4 16.6 49.8 60.6 100.0 109.7 109.7
P6 18 31.1 7.0 10.7 61.5 61.5 69.7 44.6 25.8 62.0 43.6 56.1 74.4
P7 19 98.7 4.1 10.8 17.5 29.8 80.4 2.5 37.5 56.7 13.7 31.5 79.1
P8 169 12.9 10.0 11.8 10.5 16.0 33.7 21.7 35.1 50.1 10.4 16.9 35.5
P9 5 140.0 10.9 11.8 100.0 86.9 101.4 38.7 71.0 75.7 59.0 75.6 97.2
P10 1 177.0 7.3 12.9 100.0 109.9 115.3 0.0 36.9 70.0 100.0 154.4 131.7
P11 19 177.0 7.0 14.9 91.1 94.1 83.1 24.1 59.2 50.9 26.5 58.9 46.0
P12 11 400.1 12.7 16.2 7.7 8.7 22.5 0.8 4.6 21.6 7.7 13.0 29.6
P13 38 198.3 12.8 20.8 33.5 48.4 72.1 25.1 49.8 73.0 7.5 29.5 67.6
P15 47 53.2 5.1 14.9 47.4 61.8 91.6 5.8 69.4 93.9 26.0 63.7 91.0
P16 12 319.2 6.7 22.1 25.9 31.9 40.9 55.8 87.6 94.0 17.5 28.5 41.5
P17 330 8.9 3.7 22.2 32.1 45.7 102.4 20.8 41.7 93.4 31.8 53.7 102.7
P18 35 494.1 16.8 20.2 100.0 97.5 98.1 55.8 62.5 83.3 56.1 61.6 69.2
P19 24 12.4 25.1 29.5 100.0 98.6 99.3 100.0 100.9 101.4 100.0 99.3 99.9
P21 136 366.5 30.7 32.1 49.1 64.9 66.9 19.5 62.7 65.2 40.0 68.1 70.2
AVG 51.9 166.1 9.9 15.6 58.8 64.6 79.7 30.6 58.3 76.0 44.1 62.0 77.4

Job Timeout. In P21, for a job where Ekstazi runs more Overhead of Ekstazi Requiring Compilation. We find jobs
tests, we find that the changes should not actually affect the where Ekstazi spends a lot of time to determine that it need
module where Ekstazi runs tests, so Ekstazi runs too many not run tests in some modules, while GIB and GIBstazi quickly
tests. We find that the immediately prior job for Ekstazi times determine to run very few modules. For example, in P4, we
out, so the cache for Ekstazi dependencies is not updated for find a job where Ekstazi and GIBstazi run no tests. GIBstazi
the subsequent job. Thus, Ekstazi compares the next commit determines this rather fast, as it first selects very few modules,
with the commit two (rather than one) before it, finding more and those modules have no tests to run. However, Ekstazi
changes than GIB and GIBstazi find by comparing the next has to analyze each and every module to determine that no
commit with the one before it. Essentially Ekstazi finishes tests should be run in it. While this analysis is generally
running the tests from the prior job in the subsequent job rather fast, e.g., 2–4 seconds per module, P4 is a project
(which does not time out). This example demonstrates how with over 60 modules, so the time adds up. Furthermore,
much Ekstazi depends on completing prior runs. Timeouts can P4 is configured to run other plugins in each module, such
occur more often on CI machines in the cloud, out of the as cobertura instrumentation, adding even more time per
developers’ control, so this example also demonstrates issues module. GIBstazi’s skipping of all the unaffected modules
with using Ekstazi in such a CI environment. leads to substantial speedup against Ekstazi.
Trying to better understand differences between the tech- 3) Non-Source-Code Changes: In most cases, GIBstazi
niques rather than differences due to tool engineering, we runs at least as many tests as Ekstazi because GIBstazi defaults
further filter out jobs where Ekstazi runs more tests than the to GIB behavior when there are non-source-code changes. To
other techniques. We obtain the same trends between the three estimate the potential impact of such changes on RTS, we
techniques, with Ekstazi seeming even better. Overall, for GIB, measure how much tests depend on non-source-code files.
Ekstazi, and GIBstazi, respectively, the average percentages For each project, we first use fabricate [1], a tool that
of tests selected are 62.6%, 25.9%, and 47.4%; test times are traces what files are accessed when executing a command
68.1%, 55.2%, and 66.7%; and total times are 82.1%, 74.8%, (in our case, running mvn test), on the latest commit of
and 80.2% (not shown in tables due to space limits). the project. We record the non-source-code files used by
2) Ekstazi Runs Slower: Even when we consider only the tests. We find many file names with extensions such as
the jobs where Ekstazi runs no more tests than the other .json or .properties, and many files under test/resources,
techniques, we still find jobs where Ekstazi test time is suggesting these files are used as inputs or configuration for
longer. Overall, for GIB and Ekstazi, respectively, the average tests. We then measure how many commits in the project made
percentages of tests selected are 62.6% and 25.9%, and test changes to any of these files. While one project had no commit
times are 68.1% and 55.2%; the difference is much higher in that changed any of the dependencies found using fabricate,
tests selected than in test time. We examine several of these the other projects had on average 7.5% commits with such
jobs and find two reasons why Ekstazi runs slower. a change. Thus, Ekstazi has a relatively high risk to miss
Overhead of Ekstazi Instrumentation. Especially noticeable selecting a test affected by non-source-code changes.
in jobs where Ekstazi runs no more tests than GIB, Ekstazi
test time is longer primarily due to the runtime overhead of
extra instrumentation Ekstazi needs to track dependencies; B. RQ2: Test Failure Analysis
GIB requires no dynamic analysis. Note that in such cases RTS aims to select only tests affected by changes. A key
GIBstazi also runs roughly the same as Ekstazi, as GIBstazi question is whether RTS misses to select some tests that fail
also relies on the same instrumentation. due to the changes. If RTS misses such a real test failure, the
 8

TABLE IV: Percentage of test failures selected or not selected by RTS

Project # Failed GIB % Ekstazi % GIBstazi %

Tests Selected Not Selected Unknown Selected Not Selected Unknown Selected Not Selected Unknown
apache/flink 2 100.0 0.0 0.0 50.0 50.0 0.0 50.0 50.0 0.0
apache/incubator-pulsar 13 53.9 46.2 0.0 84.6 0.0 15.4 38.5 46.2 15.4
apache/storm 7 71.4 28.6 0.0 57.1 42.9 0.0 71.4 28.6 0.0
aws/aws-sdk-java 1 100.0 0.0 0.0 0.0 100.0 0.0 100.0 0.0 0.0
brianfrankcooper/YCSB 2 50.0 50.0 0.0 50.0 50.0 0.0 50.0 50.0 0.0
druid-io/druid 10 70.0 30.0 0.0 20.0 70.0 10.0 20.0 70.0 10.0
iluwatar/java-design-patterns 5 0.0 100.0 0.0 20.0 60.0 20.0 0.0 100.0 0.0
javaee-samples/javaee7-samples 7 0.0 100.0 0.0 0.0 14.3 85.7 0.0 100.0 0.0
killbill/killbill 47 100.0 0.0 0.0 100.0 0.0 0.0 100.0 0.0 0.0
languagetool-org/languagetool 1 0.0 100.0 0.0 0.0 100.0 0.0 0.0 100.0 0.0
prestodb/presto 1 0.0 100.0 0.0 0.0 100.0 0.0 0.0 100.0 0.0
SUM/AVG 96 72.9 27.1 0.0 69.8 19.8 10.4 64.6 32.3 3.1

developer could miss a regression fault, defeating the purpose For some tests, we find that one of the RTS techniques did
of regression testing. However, some failures are due to flaky run the test and yet the test passed. For the remaining tests,
tests and undesirable, so it would be beneficial to miss such we rerun the same job for RetestAll up to six times and check
failures. RTS is not explicitly designed to avoid flaky tests, but that the test passes in any rerun. In summary, all test failures
it can miss them by chance. In our study, we find several test are flaky except for one, from brianfrankcooper/YCSB, which
failures in RetestAll, so we check whether the RTS techniques consistently fails in those reruns. Examining the test logs and
select those tests, and whether those failures are desirable. code changes, we determine this one test failure to represent a
Table IV shows the number of test failures from RetestAll real fault, but we also find that all the RTS techniques selects
and the breakdown of percentage of those tests selected by this test, so none of them misses this regression fault.
the RTS techniques. We obtain the test failures by parsing the The failed jobs we have used in evaluation so far are from
Travis logs using tools from TravisTorrent [8]. Unfortunately, pushes to the master branch, so their failures are expected to
the parsing cannot successfully identify test failures for all be due to flaky tests if the developers follow good CI practices
failed jobs due to project-specific “noise” in the logs (e.g., and only merge in pull requests with passing jobs. However,
not printing test progress in the default format). Also, just the failures from pull request jobs may more likely be real failures
Travis status of the job itself does not necessarily indicate from attempting to merge. We apply RTS techniques on failed
there are test failures, as other Travis phases could mark the pull requests to examine whether the techniques miss real test
job as fail for other issues, so such jobs have no failed tests in failures. We first obtain recent (up to 10) failed pull requests
the logs. In total, we find 11 projects with test failures we can from our evaluation projects. We then rerun their jobs six
parse out, leading to a total of 96 test failures. We categorize times in the RetestAll configuration to confirm that the failed
those test failures for each RTS technique into three categories: tests fail consistently, resulting in 37 failed jobs. We then
selected, not selected, or unknown. We use unknown if we replay these failed jobs with Ekstazi, resulting in 19 successful
cannot tell from the logs whether the test would have been replays across five projects; Ekstazi fails to run in 18 jobs due
selected, which happens if another test in a module before the to the JVM setting. We find that Ekstazi does not miss any
relevant test fails in the RTS run, leading to an early failure real test failures in these 19 pull request jobs, and by extension
that skips running the tests in later modules. both GIB and GIBstazi also would not miss any test failures.
We inspect all the test failures from RetestAll. For 29 test In sum, our inspection shows that all tests that failed in
failures, we find that the job with the test failure was imme- RetestAll for pushes to the master branch but are not selected
diately preceded by a job that had the same test failure [22]. by any RTS technique are either failing from before (so the
When RTS does not select a test, the outcome of that not test outcome is already known) or due to flaky tests (so do not
selected test is not necessarily pass but the same outcome from reveal real regressions). As such, it is actually beneficial for
the prior run (may be fail), i.e., the failed outcome of the test RTS to not select these failed tests from RetestAll: the higher
is known from before. For example, if the developer makes the percentage of failed tests not selected, the better. GIBstazi
no change related to fixing a test failure, there is no need to does not select such tests at a higher percentage than the other
run the unaffected test again for the same known failure. An two techniques. Even if all the test failures categorized as
RTS tool can simply copy the prior outcome of the tests from unknown for Ekstazi are actually not selected, its percentage
the prior job if the tests are not run, allowing the developer to of not selected tests would be 30.2%, still lower than the not
consistently see the test failures that have not yet been fixed. selected percentage for GIBstazi, 32.3%. If we consider pull
request jobs where we confirm that test failures are not flaky,
We examine in detail the remaining 67 test failures where
the RTS techniques do not fail to select any failed test.
the test passed in the prior job. We find only one real failure
and all others due to flaky tests that can pass or fail non-
deterministically even on the same code [17], [24], [36]. Most C. Shadowing Projects
flaky test failures are not related to code changes and are In our experiments with replaying, we are unable to suc-
undesirable as they do not reveal real faults due to code cessfully replay many commits due to significant differences
changes [7]. We confirm that these tests are flaky in two ways. between the environment used for the jobs back when they
 9

TABLE V: Time savings from RTS on shadowed jobs with GIBstazi. The table only shows these numbers for the
jobs where both the original job and our shadowed job pass.
Project # Jobs Original Shadowed
Time (m) Time (%) GIBstazi runs 69.0% of the original time, similar to before.
aws/aws-sdk-java 8 8.9 114.0 RQ2. We inspect all cases where the original jobs fail and
apache/incubator-skywalking 15 11.8 72.4 TravisTorrent [8] could parse out test failures from the logs.
google/error-prone 4 15.7 70.5
languagetool-org/languagetool 93 16.0 53.9
We obtain 24 test failures from three projects: languagetool-
javaparser/javaparser 6 16.2 52.9 org/languagetool, google/guava, and iluwatar/java-design-
alibaba/dubbo 4 18.2 56.4 patterns. GIBstazi selects to run 17 of the failed tests (70.8%)
linkedin/pinot 36 18.3 42.1
google/guava 5 24.3 75.4
and does not select to run six of the failed tests (25.0%). We
iluwatar/java-design-patterns 2 25.9 83.5 cannot tell for one test if GIBstazi selected it, due to a (flaky!)
SUM/AVG 173 17.3 69.0 test failure from an earlier module.
For 15 test failures, we find that GIBstazi runs the failed
test in the prior job, where it also failed, so the test failure
would have been known. Of the remaining nine test failures,
were originally triggered and the current environment, e.g., the we confirm four to be flaky either from GIBstazi running
differences in external dependencies a project needs. For ex- the test and passing, or from our repeating the original job
ample, one of the projects we failed to replay is google/error- and observing the test passing. Excluding the one test failure
prone, which has a dependency on a SNAPSHOT version of that is skipped, we believe the remaining four test failures
JUnit. The JUnit developers can overwrite this version with are all real test failures. GIBstazi selects to run three of
new changes, so the dependency name does not uniquely those failed tests, so the developers would have noticed the
determine its content. One such change was to the API of failures. However, GIBstazi did not select to run one failed
certain methods that google/error-prone in an earlier commit test. This test failure is from languagetool-org/languagetool,
relies on. Since the SNAPSHOT version in the central repository and further inspection shows that the change made was to a
was updated, our replay uses this latest version, leading to .txt file. The regex we use for GIBstazi configuration ignores
compilation errors. As such, we do not use google/error-prone changes to .txt files, but for a project like languagetool-
in our evaluation on historical commits. org/languagetool, the .txt files are an integral part of tests.
To gain better understanding of RTS for RQ1 and RQ2, Thus, we made a configuration error [33] in GIBstazi, using
instead of replaying historical commits, we shadow the Travis generic filtering that applies to most projects. The developers
builds from 40 projects (including some whose RetestAll of languagetool-org/languagetool should apply project-specific
jobs do not compile in our earlier experiments), similar to filtering to ensure such test failures are not missed. Note that
Bell et al. [7]: we set up automatic tracking of projects Ekstazi would also have missed to select this failed test.
such that when their developers trigger Travis builds on the
master branch, our setup replays those builds close in time VI. T HREATS TO VALIDITY
to the triggered builds. One advantage of shadowing current Our results may not generalize beyond projects used in
builds is evaluating in an environment similar to when the our study. We use a diverse set of projects from GitHub,
builds are actually built (e.g., less likely to have out-of-date the most popular service for hosting open-source projects.
dependencies). Another advantage is that we can observe how We choose as many projects that could satisfy our filtering
well RTS helps the project for its current state, which may requirements, which includes choosing only Java and Maven
have different characteristics from the project state in the jobs projects due to tool constraints. We focus on projects that take
we replayed before. Finally, shadowing allows us to better a relatively long time to build and test, which are projects
evaluate the feasibility of utilizing RTS in a realistic cloud- where developers would want to use RTS to save regression
based CI environment as it stands currently. testing time. We believe that the 22 projects used in our study
To enable shadowing, we first fork each project into a new are fairly representative of such projects.
account. We then set up a cron job to query Travis once an The historical replays we perform are not exactly the same
hour for each project to check if some new builds occurred as if they had been run when the developers triggered the
since the last time the cron job was run. If there is a new build. A particular problem are any external dependencies
build, the cron job pulls the commits corresponding to the new the developers used at the time of the build but are now
build(s) into our shadowing fork and replays them with RTS. no longer available or, even worse, changed the content
We perform these runs only for GIBstazi, because we find it to and behavior while they still having the same name (e.g.,
strike a good balance among all three RTS techniques, saving SNAPSHOT dependencies). Replayed jobs from such builds can
more time than GIB, being designed to be safer than Ekstazi, fail although they would have passed when the developers
and having avoided the most flaky test failures. To limit our built. To alleviate this issue, we replay using not just the RTS
usage of Travis, we shadow for 20 days, obtaining results from techniques but also RetestAll, so we do not compare RTS
217 jobs of nine projects. We obtain shadowing results for two against the RetestAll job that happened potentially a while
projects (google/error-prone and linkedin/pinot) for which we ago with a drastically different setup. Moreover, we replay
have no results from before due to the issues with replaying. on Travis, the same environment the developers use, to more
RQ1. Table V shows the total time for the original job and closely imitate how the developers build their code using CI.
the percentage of that time taken by the shadowed version Finally, we use live shadowing and not just historical replays.
 10

VII. R ELATED W ORK between commits to gauge how effective RTS could be in
a CI environment. They also compared a class-level and a
Regression Test Selection. RTS has been studied for several method-level static RTS technique against RetestAll. Our work
decades [34]. Researchers have proposed various different RTS differs from theirs in that we investigate a module-level, class-
techniques, selecting tests by tracking dependencies at differ- level, and hybrid module/class-level dynamic RTS techniques
ent levels of granularities, ranging from precise control-flow in a CI environment. We also evaluate in a cloud-based CI
edges [18], [28] to methods [38] to classes [15], [23], [37]. environment, Travis, which differs from Yu and Wang’s choice
Recent work has emphasized the need for RTS to provide time of Jenkins, a CI environment on a local server that is more
savings in end-to-end regression testing. For example, Gligoric under the control of the developers. Finally, we further analyze
et al. proposed Ekstazi [15], which tracks dependencies at the the test failures that occur during the runs and compare how
class level and selects test classes as opposed to test methods, well the RTS techniques select the failing tests.
leading to a larger number of tests run compared to tracking There has been more work studying test failures on Travis
at a finer granularity. However, the analysis for RTS at the (but not in combination with RTS as we do). Labuschagne
class level is very quick, eventually leading to better time et al. [22] studied how often regression testing on Travis
savings. Zhang proposed HyRTS [37] that tracks dependencies reveals faults that developers fix. They queried Travis for the
at both class and method levels. Zhang found that HyRTS results of builds and focused on patterns of builds that toggle
outperformed class-level RTS in terms of tests selected but pass and fail outcomes, indicating where a change caused an
could not always outperform class-level RTS in time due to originally passing build to start to fail, followed by changes
the costs of method-level dependency collection. Companies that lead to the build passing again. They found 74% of the
such as Google and Microsoft rely on even coarser-grained non-flaky failed builds were caused by a fault in the code
dependency tracking, at the module level, due to the even under test, with the remaining due to incorrect/obsolete tests.
quicker analysis time [12], [13], [29]. This work compares They reported flaky tests to affect 13% of the failed builds they
module- and class-level RTS in a cloud-based CI environment. studied (although this percentage is an underestimate, as they
Our work is quite similar to work by Vasic et al. [31] that considered historically failed builds that consistently passed
created Ekstazi#, a tool that performs class-level RTS, like during their reruns as non-flaky). We also find flaky tests in
Ekstazi, for the .NET framework. Vasic et al. also evaluated our study, although we find that almost all the test failures
running Ekstazi# on top of an incremental build system Con- from RetestAll in our study are flaky test failures. One reason
cord, which inherently performs module-level RTS. For one for the different percentage of flaky tests is that they studied all
project on which they evaluated, they found that adding class- builds (from the master branch and other branches, as well
level RTS improves module-level RTS time by 65.26%. Our as pull-request builds) whereas we studied builds from the
hybrid RTS technique GIBstazi follows the ideas introduced master branch (because many other builds cannot be replayed
by Ekstazi# and Concord. However, GIBstazi differs from their as they are not available).
combination in that when changes are not source-code related,
GIBstazi defaults back to GIB behavior, running all tests VIII. C ONCLUSIONS
within affected modules, thereby being safer than just running
Ekstazi (or Ekstazi#), which does not track those changes and Regression testing is widely practiced but costly, and RTS
runs no tests within affected modules. As such, we find that reduces the cost. Industry has adopted module-level RTS,
GIBstazi improves over GIB much less, 62.0% versus 64.6% while research has reported class-level RTS to be effective.
of RetestAll time, respectively. Furthermore we compare both We compare module- and class-level RTS in a cloud-based
module-level RTS and class-level RTS, as well as against CI environment. We find that RTS techniques improve testing
GIBstazi, in a cloud-based CI environment, where every build time over RetestAll in this environment, and GIBstazi, our
starts fresh on a new machine; Vasic et al. evaluated Ekstazi# new hybrid module- and class-level RTS technique, offers a
on a dedicated machine. good trade-off. Our investigation of test failures from RetestAll
Continuous Integration. Continuous integration is widely shows that the RTS techniques often miss to select some failed
used in industry. Recent work has studied why developers tests, but this happens to be desirable because these tests are
use CI and the benefits they experience [20], [21], [39]. One flaky and not indicative of faults introduced by code changes.
main reason for the rise in CI research is the increase in In sum, the results show that RTS offers benefit to developers,
developers using CI, particularly with services such as Travis, not only to reduce machine time but also to avoid false alarms
which provides CI for free for open-source projects on GitHub. from flaky tests.
Moreover, Travis exposes the logs from the builds that occur
on their servers, allowing ease of access to build results. ACKNOWLEDGMENTS
TravisTorrent [8] provides a dataset of logs from Travis and We thank Tianyin Xu for his comments on an earlier draft
also some tooling for parsing the logs. We utilize Travis for of this paper and Qianyang Peng for providing some data
our evaluation, and we also use the TravisTorrent tooling to on test failures for our evaluation. This work was partially
parse the logs for our analysis of failed tests. supported by NSF grants. CCF-1421503, CNS-1646305, CNS-
Yu and Wang [35] recently studied the potential of RTS 1740916, CCF-1763788, and OAC-1839010. We acknowledge
in a CI environment. They analyzed factors such as the support for research on regression testing and flaky tests from
commit frequency of projects and the size of the changes Facebook, Futurewei, Google, Microsoft, and Qualcomm.
 11

R EFERENCES [21] M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig. Usage, costs,
and benefits of continuous integration in open-source projects. In ASE,
[1] fabricate. [Link]
pages 426–437, 2016.
[2] gitflow-incremental-builder. [Link]
[22] A. Labuschagne, L. Inozemtseva, and R. Holmes. Measuring the cost of
gitflow-incremental-builder.
regression testing in practice: A study of Java projects using continuous
[3] gitflow-incremental-builder with GIBstazi. [Link]
integration. In ESEC/FSE, pages 821–830, 2017.
august782/gitflow-incremental-builder.
[23] O. Legunsen, F. Hariri, A. Shi, Y. Lu, L. Zhang, and D. Marinov. An
[4] Maven. [Link]
extensive study of static regression test selection in modern software
[5] Travis-CI. [Link]
evolution. In FSE, pages 583–594, 2016.
[6] A. Alali, H. Kagdi, and J. I. Maletic. What’s a typical commit? A
[24] Q. Luo, F. Hariri, L. Eloussi, and D. Marinov. An empirical analysis of
characterization of open source software repositories. In ICPC, pages
flaky tests. In FSE, pages 643–653, 2014.
182–191, 2008.
[25] A. Memon, Z. Gao, B. Nguyen, S. Dhanda, E. Nickell, R. Siemborski,
[7] J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov.
and J. Micco. Taming Google-scale continuous testing. In ICSE-SEIP,
DeFlaker: Automatically detecting flaky tests. In ICSE, pages 433–444,
pages 233–242, 2017.
2018.
[26] A. Orso, N. Shi, and M. J. Harrold. Scaling regression testing to large
[8] M. Beller, G. Gousios, and A. Zaidman. TravisTorrent: Synthesizing
software systems. In FSE, pages 241–251, 2004.
Travis CI and GitHub for full-stack research on continuous integration.
[27] G. Rothermel, S. Elbaum, A. Malishevsky, P. Kallakuri, and B. Davia.
In MSR, pages 447–450, 2017.
The impact of test suite granularity on the cost-effectiveness of regres-
[9] C. Bird, P. C. Rigby, E. T. Barr, D. J. Hamilton, D. M. German, and
sion testing. In ICSE, pages 130–140, 2002.
P. Devanbu. The promises and perils of mining Git. In MSR, pages
[28] G. Rothermel and M. J. Harrold. A safe, efficient regression test
1–10, 2009.
selection technique. TOSEM, 6(2):173–210, 1997.
[10] H. Borges, A. Hora, and M. T. Valente. Predicting the popularity of
[29] A. Shi, S. Thummalapenta, S. K. Lahiri, N. Bjorner, and J. Czerwonka.
GitHub repositories. In PROMISE, pages 9:1–9:10, 2016.
Optimizing test placement for module-level regression testing. In ICSE,
[11] C. Brindescu, M. Codoban, S. Shmarkatiuk, and D. Dig. How do
pages 689–699, 2017.
centralized and distributed version control systems impact software [30] A. Shi, P. Zhao, and D. Marinov. Dataset for understanding and
changes? In ICSE, pages 322–333, 2014. improving regression test selection in continuous integration. 2019.
[12] S. Elbaum, G. Rothermel, and J. Penix. Techniques for improving [Link]
regression testing in continuous integration development environments. [31] M. Vasic, Z. Parvez, A. Milicevic, and M. Gligoric. File-level vs.
In FSE, pages 235–245, 2014. module-level regression test selection for .NET. In ESEC/FSE, pages
[13] H. Esfahani, J. Fietz, Q. Ke, A. Kolomiets, E. Lan, E. Mavrinac, 848–853, 2017.
W. Schulte, N. Sanches, and S. Kandula. CloudBuild: Microsoft’s [32] G. Xu and A. Rountev. Regression test selection for AspectJ software.
distributed and caching build service. In ICSE, pages 11–20, 2016. In ICSE, pages 65–74, 2007.
[14] M. Gligoric, L. Eloussi, and D. Marinov. Ekstazi: Lightweight test [33] T. Xu, L. Jin, X. Fan, Y. Zhou, S. Pasupathy, and R. Talwadker. Hey,
selection. In ICSE DEMO, pages 713–716, 2015. you have given me too many knobs! Understanding and dealing with
[15] M. Gligoric, L. Eloussi, and D. Marinov. Practical regression test over-designed configuration in system software. In ESEC/FSE, pages
selection with dynamic file dependencies. In ISSTA, pages 211–222, 307–319, 2015.
2015. [34] S. Yoo and M. Harman. Regression testing minimization, selection and
[16] A. Gyori, O. Legunsen, F. Hariri, and D. Marinov. Evaluating regression prioritization: A survey. STVR, 22(2):67–120, 2012.
test selection opportunities in a very large open-source ecosystem. In [35] T. Yu and T. Wang. A study of regression test selection in continuous
ISSRE, pages 112–122, 2018. integration environments. In ISSRE, pages 135–143, 2018.
[17] M. Harman and P. O’Hearn. From start-ups to scale-ups: Opportunities [36] A. Zaidman and F. Palomba. Does refactoring of test smells induce
and open problems for static and dynamic program analysis. In SCAM, fixing flaky tests? In ICSME, pages 1–12, 2017.
pages 1–23, 2018. [37] L. Zhang. Hybrid regression test selection. In ICSE, pages 199–209,
[18] M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings, 2018.
S. Sinha, S. A. Spoon, and A. Gujarathi. Regression test selection for [38] L. Zhang, M. Kim, and S. Khurshid. FaultTracer: A change impact and
Java software. In OOPSLA, pages 312–326, 2001. regression fault analysis tool for evolving Java programs. In FSE, pages
[19] K. Herzig, M. Greiler, J. Czerwonka, and B. Murphy. The art of testing 40:1–40:4, 2012.
less without sacrificing quality. In ICSE, pages 483–493, 2015. [39] Y. Zhao, A. Serebrenik, Y. Zhou, V. Filkov, and B. Vasilescu. The impact
[20] M. Hilton, N. Nelson, T. Tunnell, D. Marinov, and D. Dig. Trade- of continuous integration on other software development practices: a
offs in continuous integration: Assurance, security, and flexibility. In large-scale empirical study. In ASE, pages 60–71, 2017.
ESEC/FSE, pages 197–207, 2017.