Problem Formulation and Fairness

Samir Passi Solon Barocas

Information Science Information Science
Cornell University Cornell University
sp966@[Link] sbarocas@[Link]

ABSTRACT ACM Reference Format:

Samir Passi and Solon Barocas. 2019. Problem Formulation and
Formulating data science problems is an uncertain and Fairness. In Proceedings of the ACM Conference on Fairness,
difficult process. It requires various forms of discretionary Accountability, and Transparency (FAT* '19), January 29-31,
work to translate high-level objectives or strategic goals into 2019. ACM, Atlanta, GA. 10 pages.
tractable problems, necessitating, among other things, the [Link]
identification of appropriate target variables and proxies.
While these choices are rarely self-evident, normative 1 INTRODUCTION
assessments of data science projects often take them for Undertaking a data science1 project involves a series of
granted, even though different translations can raise difficult translations. As Provost and Fawcett point out,
profoundly different ethical concerns. Whether we consider a “[b]usiness problems rarely are classification problems, or
data science project fair often has as much to do with the regression problems or clustering problems” [38:293]. They
formulation of the problem as any property of the resulting must be made into questions that data science can answer.
model. Building on six months of ethnographic fieldwork with Practitioners are frequently charged with turning amorphous
a corporate data science team—and channeling ideas from goals into well-specified problems—that is, problems faithful
sociology and history of science, critical data studies, and early to the original business objectives, but also problems that can
writing on knowledge discovery in databases—we describe the be addressed by predicting the value of a variable. Often, the
complex set of actors and activities involved in problem outcome or quality that practitioners want to predict—the
formulation. Our research demonstrates that the specification ‘target variable’—is not one that has been well observed or
and operationalization of the problem are always negotiated measured in the past. In such cases, practitioners look to other
and elastic, and rarely worked out with explicit normative variables that can act as suitable stand-ins—‘proxies’.
considerations in mind. In so doing, we show that careful This process is challenging and far from linear. As Hand
accounts of everyday data science work can help us better argues, “establishing the mapping from the client’s domain to
understand how and why data science problems are posed in a statistical question is one of the most difficult parts of
certain ways—and why specific formulations prevail in statistical analysis,” [22:317] and data scientists frequently
practice, even in the face of what might seem like normatively devise ways of providing answers to problems that differ from
preferable alternatives. We conclude by discussing the those that seemed to motivate the analysis. In most normative
implications of our findings, arguing that effective normative assessments of data science, this work of translation drops out
interventions will require attending to the practical work of entirely, treating the critical task as one of interrogating
problem formulation. properties of the resulting model. However, ethical concerns
can extend to the formulation of the problem that a model
CCS CONCEPTS aims to address, not merely to whether the model exhibits
• Information systems → Data mining; • Computing discriminatory effects.
methodologies → Machine learning; • Human-centered To aid in hiring decisions, for example, machine learning
computing → Ethnographic studies needs to predict a specific outcome or quality of interest. One
might want to use machine learning to find “good” employees
KEYWORDS to hire, but the meaning of “good” is not self-evident. Machine
Data Science; Machine Learning; Problem Formulation; learning requires specific and explicit definitions, demanding
Fairness; Target Variable that those definitions refer to something measurable. While an
employer might want to find personable applicants to join its

Permission to make digital or hard copies of all or part of this work for personal 1 The term ‘data science,’ as used in this paper, refers to the practice of
or classroom use is granted without fee provided that copies are not made or assembling, organizing, processing, modeling, and analyzing data using
distributed for profit or commercial advantage and that copies bear this notice computational and statistical techniques. In this paper, we focus specifically on
and the full citation on the first page. Copyrights for components of this work supervised machine learning, but the analysis applies more broadly.
owned by others than ACM must be honored. Abstracting with credit is
permitted. To copy otherwise, or republish, to post on servers or to redistribute
to lists, requires prior specific permission and/or a fee. Request permissions from
Permissions@[Link]. FAT* '19, January 29–31, 2019, Atlanta, GA, USA © 2019
Association for Computing Machinery. ACM ISBN 978-1-4503-6125-5/19/01
$15.00 [Link]
 PROBLEM FORMULATION AND FAIRNESS

sales staff, such a quality can be difficult to specify or measure. by population, then the model might subject certain groups to
What counts as personable? And how would employers greater errors than others.
measure it? Given the challenge of answering these questions, Across all three cases, we find that whether a model
employers might favor a definition focused on sales figures, ultimately violates a specific notion of fairness is often
which they may find easier to monitor. In other words, they contingent on what the model is designed to predict. Which
might define a “good” employee as the person with the highest suggests that we should be paying far greater attention to the
predicted sales figures. In so doing, the problem of hiring is choice of the target variable, both because it can be a source of
formulated as one of predicting applicants’ sales figures, not unfairness and a mechanism to avoid unfairness.
simply identifying “good” employees. The non-obvious origins of obvious problems. This
As Barocas and Selbst [3] demonstrate, choosing among might not be surprising because some problem formulations
competing target variables can affect whether a model used in may strike us as obviously unfair. Consider the case of
hiring decisions ultimately exhibits a disparate impact. There ‘financial-aid leveraging’ in college admissions—the process
are three reasons why this might happen. First, the target by which universities calculate the best possible return for
variable might be correlated with protected characteristics. In financial aid packages: the brightest students for the least
other words, an employer might focus on a quality that is amount of financial aid. To achieve this bargain, the university
distributed unevenly across the population. This alone would must predict how much each student is willing to pay to attend
not constitute illegal discrimination, as the quality upon which the university and how much of a discount would sway an
the employer hinges its hiring decisions could be rational and applicant from competitors. In economic terms, ‘financial-aid
defensible. But, the employer could just as well choose a target leveraging’ calculates each applicant’s responsiveness to price,
variable that is a purposeful proxy for race, gender, or other which enables the university to make tailored offers that
protected characteristics. This would amount to a form of maximize the likely impact of financial aid on individual
disparate treatment, but one that might be difficult to establish enrollment decisions. As Quirk [39] explains: “Take a $20,000
if the decision rests on a seemingly neutral target variable. The scholarship—the full tuition for a needy student at some
employer could also choose a target variable that seems to schools. Break it into four scholarships each for wealthier
serve its rational business interests but happens to generate an students who would probably go elsewhere without the
avoidable disparate impact—for instance, the employer could discount but will pay the outstanding tuition if they can be
choose a different target variable that serves its business lured to your school. Over four years the school will reap an
objective at least as well as the original choice while also extra $240,000, which can be used to buy more rich students—
reducing the disparate impact. or gifted students who will improve the school’s profile and
Second, the chosen target variable might be measured less thus its desirability and revenue.” Such strategies are in effect
accurately for certain groups. For example, arrests are often in schools throughout the United States, and the impact has
used as a proxy for crime in applications of machine learning been an increase in support for wealthier applicants at the
to policing and criminal justice, even though arrests are a expense of their equally qualified, but poorer peers [42,43].
racially biased representation of the true incidence of crime One might, therefore, conclude, as Danielson [11:44] does,
[28]. In treating arrests as a reliable proxy for crime, the model that “data mining technology increasingly structures
learns to replicate the biased labels in its predictions. This is a recruiting to many U.S. colleges and universities,” and that the
particularly pernicious problem because the labeled examples technology poses a threat to such important values as equality
in the training data serve as ground truth for the model. and meritocracy. Alternatively, one could find, like Cook [9]
Specifically, the model will learn to assign labels to cases in a similar thought experiment, that “[t]he results would have
similar to those that received the label in the training data, been different if the goal were to find the most diverse student
whether or not the labels in the training data are accurate. population that achieved a certain graduation rate after five
Worse, evaluations of the model will likely rely on test data years. In this case, the process was flawed fundamentally and
that were labeled using the same process, resulting in ethically from the beginning.” For Cook, agency and ethics are
misleading reports about the model’s real-world performance: front-loaded: a poorly formed question returns undesirable, if
these metrics would reflect the model’s ability to predict the correct, answers. Data science might be the enabling device,
label, not the true outcome. Indeed, when the training and test but the ethical issue precedes the analysis and
data have been mislabeled in the same way, there is simply no implementation. The objective was suspect from the start. For
way to know when the model is making mistakes. Choosing a Danielson, however, certain ethics seem to flow from data
target variable is therefore often a choice between outcomes mining itself. Data science is not merely the enabling device,
of interest that are labeled more or less accurately. When these but the impetus for posing certain questions. Its introduction
outcomes are systematically mismeasured by race, gender, or affords new, and perhaps objectionable, ways of devising
some other protected characteristic, a model designed to admissions strategies.
predict them will invariably exhibit a discriminatory bias that Though they are quite different, these positions are not
does not show up in performance metrics. necessarily incompatible: data science might invite certain
Finally, different target variables might be more difficult to kinds of questions, and ‘financial-aid leveraging’ could be one
predict than others depending on the available training data such example. One might say that data science promotes the
and features. If the ability to predict the target variable varies formulation of questions that would be better left unasked.
 PROBLEM FORMULATION AND FAIRNESS

But, this is a strangely unhelpful synthesis: while according The emerging field of critical data studies has brought
agency to the person or people who might formulate the similar insights to data science: data scientists do not just
problem, it simultaneously imparts overwhelming influence to apply algorithms to data, but work with algorithms and data—
the affordances of data science. The effort of getting the iteratively and often painstakingly—aligning the two together
question to work as a data science problem drops out entirely, in meaningful ways. Data science work, Passi and Jackson
even though this process is where the question actually and argue [35:2439], “is not merely a collection of formal and
ultimately takes shape. The issues of genuine concern—how mechanical rules, but a situated and discretionary process
universities arrive at a workable notion of student quality, requiring data analysts to continuously straddle the
how they decide on optimizing for competing variables competing demands of formal abstraction and empirical
(student quality, financial burden, diversity, etc.), how the contingency.” Algorithmic results embody specific forms of
results are put to work in one of many possible ways—are left “data vision”—rule-based, as opposed to rule-bound,
largely out of view. The indeterminacy of the process, where applications of algorithms, necessitating judgment-driven
many of the ethical issues are actually resolved, disappears. work “to apply and improvise around established methods and
Problem formulation in practice. While a focus on the tools in the wake of empirical diversity” [35:2436].
work of problem formulation in real-world applied settings Data science requires “thoughtful measurement, […]
has the potential to make visible the plethora of actors and careful research design, […and] creative deployment of
activities involved in data science work, it has not been the statistical techniques” [21:80] to identify units of
focus of much empirical inquiry to date. We still know very measurement, clean and process data, construct working
little about the everyday practice of problem formulation. In models, and interpret quantified results [4,5,19,32,34–36].
this paper, we attempt to fill this gap. How and why are Subjective decision making is necessary throughout the
specific questions posed? What challenges arise and how are process. Each of these practical choices can have profound
they resolved in everyday practice? How do actors’ choices ethical implications [10,13,24,31,40], of which data scientists
and decisions shape data science problem formulations? are sometimes well aware. Their everyday work is shot
Answers to these questions, we argue, can help us to better through with “careful thinking and critical reflection” [1:23].
understand data science as a practice, but also the origin of the Neff et al. [33], through ethnographic work on academic data
qualities of a data science project that raise normative science research, show how data scientists often
concerns. As researchers work to unpack the normative values “acknowledge their interpretive contributions” and “use data
at stake in the uses of data science, we offer an ethnographic to surface and negotiate social values.” Data, the authors
account of a special financing project for auto lending to make argue, are the starting, and not the end, points in data science.
visible the work of problem formulation in applied contexts. In academic and research settings—the contexts that
In so doing, we show how to trace the ethical implications of inform most of our current understanding of data science—the
these systems back to the everyday challenges and routine work of data science comes across mainly as the work of data
negotiations of data science. scientists. Data science projects in applied corporate settings,
In the following sections, we first situate the paper in a however, are inherently collaborative endeavors—a world as
longer history attending to the practical dimensions of data much of discretion, collaboration, and aspiration as of data,
science, specifically the task of problem formulation. We then numbers, and models. In such projects, several actors work
describe our research site and methodology, before moving to together to not only make sense of data and algorithmic
the empirical case-study. We conclude by discussing the results but also to negotiate and resolve practical problems.
implications of our findings, positioning the practical work of Passi and Jackson [36:18–19], through an ethnography of a
problem formulation as an important site for normative corporate data science team, describe how specific issues with
investigation and intervention. data, intuition, metrics, and models pose challenges for
corporate data science work, and how organizational actors
2 BACKGROUND collaborate, through specific strategies, to manage these
problems “in the service of imperfect but ultimately pragmatic
Our understanding of the role of problem formulation in data
and workable forms of analysis.” As the authors conclude,
science work draws from a long line of research within the
“[p]roject managers, product designers, and business analysts
history and sociology of science that describes how scientific
are as much a part of applied real-world corporate data science
methods are not just tools for answering questions, but in fact
as are data scientists” [36:16].
influence the kind of questions we ask and the ways in which
These strands of research call attention to the role of the
we define and measure phenomena [8,25,27,37]. Through
work of problem formulation within data science. The
different methods, scientists “mathematize” [29,30] the world
relationship between formulated problems and the data we
in specific ways, producing representations that are both
choose to address them is not a one-way street—data are not
contingent (i.e., they change with a change in methods) and
merely things to answer questions with. Instead, the very
real (i.e., they provide actionable ways to analyze the
formulations of data-driven problems (i.e., the kind of
world). Our practical understanding of a given phenomenon
questions we can and do ask) are determined by contingent
is contingent on the data we choose to represent and measure
aspects such as what data are available, what data we consider
it with.
relevant to a phenomenon, and what method we choose to
 PROBLEM FORMULATION AND FAIRNESS

process them. Problem formulation is as much an outcome of approach grows directly out of the earliest KDD writing.
our data and methods as of our goals and objectives. Indeed, Fayyad, Piatetsky-Shapiro, and Smyth [14] make a point of
defining the data science problem is not only about making saying that “data mining is a legitimate activity as long as one
the data science process fit specific and specifiable objectives understands how to do it,” suggesting that there is a particular
but also making the objectives fit the data science process. way to go about mining data to ensure appropriate results.
Data miners have long grappled with the role of human Indeed, the main impetus for developing process models were
judgment and discretion in their practice. The field of fears of mistakes, missteps, and misapplications, rather than a
Knowledge Discovery in Databases (KDD)—an important simple desire to explicate what it is that data miners do. As
predecessor to what we now call data science—emerged to Kurgan and Musilek [26] explain, the push “to formally
address how choices throughout the data mining process structure [data mining] as a process results from an
could be formalized in so-called process models. observation of problems associated with a blind application of
[data mining] methods to input data.” Notably, CRISP-DM,
Knowledge Discovery in Databases like the earlier models that preceded it in the academic
The iterative process of applied data mining. While KDD literature [14–16], emphasized the iterative nature of the
is commonly associated with data mining and machine process and the need to move back and forth between steps.
learning, the history of the field has less to do with innovations The attention to feedback loops and the overall dynamism of
around these techniques than with the process that surrounds the process were made especially evident in the widely
their use. Dissatisfied with a lack of applied research in reproduced visual rendering of the process that adopted a
artificial intelligence, scholars and practitioners founded the circular form to stress cyclicality [26].
new sub-field to draw together experts in computer science, Negotiated, not faithful, translations. Business
statistics, and data management who were interested and understanding, the first step in the CRISP-DM model, is
proficient in the practical applications of machine learning perhaps the most crucial in a data mining project because it
[see: 18]. The significance of this move owed to a shift in the involves the translation of an amorphous problem (a high-
field’s professional focus, not to a change in the substance of level objective or a business goal) into a question amenable to
its computational techniques. When KDD established itself as data mining. CRISP-DM describes this step as the process of
an independent field, 2 it also instituted a method for applying “understanding the project objectives and requirements from
machine learning to real-world problems—the KDD process, a business perspective [and] then converting this knowledge
consisting of a set of computational techniques and specific into a data mining problem definition” [7:10]. This process of
procedures through which questions are transformed into ‘conversion,’ however, is underspecified in the extreme.
tractable data mining problems [15,16]. Although the terms Translating complex objectives into a data mining problem is
KDD and data mining are now used interchangeably—if they not self-evident: “a large portion of the application effort can
are used at all—the original difference between the two is go into properly formulating the problem (asking the right
telling. While data mining referred exclusively to the question) rather than into optimizing the algorithmic details
application of machine learning algorithms, KDD referred to of a particular data-mining method” [14:46]. Indeed, the open-
the overall process of reworking questions into data-driven endedness that characterizes such forms of translation work is
problems, collecting and preparing relevant data, subjecting often described as the ‘art’ of data mining [12]. Recourse to
data to analysis, and interpreting and implementing results. such terms reveals the degree to which the creativity of the
The canon of KDD devoted extensive attention not only to the translation process resists its own translation into still more
range of problems that lend themselves to machine learning specific parts and processes (i.e., it is artistic only insofar as it
but also to the multi-step process by which these problems can resists formalization). But it also highlights the importance of
be made into practicable instances of machine learning. In this initial task in determining the very possibility of mining
their seminal paper, Fayyad, Piatetsky-Shapiro, and Smyth, for data for some purpose.
example, insist on the obvious applicability of data mining CRISP-DM and other practical guidance for data miners or
while paradoxically attempting to explain and advocate how data scientists [see: 6,26] tend to describe problem formulation
to apply it in practice—that is, how to make it applicable [14]. mainly as part of a project’s first phase—an initial occasion
KDD covered more than just a set of computational for frank conversations between the managers who set
techniques; it amounted to a method for innovating and strategic business goals, the technologists that manage an
executing new applications. organization’s data, and the analysts that ultimately work on
The focus on process led to the development of a series of data. Predictably, those involved in data science work face the
process models—formal attempts to explicate how one difficult challenge of faithful translation—finding the
progresses through a data mining project, breaking the correct mapping between, say, corporate goals, organizational
process into discrete steps [26]. The Cross Industry Standard data, and computational problems. Practitioners themselves
Process for Data Mining (CRISP-DM) [7], the most widely have long recognized that even when project members reach
adopted model, seems to simultaneously describe and consensus in formulating the problem, it is a negotiated
prescribe the relevant steps in a project’s lifecycle. Such an translation—contingent on the discretionary judgments of

2 Semi-annual workshops started in 1989. Annual conferences began in 1995.

 PROBLEM FORMULATION AND FAIRNESS

various actors and further impacted by the choice of methods, rounds of qualitative analysis. In our analysis, we coded the
instruments, and data. data in two rounds, focusing on the identification of key
These insights speak to the conditions that motivate data categories, themes, and topics as well as the relation between
science projects in a way that escapes the kind of technological them in the data. While we focus on a specific corporate data
determinism or data imperative that pervades the current science project in this paper, we observed similar dynamics
discourses—as if the kinds of questions that data science can across several other projects. We chose this case because the
answer are always already evident. Getting the automation of work of problem formulation was particularly salient in this
machine learning to return the desired results paradoxically project.
involves an enormous amount of manual work and subjective
judgment [3]. The work of problem formulation—of iteratively 4 CASE-STUDY: SPECIAL FINANCING
translating between strategic goals and tractable problems—is
CarCorp, a DataVector subsidiary, collects special financing
anything but self-evident, implicated with several practical
data: information on people who need car financing but have
and organizational aspects of data science work. As Hand
either low/bad credit scores (between 300-600) or limited
points out, “[t]extbook descriptions of data mining tools […]
credit histories. The company’s clientele mainly consists of
and articles extolling the potential gains to be achieved by
auto dealers who pay to receive this data (called lead data)
applying data mining techniques gloss over [these]
that include information such as name, address, mortgage, and
difficulties” [23:8].
employment details (sometimes even the make of the desired
In the following two sections, we look at the work of data
automobile). The company collects lead data primarily online:
science that is traditionally glossed over. We first describe our
people who need special financing submit their data so that
research site and methods before moving on to the empirical
interested dealers can contact them. People requiring special
case-study through which we show how the initial problem
financing face several challenges ranging from the lack of
formulation comprises a series of elastic translations—a set of
knowledge about available credit sources to difficulties in
placeholder articulations that is susceptible to change as the
negotiating interest rates. As liaisons between borrowers and
project moves through its many iterations.
lenders, companies such as CarCorp and its affiliates act as
important, sometimes necessary, intermediaries for people
3 RESEARCH SITE AND METHODS requiring special financing. CarCorp serves several dealers
This paper builds on six months of ethnographic fieldwork across the country.4 Few dealers collect their own lead data as
with DataVector3, a multi-billion-dollar US-based e-commerce the money, effort, and technical skills required to do so is
and new media organization. Established in the 1990s, enormous. This is a key reason why dealers pay companies
DataVector owns several companies in domains such as health such as CarCorp to buy lead data.
and automotive. Many of these are multi-million-dollar CarCorp’s technology development and project manager
companies with several thousand clients each. DataVector has Brian wanted to leverage data science to “improve the quality”
a core data science team based on the west coast of the United of leads. Improving lead quality, Brian argued, will ensure that
States that works with companies across different domains. existing dealers do not churn (i.e., they continue to give their
There are multiple teams of data engineers, software business to CarCorp).
developers, and business analysts, both at DataVector and its Brian (project manager): “The main goal [is] to improve
subsidiaries. One of us worked as a data scientist with the the quality of our leads for our customers. We want to give
organization’s core data science team between June and actionable leads. […] That is what helps us make money,
November 2017, serving as the lead scientist on two corporate makes customers continue to use our services” (Interview,
November 1, 2017).
data science projects (not reported in this paper) and
participating in many others. During ethnographic research, Initial discussions between the business and data science
the data science team had eight to eleven members (including teams revolved around two themes: (a) defining lead “quality”
one of the authors). The team is headed by Cliff—DataVector’s and (b) finding ways to measure it. Defining lead quality was
Director of Data Science with 30+ years of industry experience not straightforward. There were “many stakeholders with
in major technology firms. Cliff and the team report directly different opinions about leads” (ibid.). Some described lead
to Bill—DataVector’s Chief Technology Officer with 20+ years quality as a function of a lead’s salary data, while some argued
of experience in the technology industry. that a lead was good if the dealer had the lead’s desired car in
During the six-month period, one of us conducted 50+ their inventory. Everyone on the business team, however,
interviews with data scientists, product managers, business agreed on one thing—as CarCorp’s business analyst Ron put
analysts, project managers, and company executives and it: a “good” lead provided business to the dealer.
produced 400+ pages of fieldwork notes and 100+ Ron (business analyst): “The business team has been
photographs. Interviews and fieldwork data were transcribed talking about [lead quality] for a long time. […] We have
narrowed down the lead quality problem to how likely is
and coded according to the principles of grounded-theory
someone to purchase or to be able to finance a car when
analysis [20,41], inductively analyzing data through several

3Organization, personnel, and project names in this paper have been replaced 4 The exact number is omitted to preserve company anonymity.
with pseudonyms to preserve participant anonymity.
 PROBLEM FORMULATION AND FAIRNESS

you send them to that dealer?” (Interview, November 8, score from credit bureaus such as Equifax. The Fair Credit
2017). Reporting Act (FCRA) prohibits CarCorp from getting a lead’s
Lead “quality” was equated with lead “financeability.” It was, exact credit score from credit bureaus without a lead’s explicit
however, difficult to ascertain financeability. Different dealers consent. Leads have no reason to authorize CarCorp to
had different special financing approval processes. A lead retrieve their credit data because the company does not make
financeable for one dealer can be, for various reasons, lending decisions; it only collects information about a lead’s
unfinanceable for another. The goal thus was to determine interest in special financing. CarCorp had to rely on either
dealer-specific financeability (i.e., predicting which dealer leads’ self-reported credit scores that were collected by a few
was most likely to finance a lead). The teams settled on the affiliates or on credit scores provided as part of lead data
following definition of “quality”: a good lead for a dealer was bought from third-party lending agencies.5 Business affiliates
a lead financeable for that dealer. This, in turn, framed the and third-party agencies provide credit scores in the form of
problem as one of matching leads to dealers that were most an approximate range (e.g., 476-525). CarCorp had hoped that
likely to finance them. this data would help them to, for example, differentiate
CarCorp had a large amount of historical lead data. In 2017 between a subset of leads that appeared identical but exhibited
alone, the company had processed close to two million leads. different financeability.
CarCorp, however, had relatively less data on which leads had Ron (business analyst): “Two individuals [with] the
been approved for special financing (let alone data on why a same age, same income, same housing payment, same
lead was approved). The business team asked the data science everything […] could have wildly different credit scores.
team to contact data engineers to identify and assess the [...] You have those two, and you send them to the same
dealer. From our perspective, lead A and B are […] maybe
relevant data sources. The data science team, after not exactly [the] same, but close. [...] But, the dealer can
investigating the data sources, however, declared that there finance person A, and they cannot finance person B [...] So,
wasn’t enough data on dealer decisions—without adequate when they [dealers] are evaluating […] whether they
data, they argued, it was impossible to match leads with would renew their product with us, if we had sent them a
bunch from bucket B, and none from A, they are likely to
dealers. Few dealers in CarCorp’s network shared their churn. But, we have no way of knowing who is in bucket
approval data with the company. The scarcity of this data A and who is in bucket B. […] You can have two individuals
stemmed from the challenge of creating business incentives who measure the same on data points, and they have two
for dealers to provide up-to-date feedback. The incentives for different credit scores.” (Interview, November 8, 2017).
dealers to share information about their approval process with It is not surprising that a lead with a credit score greater than
CarCorp were too attenuated. another lead is more likely to secure special financing (even
While the data science team instructed the business team when the leads are otherwise identical). Credit score data is a
to invest in the collection of up-to-date data on dealer significant factor in special financing approval processes.
decisions, further discussions ensued between the two teams CarCorp had this data for about ~10% of their leads (~100,000).
to find alternate ways to predict dealer-specific financeability While business analysts found it challenging to predict credit
using the data that happened to be available already. In score ranges from the other features using traditional
debates over the utility of the available data, business analysts statistical analyses, adding credit score ranges in as an
and data scientists, however, voiced several concerns ranging additional feature did improve the company’s ability to predict
from inconsistency (e.g., discrepancies in data values) to lead financeability. The business team wondered if it was
unreliability (e.g., distrust of data sources). Business analyst possible to use data science to predict credit score ranges for
Ron, for instance, was wary of the multiple ways in which the the remaining 90% of the leads (i.e., perhaps machine learning
data was collected and generated: could work where traditional analysis had failed). If successful,
Ron (business analyst): “The entire complex lead credit scores could help ascertain lead financeability—a
ecosystem […] to outsiders does not make any sense. financeable lead was a lead with a high credit score.
[…Some data] came from an affiliate. […] They give a credit The data science team’s attempt to assess if they could
score range for a bunch of those leads. So, not exactly the predict credit scores, however, faced a practical challenge. As
score, but they could say ‘this person is between 450-475,
and 525-550,’ different buckets. [...] Never realistic, but we mentioned above, credit score data received from affiliates
pay money, and we have this data. [...] We [also] generate took the form of ranges (e.g., 476-525), not discrete numbers.
leads organically [online], then there are leads that we [buy Different affiliates marked ranges differently. For example,
from third-parties]. […] Different pieces of information one affiliate may categorize ranges as 476-525, 526-575, etc.,
[are] appended to those leads depending on where it came
from” (Interview, November 8, 2017). while another as 451-500, 501-550, etc. It was not possible to
directly use the ranges as class labels for training as the ranges
Only a few CarCorp affiliates augmented lead data with
overlapped. The data science team first needed to reconcile
additional information such as credit scores. Dealers run
different ranges.
background checks on leads (with their consent) as part of the
As data scientist Alex started working to make the ranges
financing procedure and in the process get a lead’s exact credit
consistent, business analysts came up with a way to make this

5 Third-party lending agencies offer services to help put borrowers in touch with

multiple lenders. These agencies get consent from people to perform a soft-pull
on their official credit reports from credit bureaus such as Equifax.
 PROBLEM FORMULATION AND FAIRNESS

process easier. Pre-existing market analysis (and, to some Alex: We will have to buy it first for me to know if it helps
extent, word-of-mouth business wisdom) indicated that with prediction.
having a credit score higher than 500 greatly increased a lead’s Cliff (director of data science): Check out the online info
likelihood of obtaining special financing approval. This piece on it, and then you and Marcus can figure it out (Fieldwork
Notes, June 16, 2017).
of information had a significant impact on the project. With
500 as the crucial threshold, only two credit score ranges were Without access to the data, Alex argued, it was not possible to
now significant: below-500 and above-500. Alex did not need clearly know its usefulness. If the data was not going to be
to figure out ways to reconcile ranges such as 376-425 and 401- helpful, however, it made no sense to buy it—the decision
450 but could bundle them in the below-500 category. The needed to be made based on the available description on
above-500 credit score range could act as the measure of Experian’s website. They eventually ended up not investing in
financeability—a financeable lead was a lead with a credit it. Even after analyzing the dataset’s description, Alex was not
score above-500. The matching problem (which leads are convinced that the data would increase model performance.
likely to get financed by a dealer) was now a classification Two months later, the project was halted in the absence of
task (which leads have a credit score of over 500). actionable progress.
Decreasing the number of classes to two helped attenuate the Different actors justified the project’s seeming failure in
difficulty of reconciling different ranges but did not help to different ways. Data scientist Alex felt that the data was the
circumvent it. culprit. Business analyst Ron felt that perhaps the business
team unreasonably expected “magic” from data science. For
Alex (data scientist): If the credit score is below 500, the
dealer will simply kill the deal. […] The problem is there him, the culprit was the nature of the problem itself:
are too many records in the 476-525 range. […] This makes Ron (business analyst): “[It is a] selection-bias problem.
it difficult (Fieldwork Notes, June 13, 2017). We are not dealing with a random sample of the
Leads in the 476-525 range were an issue because this range population. […] These individuals [...] why are they
submitting a lead with us? Because it was not easy for them
contained not only below-500 leads, but also above-500 leads. to get financed. By definition, our population is people
Making mistakes close to the decision boundary is especially with at least not great credit, and usually bad credit. Why
consequential for special financing where you want to find do you have bad credit? There is like one reason why you
people just above the threshold. Alex tried many ways to have good credit. There are a thousand reasons why you
have […] bad credit. [...] If you show me someone with
segregate the leads in this range but the models, according to good credit, I will show you they pay bills on time, have a
him, didn’t work. Their accuracy, at best, was slightly better steady income, etc. If you show me someone with bad
than a coin flip. Alex attributed the model’s bad performance credit, they can have a gambling problem, they can be
not only to the presence of leads in the 476-525 range, but also divorced, they could […] prove income now, but maybe it
has been unstable in the past, and we have no way of
to the limited number of available features (i.e., the data did
knowing that. There are literally thousands of reasons that
not present sufficient information for the model to aren’t that capture-able” (Interview, November 8, 2017).
meaningfully differentiate between leads). While the in-house
Business analyst Ron described the failure not in terms of the
lead dataset was a good starting point, the data scientists knew
initially articulated business goal but in terms of the project’s
that accurately classifying leads would require not only
current data science problem formulation—not the difficulty
creative ways to work with available data, but also a wide
of defining the quality of a lead, but the challenge of
variety of data. They had already been scouting for external
classifying leads with scores in a specific part of the credit
datasets to augment the in-house lead dataset.
score spectrum. He believed it was possible to classify people
Director of data science Cliff had tasked data science
with high/low credit scores on the full 300-850 credit score
project manager Marcus with the work of finding third-party
spectrum (e.g., differentiating between a person with a 750
datasets that could help with classification. Their approach
score and a person with a 450 score). He argued, however, that
was first to use freely available data to see “how far we get”
CarCorp’s focus on the special financing population meant
before buying paid data from companies such as Experian.6
that the goal was not to classify high/low scores on the full
Free, yet reliable, datasets, however, were hard to come by.
credit spectrum but to demarcate between different kinds of
Marcus found only a few datasets—Internal Revenue Service
low scores on one side of the spectrum (roughly between 300
(IRS) zip-code-level data on features such as income range, tax
and 600). Note how Ron, a business analyst, describes the
returns, and house affordability (i.e., how much an owner
project’s failure in relation to a specific definition of lead
might be able to pay for a property). Data scientist Alex tried
“quality”—it was difficult to know which leads were above or
each dataset but declared that none improved model
below the credit score threshold of 500. The project was halted
performance. Members of the data science team wondered if it
when developing an accurate model based on this definition
was worth investing in high-quality paid datasets.
proved impossible. The business and data science team could
Alex: I used all the data, but the model does not converge. not figure out any other way to formulate the problem at this
Marcus: What about Experian data? We can get it if you stage with the data they had.
think it will help.

6 For example, datasets such as Experian’s Premier Aggregated Credit Statistics [Link]). Even such data only contained aggregated information on credit
([Link] scores and ranges, and not lead-specific credit scores (which required consent).
 PROBLEM FORMULATION AND FAIRNESS

5 DISCUSSION This is not to argue that the high-level goal to improve lead
quality, at some point, transformed into a completely
Through the above description, we see how the data science
different objective. Instead, it shows that the translation
problem was formulated differently at different points in the
between high-level goals and tractable data science problems
project based on two different sets of targets variables and
is not a given but a negotiated outcome—stable yet elastic.
their possible proxies.
Throughout the project, the goal of improving lead quality
Proxy #1: Dealer Decisions. The business team initially
remains recognizably similar but practically different,
described the project goal as the improvement of lead
evident in the different descriptions, target variables, and
quality—a formulation of what the business team thought the
proxies for lead quality. Each set of target variables and
dealers wanted. Note that this goal was in turn related to the
proxies represents a specific understanding of what a lead’s
broader objective of minimizing churn rate—a formulation
quality is and what it means to improve on it. The quality of a
of what CarCorp itself wanted. In this way, the problem
lead is not a preexisting variable waiting to be measured, but
specification was just as much about keeping clients as it was
an artifact of how our actors define and measure it.
about satisfying their business objectives. These high-level
The values at stake in problem formulation. Scholars
goals impacted the actors’ initial understanding of the
concerned with bias in computer systems have long stressed
project’s goal—the quality of leads was seen in relation to
the need to consider the original objectives or goals that
dealers and CarCorp’s own success. CarCorp decided that if
motivate a project, apart from any form of bias that may creep
dealers could finance a lead, it was a good lead. The fact that
into the system during its development and implementation
different dealers had different special financing approval
[17]. On this account, the apparent problem to which data
processes further impacted the contingent relationship
science is a solution determines whether it happens to serve
between quality, dealers, and financeability: if a lead was
morally defensible ends. Information systems can be no less
financeable by a specific dealer, it was a good lead for that
biased than the objectives they serve.
dealer. The data science problem, therefore, became the task
These goals, however, rarely emerge ready-formed or
of matching leads with dealers that were likely to finance
precisely specified. Instead, navigating the vagaries of the data
them.
science process requires reconceiving the problem at hand and
Proxy #2: Credit Score Ranges. Data available to support
making it one that data and algorithms can help solve. In our
the use of dealer decisions as a proxy, however, were limited.
empirical case, we do not observe a data science project
While business analysts did not fully understand how dealers
working in the service of an established goal, about which
made decisions, they acknowledged, based on market
there might be some normative debate. Instead, we find that
research, the import of credit scores in the special financing
the normative implications of the project evolve alongside
approval process—leads with scores higher than 500 were
changes in the different problem formulations of lead quality.
highly likely to get special financing. Credit scores thus
On the one hand, for the proxy of dealer-decisions, leads
became a proxy for a dealer’s decision, which was itself a
are categorized by their dealer-specific financeability—a lead
proxy for a lead’s financeability, which was, by extension, a
is only sent to the dealer that is likely to finance them. In
proxy for a lead’s quality—indeed, a chain of proxies. The data
formulating the problem as a matching task, the company is
science problem thus became the task of classifying leads into
essentially catering to dealer preferences. This approach will
below- and above-500 classes.
recommend leads to dealers that align with the preferences
Problem formulation is a negotiated translation. At
expressed in dealers’ previous decisions. In this case, lead
face value, the relationship between the project’s high-level
financeability operates on a spectrum. Financeability emerges
business goal (improving lead quality) and its two different
as a more/less attribute: each lead is financeable, some more
problem formulations (the two sets of target variables and
than others depending on the dealer. Effectively, each lead has
their proxies) may seem like a one-to-many relation—different
at least a chance of being sent to a dealer (i.e., the dealer with
translations of, in effect, the same goal. Such an
the highest probability of financing a lead above some
understanding, however, fails to account not only for the
threshold).7
amorphous nature of high-level goals (i.e., the difficulty of
On the other hand, for the credit-score proxy, leads are
defining the quality of a lead), but also for the project’s
categorized into two classes based on their credit score ranges
iterative and evolving nature (i.e., problem formulations are
and only leads with scores greater than 500 are considered
negotiated, dependent on, for instance, actors’ choice of
financeable. In formulating the problem as the task of
proxy). In our case, actors equated (in order): lead quality with
classifying leads above or below a score, the company reifies
financeability, financeability with dealer decisions, and dealer
credit scores as the sole marker for financeability. Even if
decisions with credit score ranges. Each of these maneuvers
dealers had in the past financed leads with credit scores less
produced different formulations of the objective, in turn
than 500, this approach only recommends leads with scores
impacting actors’ articulation and understanding of the
higher than 500, shaping dealers’ future financing practices. In
project’s high-level goal (as seen, for instance, in the way
this approach, financeability operates as a binary variable: a
business analyst Ron ultimately accounts for the project’s
lead is financeable only if its credit score is higher than 500.
failure).

7 Of course, depending on the threshold, some leads will never be sent.

 PROBLEM FORMULATION AND FAIRNESS

Consequently, leads in the below-500 category may never see Yet, as we show in this paper, normative implications of
the light of day, discounted entirely because the company data science systems do in fact find their roots in problem
believes that these leads are not suitable for dealers. formulation work—the discretionary judgments and practical
Different principles; different normative concerns. work involved in translations between high-level goals and
Seen this way, the matching version of the problem tractable problems. Each translation galvanizes a different set
formulation may appear normatively preferable to the of actors, aspirations, and practices, and, in doing so, creates
classification version. But, is this always true? If we prioritize opportunities and challenges for normative intervention—
maximizing a person’s lending opportunities, the matching upstream sites for downstream change. As Barocas et al. [2:6]
formulation of the problem may seem better because it argue:
increases a lead’s chances of securing special financing. If, “A robust understanding of the ethical use of data-driven
however, we prioritize the goal of mitigating existing biases in systems needs substantial focus on the possible threats to
lending practices (i.e., of alleviating existing dealer biases), the civil rights that may result from the formulation of the
classification problem formulation may come across as the problem. Such threats are insidious, because problem
formulation is iterative. Many decisions are made early and
better alternative because it potentially encourages dealers to quickly, before there is any notion that the effort will lead
consider leads different from those they have financed in the to a successful system, and only rarely are prior problem-
past. Through the two scenarios, we see how proxies are not formulation decisions revisited with a critical eye.”
merely ways to equate goals with data but serve to frame the If we wish to take seriously the work of unpacking the
problem in subtly different ways—and raise different ethical normative implications of data science systems and of
concerns as a result. intervening in their development to ensure greater fairness,
It is far from obvious which of the two concerns is more we need to find ways to identify, address, and accommodate
serious and thus which choice is normatively preferable— the iterative and less visible work of formulating data science
shifting our normative lens alters our perception of fairness problems—how and why problems are formulated in specific
concerning the choice of target variables and proxies. In this ways.
paper, we have demonstrated how approaching the work of
problem formulation as an important site for investigation 6 CONCLUSION
enables us to have a much more careful discussion about our
own normative commitments. This, in turn, provides insights In this paper, we focused on the uncertain process by which
into how we can ensure that projects align with those certain questions come to be posed in real-world applied data
commitments. science projects. We have shown that some of the most
Always imperfect; always partial. Translating strategic important normative implications of data science systems find
goals into tractable problems is a labored and challenging their roots in the work of problem formulation. The attempt
process. Such translations do necessary violence to the world to make certain goals amenable to data science will always
that they attempt to model, but also provide actionable and involve subtle transformations of those objectives along the
novel ways to address complex problems. Our intention to way—transformations that may have profound consequences
make visible the elasticity and multiplicity of such translations for the very conception of the problem to which data science
was thus not to criticize actors’ inability to find the perfectly has been brought to bear—and what consequently appear to
faithful translation. Quite the opposite: we recognize that be the most appropriate ways of handling those problems.
translations are always imperfect and partial, and wanted to Thus, the problems we solve with data science are never
instead shift the attention to the consequences of different insulated from the larger process of getting data science to
translations and the everyday judgments that drive them. return actionable results. As we have shown, these ends are
Our actors, however, did not explicitly debate the ethical very much an artifact of a contingent process of arriving at a
implications of their own systems—neither in the way we, as successful formulation of the problem, and they cannot be
researchers, have come to recognize normative issues, nor in easily decoupled from the process at arriving at these ends. In
the way we, as authors, have analyzed the implications of their linking the normative concerns that data science has provoked
problem formulations in this paper. Practical and to more nuanced accounts of the on-the-ground process of
organizational aspects such as business requirements, the undertaking a data science project, we have suggested new
choice of proxies, the nature of the algorithmic task, and the objects for investigation and intervention: which goals are
availability of data impact problem formulations in much posed and why; how goals are made into tractable
more significant and actionable ways than, for instance, the questions and working problems; and, how and why
certain problem formulations succeed.
practitioners’ normative commitments and beliefs. Indeed, our
analysis of the empirical case makes visible how aspects such ACKNOWLEDGMENTS
as analytic uncertainty and financial cost impact problem
The funding for this research was provided by the National
formulations. For example, the high cost of datasets coupled
Science Foundation grant CHS-1526155, and the Harvard-MIT
with the challenge of assessing the data’s efficacy without
Ethics and Governance of AI Initiative. We wish to thank our
using it made it particularly challenging for actors to leverage
anonymous reviewers for their feedback, and Shira Mitchell,
additional sources of information.
Foster Provost, Malte Ziewitz, and members of Cornell
 PROBLEM FORMULATION AND FAIRNESS

University’s Artificial Intelligence, Policy, and Practice (AIPP) [29] Michael Lynch. 1985. Discipline and the Material Form of Images: An
Analysis of Scientific Visibility. Soc. Stud. Sci. 15, 1 (1985), 37–66.
initiative for comments on an earlier version of this work. [30] Michael Lynch. 1988. The externalized retina: Selection and
mathematization in the visual documentation of objects in the life sciences.
Hum. Stud. 11, (1988), 201–234.
REFERENCES [31] Daniel A. McFarland and H. Richard McFarland. 2015. Big Data and the
[1] Solon Barocas and Danah Boyd. 2017. Engaging the Ethics of Data Science danger of being precisely inaccurate. Big Data Soc. 2, 2 (2015).
in Practice. Commun. ACM 60, 11 (2017), 23–25. [32] Michael Muller, Shion Guha, Eric P.S. Baumer, David Mimno, and N Sadat
[2] Solon Barocas, Elizabeth Bradley, Vasant Honavar, and Foster Provost. Shami. 2016. Machine Learning and Grounded Theory Method:
2017. Big Data, Data Science, and Civil Rights. A white paper prepared for Convergence, Divergence, and Combination. In Proceedings of the 19th
the Computing Community Consortium committee of the Computing International Conference on Supporting Group Work, 3–8.
Research Association. Retrieved from [Link] [33] Gina Neff, Anissa Tanweer, Brittany Fiore-Gartland, and Laura Osburn.
led-whitepapers/ 2017. Critique and Contribute: A Practice-Based Framework for Improving
[3] Solon Barocas and Andrew D. Selbst. 2016. Big Data’s Disparate Impact. Critical Data Studies and Data Science. Big Data 5, 2 (2017), 85–97.
104 Calif. Law Rev. 671, (2016). [34] Frank Pasquale. 2015. The Black Box Society: The Secret Algorithms that
[4] danah boyd and Kate Crawford. 2012. Critical Questions for Big Data: Control Money and Information. Harvard University Press, Cambridge,
Provocations for a cultural, technological, and Scholarly phenomenon. MA.
Information, Commun. Soc. 15, 5 (June 2012), 662–679. [35] Samir Passi and Steven J. Jackson. 2017. Data Vision: Learning to See
[5] Lawrence Busch. 2014. A Dozen Ways to Get Lost in Translation: Inherent Through Algorithmic Abstraction. In Proceedings of the 2017 ACM
Challenges in Large Scale Data Sets. Int. J. Commun. 8, (2014), 1727–1744. Conference on Computer Supported Cooperative Work and Social
[6] Kevin Daniel André Carillo. 2017. Let’s stop trying to be “sexy” – preparing Computing (CSCW ’17), 2436–2447.
managers for the (big) data-driven business era. Bus. Process Manag. J. 23, [36] Samir Passi and Steven J. Jackson. 2018. Trust in Data Science:
3 (2017), 598–622. Collaboration, Translation, and Accountability in Corporate Data Science
[7] Pete Chapman, Julian Clinton, Randy Kerber, Thomas Khabaza, Colin Projects. In Proceedings of the ACM on Human-Computer Interaction,
Shearer, and Rüdiger Wirth. 2000. CRISP-DM 1.0: Step by step data Vol. 2, CSCW, Article 136, 1–28.
mining guide. [37] Trevor J. Pinch and W. E. Bijker. 1984. The Social Construction of Facts and
[8] Harry M. Collins. 1985. Changing Order: Replication and Induction in Artefacts: or How the Sociology of Science and the Sociology of
Scientific Practice. Sage, London. Technology might Benefit Each Other. Soc. Stud. Sci. 14, 3 (August 1984),
[9] Jack Cook. 2009. Ethics of Data Mining. In Encyclopedia of Data 399–441.
Warehousing and Mining, John Wang (ed.). IGI Global, Hershey, 783– [38] Foster Provost and Tom Fawcett. 2013. Data Science for Business: What
788. You Need to Know About Data Mining and Data-Analytic Thinking.
[10] Morgan Currie, Britt S Paris, Irene Pasquetto, and Jennifer Pierre. 2016. The O’Reilly Media, Sebastopol, CA.
conundrum of police officer-involved homicides: Counter-data in Los [39] Matthew Quirk. 2005. The Best Class Money Can Buy. The Atlantic
Angeles County. Big Data Soc. 3, 2 (2016), 1–14. Monthly.
[11] Peter Danielson. 2009. Metaphors and Models for Data Mining Ethics. In [40] Gernot Rieder and Judith Simon. 2016. Datatrust: Or, the political quest for
Social Implications of Data Mining and Information Privacy: numerical evidence and the epistemologies of Big Data. Big Data Soc. 3, 1
Interdisciplinary Frameworks and Solutions, Ephrem Eyob (ed.). IGI (2016), 1–6.
Global, Hershey, 33–47. [41] Anselm Strauss and Juliet M. Corbin. 1990. Basics of Qualitative
[12] Pedro Domingos. 2012. A Few Useful Things to Know About Machine Research: Grounded Theory Techniques and Procedures. Sage, New
Learning. Commun. ACM 55, 10 (2012), 78–87. York.
[13] Paul Dourish and Edgar Gómez Cruz. 2018. Datafication and data fiction: [42] Marian Wang. 2013. Public Universities Ramp Up Aid for the Wealthy,
Narrating data and narrating with data. Big Data Soc. 5, 2 (2018), 1–10. Leaving the Poor Behind. ProPublica.
[14] Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth. 1996. [43] Marian Wang. 2014. How Exactly Do Colleges Allocate Their Financial
From Data Mining to Knowledge Discovery in Databases. AI Mag. 17, 3 Aid? They Won’t Say. ProPublica.
(1996), 37–54.
[15] Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth. 1996. The
KDD Process for Extracting Useful Knowledge from Volumes of Data.
Commun. ACM 39, 11 (November 1996), 27–34.
[16] William J. Frawley, Gregory Piatetsky-Shapiro, and Christopher J.
Matheus. 1992. Knowledge Discovery in Databases: an Overview. AI Mag.
13, 3 (1992), 57–70.
[17] Batya Friedman and Helen Nissenbaum. 1996. Bias in computer systems.
ACM Trans. Inf. Syst. 14, 3 (1996), 330–347.
[18] Mohamed Medhat Gaber (Ed.). 2012. Journeys to Data Mining:
Experiences From 15 Renowned Researchers. Springer, Berlin, Germany.
[19] Lisa Gitelman. 2006. Raw Data is an Oxymoron. MIT Press, MA.
[20] Barney Glaser and Anselm Strauss. 1967. The Discovery of Grounded
Theory: Strategies for Qualitative Research. Aldine Transactions,
Chicago.
[21] Justin Grimmer. 2015. We Are All Social Scientists Now: How Big Data,
Machine Learning, and Causal Inference Work Together. PS Polit. Sci.
Polit. 48, 1 (2015), 80–83.
[22] David Hand. 1994. Deconstructing Statistical Questions. J. R. Stat. Soc. Ser.
A (Statistics Soc. 157, 3 (1994), 317–356.
[23] David Hand. 2006. Protection or Privacy? Data Mining and Personal Data.
In Advances in Knowledge Discovery and Data Mining, 1–10.
[24] Lucas D. Introna. 2016. Algorithms, governance, and governmentality: On
governing academic writing. Sci. Technol. Hum. Values 41, 1 (2016), 17–
49.
[25] Bernward Joerges and Terry Shinn. 2001. A Fresh Look at Instrumentation
an Introduction. In Instrumentation Between Science, State and
Industry, Bernward Joerges and Terry Shinn (eds.). Springer Netherlands,
Dordrecht, 1–13.
[26] Lukasz A. Kurgan and Petr Musilek. 2006. A survey of Knowledge
Discovery and Data Mining process models. Knowl. Eng. Rev. 21, 1 (2006),
1–24.
[27] Bruno Latour and Steve Woolgar. 1985. Laboratory Life: The
Construction of Scientific Facts (2nd ed.). Princeton University Press,
Princeton.
[28] Kristian Lum and William Isaac. To predict and serve? Significance 13, 5 ,
14–19. DOI:[Link]