PDF Download

[Link]
24 December 2025
Total Citations: 104
Total Downloads: 9551
.
.
Latest updates: hps://[Link]/doi/10.1145/3442188.3445939

.
.
Published: 03 March 2021
.
.
.
RESEARCH-ARTICLE

.
Citation in BibTeX format
The Ethics of Emotion in Artificial Intelligence Systems

.
.
FAccT '21: 2021 ACM Conference
on Fairness, Accountability, and
LUKE STARK, Western University, London, ON, Canada Transparency
.
March 3 - 10, 2021
JESSE HOEY, David R. Cheriton School of Computer Science, Waterloo, ON, Canada Virtual Event, Canada
.
.
.
.
Conference Sponsors:
.
Open Access Support provided by: ACM
.
David R. Cheriton School of Computer Science
.
Western University
.
FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (March 2021)
hps://[Link]/10.1145/3442188.3445939
ISBN: 9781450383097
.
 The Ethics of Emotion in Artificial Intelligence Systems
Luke Stark Jesse Hoey
Faculty of Information and Media Studies David R. Cheriton School of Computer Science
University of Western Ontario University of Waterloo
London ON Canada Waterloo ON Canada
cstark23@[Link] jhoey@[Link]

ABSTRACT in science fiction narratives. Computational analyses of

psychological and behavioral data pertaining to human emotional
In this paper, we develop a taxonomy of conceptual models and expression have a surprisingly long history [31], an
proxy data used for digital analysis of human emotional expression underappreciated diversity of methods [16, 108], and an
and outline how the combinations and permutations of these increasingly critical role in social machine learning (ML)
models and data impact their incorporation into artificial applications [28, 114]. AI/ML technologies are frequently used by
intelligence (AI) systems. We argue we should not take computer social media platforms for modeling and predicting human
scientists at their word that the paradigms for human emotions they emotional expression as signaling interpersonal interaction and
have developed internally and adapted from other disciplines can personal preference [22]. In sectors including mental health care
produce ground truth about human emotions; instead, we ask how [20], personal health and wellness [28], education [124], hiring
different conceptualizations of what emotions are, and how they [129], automotive design [123], and national security [119]
can be sensed, measured and transformed into data, shape the emotion detection and analysis is a rapidly growing sector for
ethical and social implications of these AI systems. AI/ML systems [75].
CCS CONCEPTS
While the fairness, accountability, and ethical and social impacts of
• Computing methodologies ~ Artificial intelligence ~
ML/AI systems have become major topics of both public
Philosophical/theoretical foundations of artificial intelligence •
Social and professional topics ~ Professional topics ~ Computing discussion and academic debate [8, 13, 18, 35, 60, 81], the ethical
profession ~ Codes of ethics • Human-centered computing ~ dimensions of AI/ML used to analyze human affective and
Human computer interaction (HCI) ~ HCI theory, concepts and emotional expression have been largely under-theorized in these
models conversations [3, 27, 44, 75, 114]. Given the increasing ubiquity of
KEYWORDS these systems, the ethics of affect/emotion recognition, and more
broadly of so-called “digital phenotyping” [57] must play a larger
emotion, affect, artificial intelligence, AI, machine learning, ML,
role in current debates around the political, ethical and social
ethics, norms, Basic Emotion Theory, Action Control Theory,
affective computing, emotion AI, privacy, fairness, AI ethics dimensions of AI/ML.

ACM Reference format: Here we develop a taxonomy of the relevant conceptual models of
Luke Stark and Jesse Hoey. 2021. The Ethics of Emotion in Artificial
human emotion and of proxy data for emotional expression; we
Intelligence Systems. In Proceedings of ACM Conference on Fairness, then outline the ways the models of emotion and the proxy data
Accountability, and Transparency (FAccT’21). ACM, New York, NY, collected according to these models influence design decisions
USA, 12 pages. [Link] made by the technologists creating AI/ML systems, and how these
decisions raise broader questions about these technologies’ social
impacts. We do not take computer scientists at their word that the
1 Introduction paradigms for human emotions they have developed internally and
Speculative and science fiction is replete with questions regarding adapted from other fields should be taken naively ground truth:
the emotional lives of artificial beings. Yet contemporary machine instead, we ask how different conceptualizations of human
learning-driven artificial intelligence (AI) systems have a much emotions shape the ways human values are built into and expressed
narrower view of human emotion than the complex questions posed by AI/ML systems.

2 Definitions and Theories of Emotion

This work is licensed under a Creative Commons Attribution-NonCommercial-
ShareAlike International 4.0 License. Affect and emotion are concepts subject to intense debate across
FAccT '21, March 3–10, 2021, Virtual Event, Canada numerous academic fields [112] [9]. The shorthand term “emotion”
ACM ISBN 978-1-4503-8309-7/21/03.
is used to describe a compound phenomenon variously consisting
[Link]
of evaluative, physiological, phenomenological, expressive,

782
 FAccT’21, March 3–10, 2021, Virtual Event, Canada L. Stark and J. Hoey

behavioral, and mental components. Psychologist Jerome Kagan paradigm, the human body is a palimpsest on which to read the
describes emotion as comprised of a variety of interrelated human eruptions of emotion. Scarantino and de Sousa argue that the
phenomena: affect (or in Kagan’s terms, “a change in brain activity motivational tradition is particularly concerned with understanding
to select incentives”); feeling or sensation (“a consciously detected how emotions motivate human actions: in opposition to what he
change in feeling that has sensory qualities”); emotion proper considered the lack of common sense in the James-Lange theory of
(“cognitive processes that interpret and/or label the feeling with emotions as derived from physiological phenomena, philosopher
words”); and reaction (“a preparedness for, or display of, a John Dewey argued in 1895 that emotions were not just
behavioral response”) [133]. Affect theorist Deborah Gould experiences, but experiences with a purpose. The centrality of
describes affects as “nonconscious and unnamed, but nonetheless behaviorism in American psychology for much of the first half of
registered, experiences of bodily energy and intensity that arise in the 20th century pushed most questions of motivation to the
response to stimuli” [42], and emotion as “what from the potential backburner [107]. However, the notion of emotions as motivating
of [affective] bodily intensities gets actualized or concretized in the states reappeared in the 1950s and 60s with the work of American
flow of living” [42]. A mood, another term often mobilized in the psychologist Silvan Tomkins [122] on affect, and the subsequent
space of emotion tracking and quantification, is “a pervasive and development of Basic Emotion Theory (BET) by Tomkins’
sustained emotion that colors the perception of the world,” [74], students Paul Ekman [34] and Carrol E. Izard [55]. Tomkins argued
one that can pass quickly or stay for long periods of time. that innate affects, or primarily physiological changes in brain and
hormone activity, are hardwired into humans, identifying nine such
This plethora of definitions points to what the philosopher of basic affective programs: interest, enjoyment, surprise, fear, anger,
emotion Jesse Prinz describes as the dual concerns of the “problem distress, shame, contempt and disgust [104] (s.8 p.2).
of parts” and “the problem of plenty” in theories of emotion. The
“problem of parts” describes the challenge of determining what Tomkins’ students, in particular Ekman, carried these arguments
component or components of emotion, be they evaluative, further: in the 1970s, Ekman performed a series of comparative
physiological, phenomenological, expressive, behavioral, or behavioral experiment which he claimed proved certain basic
mental, are essential to its definition and detection in a particular emotions were reliably recognizable in facial expressions in
context. The “problem of plenty” asks, if multiple components are different populations around the world. As such, Ekman suggested,
essential to understanding emotions, how these various “there should be bodily signatures for each basic emotion
components hang together in practice [96]. As Prinz puts it, “the consisting of highly correlated and emotion-specific changes at the
Problem of Parts asks for essential components, and the Problem of level of facial expressions, autonomic changes and preset and
Plenty asks for an essential function of emotions in virtue of which learned actions” [104] (s.8 p.5). In other words, Basic Emotion
they may have several essential components” [96]. Conceptual Theory argues human emotional expression is universal, reliably
models of emotion tend to break down into different camps legible, and critically, difficult to falsify—because emotions are
foregrounding one or another of emotion’s “parts.” These broad understood as motivational drives deriving from biophysiological
camps include understanding emotion as especially grounded in a) processes, they are difficult to conceal from the expert eye. The
experienced feeling states, as b) evaluative signals connected to ethologist Alan Fridlund [39] has developed a notable alternative
other forms of human perception of social cues, and c) as paradigm for understanding emotion as motivational, known as the
intrinsically motivating drives [104]. Behavioral Ecology view: in this theory, all externalized forms of
human emotional expression are better understood as social
Affect and emotion are not the same phenomenon, yet in computer displays, always suggesting some sort of social motivation or
science the terms are at times treated interchangeably. This elision function (e.g. a performative “suggestion”) but which provide no
is always definitional (though sometimes inadvertent): human evidence regarding the interior mental or motivational states of the
emotion is understood by computer scientists largely through the expressor [68].
expression of biophysiological signals such as facial expression,
gait, or blood conductivity [16, 75, 93]. Developers of AI systems 2.2 Emotion as Evaluative Signal
have adapted this elision as a solution, if a potentially problematic A second tradition, grounded largely in the cognitive revolution of
one, to the Problems of Parts and Plenty. As we develop in Sections the 1960s [125], treats emotions either as directly constitutive of
4-6 below, the various conceptual models of emotion have cognitive states, or caused by them. Evaluative theories suggest that
incompatible ethical and social valences depending on which human emotions are one important category of evidence
“parts” are emphasized in technical systems, and on how designers underpinning human thought and action, and that emotions are
choose to reduce the polysemy of emotion for the convenience of largely contingent on social contexts. These evaluative theories
technical constraint and business exigency. define emotions as primarily cognitive phenomena, and as “being
(or involving) distinctive evaluations of the eliciting
2.1 Emotion as Motivating Drive circumstances” (s.2 p.4). Central to this view is that emotions have
A robust tradition in affective science subscribes to the view that “intentionality”: that humans direct our emotions at particular
emotions are “distinctive motivational states” and “internal causes objects (a view synthesized and popularized in the 1870s by the
of behaviors aimed at satisfying a goal” [104] (s.8 p.1). In this German philosopher Franz Brentano). Whether understood

783
 FAccT’21, March 3–10, 2021, Virtual Event, Canada L. Stark and J. Hoey

philosophically as either simple judgments or complex appraisals, evaluative perceptions with a distinctive phenomenology and the
the evaluative tradition understands emotion as in some way latter identifying emotions as evaluative feelings with a distinctive
connected to human judgment (though there is further broad debate intentionality” [104] (s.7 p.2). Prinz’s [96] perceptual theory of
as to of what “judgment” itself consists). In psychology and emotion is one such hybrid approach, treating emotions as
affective science, Scarantino and de Sousa, reviewing [6] and [66], evaluative perceptions grounded both in physiological affective
connect the evaluative tradition to the rise of appraisal theory. changes and in the particular social context of the object or person
Appraisal theory is concerned with developing “accounts of the being perceived. Sociologist Arlie Russell Hochschild, who
structure of the processes that extract significance from stimuli and developed the concept of “emotional labor,” likewise presents a
differentiate emotions from one another” [104] (s.6 p.5), though hybrid theory articulating emotions as a mix of physiological and
such differentiation does not conflict with understanding emotions social signals: in her words, “every emotion has a signal function”
as evaluative, experiential, or motivational per se. [50]. Affect Control Theory (ACT) [49, 110, 111] a form of
structural symbolic interactionism mapping so-called patterns of
2.3 Emotion as Felt Experience emotional salience (s.7.3, p.1)—or typical social evaluations of
A third tradition in the philosophy of emotion is to treat emotions affective response of actors and behaviors—is a third hybrid theory
as primarily experiential: as Scarantino and de Sousa put it, this incorporating elements of evaluative and experiential models. [70].
“Feeling Tradition takes the way emotions feel to be their most A Bayesian extension, BayesACT [108] combines this conceptual
essential characteristic and defines emotions as distinctive framework with a motivational model based in decision theory.
conscious experiences” [104] (s.2 p.4). In essence, this way of
understanding emotion likens it to other felt experiences such as
taste, pain, or other embodied sensations. “We feel,” wrote the
3 Proxy Data for Emotional Expression
nineteenth-century philosopher and psychologist William James, If the breadth of theories for understanding what emotions are is
“sorry because we cry, angry because we strike, afraid because we not daunting enough, the empirical evidence that outside observes
tremble, and not that we cry, strike, or tremble, because we are use to identify and understand emotion also varies widely. This
sorry, angry, or fearful” [58]. This view broadly typified evidence is often quantitative data. In the context of digital and
philosophical theories of the passions from Aristotle through to the artificial intelligence system, practitioners have used or suggested
nineteenth century, contrasting emotions with the discipline various types of proxy data for analysis of emotional expressions
typifying logical reasoning and reflection. In 1884, James proposed [63]; the decision to collect a particular proxy often depends on
a variation of this longstanding view, positing that emotions were a what conceptual model of emotion the designers of the underlying
specific kind of subjective experience [58]: under what became agent or system have adopted [113, 114]. These various proxies for
known as the James-Lange hypothesis, emotions were, “sensory emotional expression include physiological data [91], such as facial
feelings constituted by perceptions of changes in physiological expression [47], gait, or infrared emanations and haptic and
conditions relating to the autonomic and motor functions” (s.3 p.3). proprioceptive data (such as skin conductivity, blood flow, and
In this view, perception of emotions as such are derived from body velocity) [86]; audio data (such as the vocal tone and cadence)
physiological process (exemplified by James’ famous invocation [99] [20]; behavioral data collected over time [57]; and semantic
that we are afraid of a bear because we are impelled by reflex to run signifiers of emotional expression, (including written words and
away from it). graphic means such as emoji and emoticons) [2].

2.4 Hybrid Theories Historian of medicine Otniel Dror has expertly documented the
Basic Emotion Theory has been enormously influential since its origins of what he terms “emotion as number” in the late nineteenth
promulgation by Ekman in the 1970s, not least in recent work on century, through the fusion of empirical physiology, a homosocial
artificial intelligence, robotics, and computer science more broadly. culture of masculine expertise, and a desire to contrast medical
However, BET has also come under sustained critique from a studies of emotion with the mobilization of feeling by anti-
number of quarters, including from proponents of what might be vivisectionists [32]. The full genealogy of how this focus on
best termed “hybrid” theories of emotion mixing elements of the physiological signs became understood as standing in for—and in
experiential, evaluative, and motivational traditions. Both James some accounts, consisting of—human emotion in computer science
Russell [102, 103] and Lisa Feldman Barrett [9, 10] have is largely outside the scope of this paper [31, 33, 36, 73, 117, 125],
vigorously critiqued Basic Emotion Theory, arguing that while core but is relevant as an example of how contingent definitions of one
affect is an important component of emotional experiences, aspect of emotional response, in this case, bodily changes, can
emotion itself emerges out of human evaluative and experiential become a dominant paradigm for explicating emotions through its
assessments of affective states in particular social contexts [95]. utility to particular sets of experts (in this case, physiologists,
cognitive scientists, and their computationally inclined
Scarantino and de Sousa note that philosophical and psychological descendants).
theories understanding emotion as a form of evaluative assessment
and as felt experience are increasingly intertwined in “hybrid” Depending on the conceptual model of emotion at hand, the
approaches, with “the former now identifying emotions as accuracy and suitability of a particular type of data taken as proxy

784
 FAccT’21, March 3–10, 2021, Virtual Event, Canada L. Stark and J. Hoey

for interior emotional states will—as with any type of social data— have regular, universal, and traceable emotional expressions and
vary widely [84]. According to some of the theoretical traditions reactions, legible across a wide range of proxy data; they make up
described above, some forms of proxy data, such as heart rate or the lion’s share of “Emotional AI” technologies currently available.
facial expression, can say little about a person’s emotional state—
while for others, these forms of data are key indicators of such. The A large segment of the discipline of “affective computing,”
current paradigm around “affective computing” was sparked involving tracking and analyzing a variety of bio-signals like heart
largely by the work of MIT’s Rosalind Picard, whose [91] book rate, is grounded in Basic Emotion Theory [26, 91, 92, 94]. Models
Affective Computing argued for emotion as a topic worthy of powering facial analysis technologies are designed around
examination by computer scientists. Picard’s approach focused on, Ekman’s thesis regarding the legibility of basic emotions through
as noted, physiological signals such as heart rate and blood flow as facial expression [40], and analyze large databases of human faces
proxies for emotions and sought to translate these signals into data [21], using various AI techniques to estimate emotional expression
a computer could find legible and tractable for analysis—treating (along with other characteristics like age and gender) [105]. These
human emotional expression as another form of digitizable systems are increasingly commercially available in areas such as
information. Scholars in social computing and critical HCI human resource management, advertising [76] and education [12].
contested this approach almost immediately, noting such an Lisa Feldman Barrett has been a strong critic of the application of
“informational reading… systematically ignores a second set of Basic Emotion Theory, both as a scientific consensus [68], and
concerns which focus on emotion as it is interactionally and particularly in its application in digital systems [11]. In a major
culturally constituted” [16]. This interactional approach to the recent review, Feldman Barrett and co-authors critiqued the
digital mediation of emotional expression and interaction, laid out proliferation of emotion detection in facial recognition
in a series of papers by critical HCI scholars including Kirsten technologies (FRTs) by calling the underlying generalizability and
Boehner, Phoebe Sengers, Katherine Isbister and Kia Höök among robustness of Basic Emotion Theory’s assumptions into question
others [15-17, 52, 54, 67], emphasizes the centrality of supporting [11]. The authors declared that “When facial movements do express
human emotive interaction through a diverse array of digital emotional states, they are considerably more variable and
technologies, instead of focusing narrowly on the sensing, tracking, dependent on context than the common view allows” [11]. Despite
quantification and analysis of those interactions via computational this caution, however, emotion recognition in FRT continues to
data. Central to the interactional approach is the variability and generate strong commercial interest.
dynamism of human social and emotional relationships: past
performance being no guarantee of future results in life, it should Proxy data for emotional expression from other sources, such as
not, as a design criterion, be emphasized when dealing with the recorded audio of the human voice, or recordings of the electrical
digitally traced “emanations” [62] of our social and emotional lives conductivity of the skin [94], are also often analyzed under
(for a further summary, see [30]). assumptions grounded in the Motivational paradigm. The
proliferation of such digital data, coupled with the popularity of
BET as a computationally tractable theory among Silicon Valley
4 A Taxonomy of Theories and Proxy Data startups, has begun to underpin various assumptions about a wide
We taxonomize AI systems and products for tracking, interpreting, range of digitally tracked behaviors and their purported connections
and modeling human emotional expression (Table 1) along two to human motivation. In the last five years, the broader universe of
axes: a) the conceptual models of emotion on which these systems such analysis for all sorts of behavioral and social data, including
are grounded, either explicitly or implicitly; and b) the types of data around emotion, are increasingly classed under the term “digital
these systems collect and use as proxies to assess human emotional phenotyping,” or “measuring behavior from smart phone sensors,
states. For simplicity and given their increasing interconnection, we keyboard interaction, and various features of voice and speech”
combine the Feeling/Evaluative models in the table below. [53]. The term itself was coined as a term in a 2015 paper by Sachin
Jain and fellow physicians at the Harvard Medical School, based
4.1 Motivational Theories of Emotion and AI on a concept drawn from the 1982 book The Extended Phenotype
Motivational theories of emotion, in particular Basic Emotion by evolutionary biologist Richard Dawkins. Dawkins had argued
Theory, have proven particularly amenable to adaptation into AI for the extension of the notion of phenotype from the set of
systems. These systems are grounded in the notion that humans observable characteristics of an individual “to include all effects
that a gene has on its environment inside or outside of the body of
Table 1. Types of Data Physiological Auditory Haptic Behavioral Semantic Social
Collected
Model of Emotion
Motivational Facial Vocal- Galvanic Digital Sentiment
recognition [47] diagnostic response [86] phenotyping [53] analysis [2]
tech [99]
Experiential/ Brain/machine E-A ML [119] CBDTE BayesACT
Evaluative Interfaces [48] [98] [108]

785
 FAccT’21, March 3–10, 2021, Virtual Event, Canada L. Stark and J. Hoey

the individual organism” [57]. Jain and coauthors reinterpreted BayesACT further provides a theory of action motivation as it
Dawkins’ use of the term “phenotype” loosely, to refer to any couples connotative meanings (cognitive appraisals of sentiments)
with decision theoretic reasoning over denotative states. Sentiments
manifestation or emanation traceable by a digital sensor. As such, are used to guide action towards socially normative behaviors.
large-scale computational analyses now implicitly equate patterns These techniques are being tested in areas such as cognitive
of behavior with intrinsic interior states, including motivation; assistive technologies for persons with dementia that are
under the Motivational paradigm, these analytics can in theory functionally and emotionally aligned with their target users [64],
correlate a wide array of data produced by humans with inferred and facilitator agents in social networks aimed at promoting
emotional states [97]. efficient and inclusive group processes.

It is also possible for the developers of AI systems to treat proxy Other scholars have proposed simulating the appraisal or evaluative
data around emotion as predominantly informing outside observers aspects of emotion as elements of algorithmic learning models
about social displays of emotion (as per Fridlund’s Behavioral themselves. For instance, the Computational Belief and Desire
Ecology view); observers need not make any assumptions about Theory of Emotions (CBDTE) developed by Reisenzein [98]
internal affective states to develop an externally consistent view of argues that emotions are caused by a combination of cognitive
emotional interactions. Sentiment analysis of textual or graphic evaluations (beliefs) and conative motivations (desires), and that
representations of emotion, such as dictionaries of emotion terms such a theory can be modeled formally within learning algorithms
or emoji characters [127], present a case in point. Contemporary used to analyze natural language. In a similar vein, Emotion-
deep learning AI systems often analyze structured natural language Augmented Machine Learning (E-A ML) focuses on developing
data, and in the best case can develop sophisticated models computer models that incorporate simulations of emotion concepts
allowing them to uncover patterns of emotive data in language. into the machine learning process [119]. Such techniques seek to
These systems have no capacity to understand the conceptual enhance the performance of reinforcement learning systems by
“motivation” behind a text; however, AI systems may not need to constraining their evaluative behavior using such simulated
“understand” motivation if they have a sufficiently clear map of emotion concepts as anxiety. In theory, such models would perform
interactions. Facebook’s introduction of “Reactions” icons in 2016, more like humans, using emotions like anxiety as a heuristic filter
enabling users to “react” to all posts with one of six basic emotive to focus on the best available course of action. Finally, recent work
symbols (themselves based on Ekman’s BET), was aimed at on Brain-Machine Interfaces [69] has suggested the possibility of
developing such a network graph of user content irrespective of the translating the brain patterns of emotional experience into digital
motivation users had for reacting in the first place [114]. data, either through direct implants or mechanisms such as audio
waves [48].
4.2 Experiential/Evaluative Theories of Emotion
and AI
Conceptualizations of emotion grounded in the
5 Emotion in Current AI Ethics Debates
experiential/evaluative tradition are less common in current AI The lack of consensus around both conceptual models and
systems, but those that do exist provide notable and instructive empirical proxies for emotion has important normative
contrasts to those of the dominant Motivational paradigm. One implications for the AI systems reliant on them. The implied ethical
method grounded in the evaluative tradition for modeling human and social responsibilities for human persons vary based on the
social and emotional interactions computationally is Bayesian causal models of emotion at issue. The social and ethical weights
Affect Control Theory or BayesACT [108]. Affect Control Theory, given to subjective human experience, motivation, or belief are
initially developed by social psychologist David R. Heise [49], is a shaped by how an observer understands the potential causes for
quantitative sociological method akin to structural symbolic emotional expression, the evidence of such expression, and how
interactionism [70]. In an ACT analysis, the interactions between emotion as a normative force is accounted for when considering the
actors, behaviors, and settings are mapped by an observer: in doing impacts of values such as fairness, transparency, or accountability
so according to ACT, “the observer realizes if the situation is in AI systems.
aligned with cultural norms or represents a deflection from cultural
norms based on the affective sentiments. In the case of deflection, Andrew McStay [7, 75, 76, 78] has been among the most active
the observer tries to restore a coherent definition of the situation” voices at the intersection of scholarship on emotion and AI ethics.
[110]. ACT implicitly defines a morality concept which is McStay’s work points to the proliferation of systems for tracking,
embedded in these cultural norms [72]. Developed out of inferring, and measuring signals of human emotive expression,
collaborations between sociologists and computer scientists [108], what McStay terms “Emotional Artificial Intelligence (EAI),” as
BayesACT combines ACT with Bayesian probabilistic decision necessitating a broader focus on both the technical affordances and
theory to make evaluative predictions about what emotions are social impacts of these technologies. With Pamila Pavliscak,
appropriate for a variety of typical situations. These predictions McStay has developed a set of guidelines for the ethical use of EAI
enable virtual agents to better calibrate their responses to users, technologies, designed explicitly to enable companies to “innovate
letting the system detect emotional cues and respond appropriately. ethically as well as legally” in the area [77]. In line with the wider

786
 FAccT’21, March 3–10, 2021, Virtual Event, Canada L. Stark and J. Hoey

proliferation of AI ethic codes, statements of principle and similar The EAI guidelines suggest further that, “collecting data about
envisioning documents [38, 43, 59], the EAI guidelines present a emotion in public spaces may be unwanted or invasive.” This
high-level thematic checklist for anyone engaged with “data about guidance points to the particular sensitivity many people ascribe to
human emotion.” Noting that, “Emotional artificial intelligence has emotion and emotional expression [3], but also elides the fact that
significant personal, interpersonal, and societal implications,” the certain emotional expression and the boundaries of public and
authors’ varied instructions for practitioners are divided by putative private space vary and intersect in granular and sometimes
relevance to the individual person, to relationships, and to society. surprising ways [118]. For instance, the oft-referenced 2014
McStay and Pavliscak’s guidelines also include a number of Facebook emotional contagion study [65], which came under fire
salutary suggestions for “taking action” as a practitioner against in the media for manipulating the semantic emotive content of user
unethical design decisions, a rarity in the ethics guidelines genre news feeds, was criticized precisely because Facebook users had a
(including suggesting either saying no joining to such efforts or different felt understanding of the privacy of their interactions than
leaving an institution based on ethical scruples). did the researchers and the platform [113]. As Siva Vaidhyanathan
[132], Frank Pasquale [87], Tero Karppi [61], and others have
A number of the document’s prescriptions, including all those noted, Facebook’s business model entails making, and shaping,
under the listing of the “Personal” implications for EAI, are assumptions about human emotion precisely so it can affect the
important but broadly applicable to all identifying data. Under the choices of both individuals and groups on the platform to engage
heading of implications for “Relationships,” however, the and interact. This molding of what Zizi Papacharrisi terms
guidelines list three checklist items specifically focused on emotion “affective publics” [85] does not always have predictable results
(as well as one guideline aiming to safeguard trust in non-human and is always subject to counter-pressures from individuals and
actors, and one dealing with procedures for identifying mental groups themselves [46], but even under an interactional model of
health challenges in users). These three items are: “Understands emotion is always a factor in design and deployment. Moreover,
that physical display of emotion is only one facet of emotion”; the varying cultural contexts of emotional norms, variations, and
“Recognizes that past expression doesn’t always predict future interactions makes broad assumptions about emotional
feeling”; and “Considers stereotypes and assumptions about universalism not just unwise, but actively deleterious.
emotion that materially affect a person or group.” Under “Societal”
implications, the guidelines provide two items focused on emotion: Finally, the guidelines ask practitioners to recognize, “the lack of
“Recognizes the lack of globally objective agreement on emotion,” globally objective agreement on emotion.” This item points to the
and “Recognizes that collecting data about emotion in public central problem at hand, both for AI practitioners building systems
spaces may be unwanted or invasive.” These five items (one-third engaging with data about human emotion and perhaps for McStay
of the total number in the guidelines) deal explicitly with emotion, and Pavliscak’s guidelines themselves: the wide divergence of
as opposed to the wider universe of concerns around the use of opinion in both philosophy and in the sciences regarding what
digital data for tracking, profiling, and behavioral nudging in the emotions are, means recognizing that diversity of opinion in the
digital economy [4, 24, 114]. These details of McStay and abstract is insufficient without considering how those differences
Pavliscak’s schema are worth examining closely, in how the might implicate AI design and deployment decisions, with their
guidelines implicitly define emotion, for what they foreground as attendant ethical valences and social effects.
ethical and social challenges unique to EAI as opposed to AI more
generally, and what they omit—all elements indexical to the
broader gaps in conversations around emotion and ethics in AI. 6 Analysis and Discussion
Digital data on human emotional expression not only imperfectly
McStay and Pavliscak’s call for practitioners to reflect on the fact reflect the complexity of human emotional response; the conceptual
that “physical display of emotion is only one facet of emotion” is models used to make sense of emotions themselves also
sound, and poses practical challenges for practitioners steeped in imperfectly reflect that complexity, as per the Problem of Parts and
the particular, and narrow, definitional discourses around Basic the Problem of Plenty described by Prinz [96]. Both models and
Emotion Theory common in computer science. In a similar vein, data are schematically representative of the multiple elements of
the EAI guideline item prompting practitioners to recognize that, human emotional experience, denoted imperfectly and quantified
“past expression doesn’t always predict future feeling” refers partially. As such, there will always be a gap between the model
implicitly to larger debates in computer science regarding how used and the experience lived, a problem increasingly well-
digital media technologies should understand and approach the articulated in critical studies of AI systems more broadly. As Selbst
longitudinal stability human of emotional expression as an element et al. observe, “abstracting away the social context in which these
in technical design. The guidelines also ask if practitioners have systems will be deployed [means] researchers miss the broader
considered “stereotypes and assumptions about emotion that context, including information necessary to create fairer outcomes,
materially affect a person or group.” Across these three questions, or even to understand fairness as a concept” [111]. The particular
the guidelines assume a stable and common definition or complexity and personal sensitivity of human emotions makes this
understanding of what emotion is in the first place. challenge especially salient, and potentially troubling, for emotion
AI systems. While some of these challenges map to broader

787
 FAccT’21, March 3–10, 2021, Virtual Event, Canada L. Stark and J. Hoey

concerns around biases and aporias in other forms of personal data, It thus matters both in terms of what investigators are measuring as
in particular healthcare data used in AI/ML analysis [41], the a proxy for emotional or emotive expression, but also what
collection of data around emotional expression and emotion investigators believe the responses measured mean about the
modeling in AI also present a number of unique additional interiority, judgments, and potential future actions of human
normative challenges. These challenges include often implicit beings. For the creators of emotion AI systems, the need to
associations between human emotions and normative categories; understand the subtleties around judging the normative significance
the particular implied norms of certain emotion models such as of emotion adds a third layer of complexity on top of the two layers
Basic Emotion Theory; human emotions as a locus for online already described (what conceptual model is being used to explain
experimentation; the dangers of reifying particular emotion what emotions are, and what data is being collected to determine
metrics; and the lack of scientific consensus underpinning the how emotions are expressed). In practice, these three categories are
models used in AI systems intended to measure and model emotion. interrelated: normative judgments can emerge from conceptual
assumptions, themselves grounded in a particular interpretation of
6.1 The Opaque Normative Weight of Emotion empirical data or the choice of what data is serving as proxy for
The most basic normative concern around the collection of data on emotive expression.
human emotional expression and the computational analysis of that
data stems from the ways in which emotions are associated, Emotions are not only irreducible to any one form of proxy data but
implicitly and explicitly, with human agency in theories of ethical are also subjective phenomena in part illegible to outside observers.
decision-making. In other words, what normative weight do the As such, any AI system engaging with models or data about human
various conceptual models we have already described place on emotions should be flagged immediately by oversight authorities as
emotions in light of their assumed relationship to human action? If requiring heightened scrutiny around its social impacts and
emotion and other intuitive processes are understood to play a normative effects, irrespective of the context of use. Abstracting
central role in ethical and moral judgments, then incorporating any away the social context of an emotional expression presents a
proxy data for human emotion into an AI system takes on fraught fundamental barrier to the comprehensive understanding of
normative importance. The choice of conceptual priors and the type emotion; many current AI-based efforts to do so contribute to
of emotion data collected—indeed, the decision to design and scientific overreach, unethical and anti-democratic
deploy an AI system engaged with human emotion in any way at experimentation and manipulation, and the internalization and
all—will invariably import particular norms and values into a reification by individuals of the same problematic metrics. The
technical system, ones that will affect the impacts of these systems notion of predicting the individual emotional states of particular
in ways often unanticipated by designers and others responsible for people using these systems is therefore always suspect, as even
their deployment. computerized iterations of evaluative models like Affect Control
Theory will extrapolate typical emotion reactions which will not
Scarantino and de Sousa observe that emotions are often hold in all cases. Designers and developers should think twice
understood as impediments to rationality, and by extension to before embarking on emotion AI projects: a necessary though not
considered rational judgment: “Emotions,” the authors write, “have sufficient condition for such projects is clear alignment between
long been thought to score poorly in terms of both cognitive and conceptual models, data, norms, and aims.
strategic rationality,” the former “consisting of their ability to
represent the world as it is,” and the latter “consisting of their ability 6.2 The Troubling Norms of Basic Emotion
to lead to actions that promote the agent’s interests” [104] (s.10.1). Theory
However, more recent scholarship in both psychology and It is also worth considering how the broad conceptual split between
neuroscience has highlighted the centrality and necessity of motivational theories of emotion and experiential/evaluative
emotion as a component in “rational” cognition. Emotions theories might shape how emotions are understood normatively,
“determine salience among potential objects of attention,” and and the implications for AI/ML systems that incorporate one or the
while this phenomenon has the potential to misdirect attention, it other of these conceptual models for emotion. The most obvious
can also help sustain long-term planning and goal setting [9]. Self- division comes around the motivational tradition’s focus on
reflexivity around emotional responses is central to their balanced emotions as causal phenomena. At least in Basic Emotion Theory,
contribution to “rational” outcomes, but such reflexivity also this view of emotions as motivational leads to an understanding of
requires weighing “rational” or cognitive considerations against the exteriorized emotional expression as “true” manifestations of inner
signal function provided by the emotion states themselves. In recent emotional states, and as uncontrollable, and thus unfalsifiable
work in affect control theory, the tradeoffs between rational and symptoms of internal subjective impulses.
emotional cognition are considered primarily in light of the relative
uncertainties between the two forms of mental process in Given the popularity of BET as the conceptual underpinning for the
emotionally charged situations; the ways in which the tradeoff is design of AI/ML systems that track and categorize emotional
made may be culturally or individually dependent [51]. expression, this fundamental division in the understanding of
emotion’s relationship to agency has an outsized effect on the
ethical and social impacts of these AI systems as they are deployed

788
 FAccT’21, March 3–10, 2021, Virtual Event, Canada L. Stark and J. Hoey

in practice. In the case of BET, this influence is unfortunate. A Insurance Portability and Accountability Act (HIPAA) in the
number of commentators [14] [27, 89, 116], one of us included United States, but rarely treated as such. Though outside the scope
[115], have argued that AI/ML systems used to analyze human of this paper, current conversations on how to adapt research ethics
faces, bodies, and gaits are engaged in a digitally-mediated form of protocols to digital contexts in both the academy and industry are
physiognomy, the discredited nineteenth- and early twentieth- only the start of a larger set of normative concerns around these
century practice of using people’s outer appearance to infer inner data practices [37, 131].
character [29, 106]. Motivational theories of emotion align,
however loosely, with both the legacy of physiognomy and the Experimentation on the part of social media companies often
logics of contemporary facial recognition technologies (FRTs) and entails modifying elements of their interfaces to affect users: these
other similar systems [40]. modifications can have many effects, including an increase in
political polarization [19], subversion of existing consent regimes
Moreover, as critical scholars of both race and gender have argued, [131], and distrust of the modifications even if they are innocuous
emotional expression is a key vector through which racist [79]. The popular furor around Facebook’s 2014 “emotional
hierarchies and misogynist tropes are produced (or “discovered”) contagion” study [65], and subsequent 2018 Cambridge Analytica
routinized, and enforced [82, 126], often through the mobilization scandal [100, 130], are exemplary of how behavioral
of motivational theories that purport to reveal “inferior” interiority experimentation around emotion online can have long-standing
through externalized emotive signals [101]. For instance, Kyla cultural consequences [114], they are also indexical to broader
Schuller [109] articulates how discourses around the “biopolitics of questions around the ethics of experimentation on the part of digital
feeling” developed in the nineteenth century equated emotional organizations on the emotional, psychological, and behavioral data
impressibility with civilizational refinement—and by extension, of their users.
defined “primitive” subjects as ones whose emotion were both However, it is also important to note that such manipulations do not
legible and predictable. Likewise, Otniel Dror’s account of imply that these companies can build an accurate model of a person
nineteenth century “emotion as number” points to how such based on their affective responses, nor that they necessarily
constructed hierarchies were quantified and solidified through understand the causal relationships between the responder and the
technical language [32, 33] . modified elements of their interface [56]. While one may deduce
that an affective response arises from a change made on a social
While many of today’s proponents of Basic Emotion Theory may media platform, it is incorrect to claim this means one can infer or
not be aware of these historical genealogies, they are nonetheless induce a person’s “phenotype” from their affective responses.
impossible to discount, to say nothing of more recent critiques of While these responses may have resulted, in part, from their
the field [9, 68]. Basic Emotion Theory does not lend sufficient reactions to the change made, it may also depend on many other
scientific evidence for even nuanced arguments grounded in the factors that are context or person dependent. Since a response is a
Motivational tradition, much less the often sensationalist claims causal effect of the change and the person, experimenters can
made by many FRT providers that such systems are able to easily deduce that the response arose because of the change and can even
determine what individuals are “really feeling” [11]. BET’s measure the results of an A/B test to see which change is more apt
influence as the chief paradigm for the digital mediation and at delivering a response —but cannot infer much about the person
interpretation of human emotion is thus a major normative given a particular response to a change because of the unknown
challenge for designers and regulators, and one which deserves other factors impacting the affective response.
heightened scrutiny from ethical and regulatory perspectives.
6.4 Reification and Interiorization of
6.3 Emotion as a Locus of Experiment Models/Metrics
A focus on emotional expression as a component of AI/ML analysis The conceptual models and proxy data for emotion in an AI system
demonstrates the broader tendency of AI/ML researchers in are not solely a concern for the system’s designers. These models
corporate settings to perform de facto human subject research and data possess a descriptive power that is also a prescriptive one
without attendant awareness of, or attention to the ethical [71, 114]. Individuals often adjust their own attitudes to conform to
complexities of such experiments [20]. In the 2020 documentary an “objective” measure, in this case of emotional expression, that
The Social Dilemma, for instance, much of the film’s central is in fact partial, constructed and potentially detached from lived
message focuses on the ability of social media platforms to control experience. As such, the digital remediation of emotional
users, and to sell this control to advertisers. Personal data is at the expression has the potential to shift subjective normative
heart of the business models of many digital technology companies, frameworks for decision-making towards the emotional models,
and the collection of information about every aspect of a user's (or and implicit values, of technology firms, not of individuals as users
even a non-user's) interactions with a site or app is now well and citizens.
understood as a major privacy and civil society problem [83, 88,
128]. Data on human emotional expression is what Nicholas Terry In a recent qualitative study [3], Nazanin Andalibi and Justin Buss
terms "medically-inflected" data [120], possessing a sensitivity on provide evidence for the outsized impacts the digital remediation
par with data protected under regulations such as the Health of emotional expression can have on individuals. Andalibi and

789
 FAccT’21, March 3–10, 2021, Virtual Event, Canada L. Stark and J. Hoey

Buss’s respondents were highly aware of, and concerned with the how the models of emotion most prevalent in representations of AI
potential power of these systems: “The majority of participants,” systems and digital media more broadly, such as BET, are affecting
the authors observe, “were uncomfortable with emotion individual and collective subjective assessments of emotional
recognition, and this discomfort was often related to concerns over agency, and how these changing subjective mental paradigms are
privacy, consent, agency, and potential harm” [3]. Participants also shaping actions and behavior in diverse cultures worldwide.
pointed to a lack of accountability on the part of the designers and
developers emotion recognition and analysis systems to engage 6.5 Lack of Scientific and Normative Consensus
with these embodied concerns. As Andalibi and Buss point out, as Disqualifying
there is not only a lack of recognition on the part of technologists Finally, attempts to quantify and standardize measures of emotion
regarding the multiple potential sources of data about emotion, and through its expression illustrates the wider conceptual difficulty in
conceptual frameworks under which it is collected; there is also a constituting ethical or normative guidelines around AI/ML systems
lack of recognition of the diversity of human attitudes towards their shared broadly across communities and societies, due in part to
own emotions, and to emotions as social phenomena impacted by what philosopher Thomas Nagel terms “the fragmentation of
digital remediation. Vernacular media products such as the 2015 value” [80]. What components of emotion are most salient in a
Disney/Pixar film Inside Out, which depicts the brain and emotions particular context? The lack of consensus raises the question of
through a combination of BET theory and metaphors of mediation whether it is ever ethically appropriate to develop and deploy such
[45, 121], are a further mechanism reinforcing the discursive power systems for public consumption: if, for instance, the science of
of particular emotion models in everyday discourse. Basic Emotion Theory cannot support the claims its AI/ML
proponents make, their incorporation into AI systems is potentially
Individuals whose emotion data are being tracked and aggregated fatally flawed regardless of other ethical safeguards. This problem
are thus often in a bind regarding how to respond to the analytic is analogous to broader debates around appropriate data collection
outputs of AI systems. If relationships between different variables versus appropriate data use in the digital privacy arena. While
correlate in the aggregate, there is a danger that modelers will safeguards on the appropriate use of emotion recognition in AI
assume the same relationship will also correlate at an individual systems are necessary, they are not sufficient, and a wider
level, an error known as the ‘ecological fallacy’ [90]. Tools like conversation around the deployment of emotion recognition
computational sentiment analysis are increasingly deployed to systems in AI is vital given the potentially toxic social effects such
chart the “ambient sentiment” of groups such as Twitter users [5]. technologies can produce [115].
Yet a focus on aggregate modeling effaces the ways that individuals
must modulate their emotional responses over time to conform to
norms produced by the aggregating institutions, such as social 7 Conclusion
media platforms: aggregated categories are represented back to The analytics of emotional expression highlight human emotion’s
individual users as norms against which they should perform.
centrality not just to ethical AI/ML systems, but also to these
[114]. Individuals must preemptively position themselves as fluent system’s broader mediating effects on social and political
in the emotional expressions, behaviors and gestures aligned with community and cohesion through their everyday use. Human
a platform’s models, able to both conform to these classificatory
emotion is a complex topic, and analysis of its effects and impacts
schemes while caught in the everyday pressure to perform in AI/ML benefit from interdisciplinary collaboration. There is thus
emotional expressions in non-virtual social situations. a critical and urgent need for scholars, policymakers, and
technologists to understand the complexity of human emotions and
The questions of aggregation and reification also cuts across global the digital economy being built on them when designing, critiquing,
cultures, where the lack of “global” agreement on emotion in more and regulating AI systems. As research and commercial interest in
ways than one also threatens to produce platform-driven regimes of “artificial emotional intelligence” (AEI) intensifies, we argue the
emotive conformity. Anthropologists have long observed the particularities of how these systems are designed—including the
cultural specificity of emotion norms and discourses [1], and the models of emotion designers use to ground their models, and the
dangers of attempting to presume stability across heterogenous types of proxy data for emotion they collect—matter greatly for the
views of emotional expression [50]. While recent scholarly work ethical appropriateness of such systems, and even whether they
has begun to document cultural variation in the use patterns for should be developed and deployed at all.
digital formats for emotive expression such as emoji [25], there is
little comparative research on how the emotional models built into
AI systems vary in their performance and interpretation based on
ACKNOWLEDGMENTS
particular geographical locations and cultural norms – let alone
whether the use of these emotional models has begun to reflexively Our thanks to Ben Green, Deborah Raji, Varoon Mathur, Casey
change those norms and homogenize emotional expression around Gollan, Alejandro Calcaño Bertorelli, Sarah Myers West, Erin
the world. A number of scholars have examined the ways McElroy, Elizabeth Kaziunas, Osonde Osoba, Fernando Diaz,
individuals perceive, understand, and interpret the workings of Benjamin Fish, Asia Biega, Alexandra Olteanu, Nazanin Andalibi,
algorithmic systems [22, 23]. More research is needed to examine Emily McBain-Ashfield, Jason Millar, Florian Martin-Bariteau,

790
 FAccT’21, March 3–10, 2021, Virtual Event, Canada L. Stark and J. Hoey

and the other organizers of the WeRobot 2020 virtual conference [28] Davies, W. 2017. How are we now? Real-time mood-monitoring as valuation.
Journal of Cultural Economy 10, 1 (2017), 34–48.
for contributing to earlier versions of this piece; and especially to [29] de Giustino, D. 2016. Conquest of Mind: Phrenology and Victorian Social
Bronwen Masemann for her invaluable editorial assistance. Thought. Routledge.
[30] Desmet, P.M.A. and Roeser, S. 2015. Emotions in Design for Values.
Handbook of Ethics, Values, and Technological Design. J. van den Hoven, P.E.
REFERENCES Vermaas, and I. van de Poel, eds. Springer. 203–219.
[1] Abu-Lughod, L. and Lutz, C.A. 1990. Introduction: emotion, discourse, and [31] Dror, O.E. 2009. Afterword: A Reflection on Feelings and the History of
the politics of everyday life. Language and the Politics of Emotion. C.A. Lutz Science. Isis. 100, 4 (2009), 848–851.
and L. Abu-Lughod, eds. Cambridge University Press. 1–23. [32] Dror, O.E. 2001. Counting the Affects: Discoursing in Numbers. Social
[2] Alashri, S., Kandala, S.S., Bajaj, V., Ravi, R., Smith, K.L. and Desouza, K.C. Research. 68, 2 (Jul. 2001), 357–378.
2016. An analysis of sentiments on Facebook during the 2016 U.S. presidential [33] Dror, O.E. 1999. The Affect of Experiment: The Turn to Emotions in Anglo-
election. 2016 IEEE/ACM International Conference on Advances in Social American Physiology, 1900-1940. Isis. 90, 2 (Jun. 1999), 205–237.
Networks Analysis and Mining (ASONAM) (Jul. 2016), 795–802. [34] Ekman, P. and Friesen, W.V. 1971. Constants across cultures in the face and
[3] Andalibi, N. and Buss, J. 2020. The Human in Emotion Recognition on Social emotion. Journal of Personality and Social Psychology. 17, 2 (Feb. 1971),
Media: Attitudes, Outcomes, Risks. CHI '20: ACM Conference on Human 124–129.
Factors in Computing Systems (New York, NY, USA, Apr. 2020), 1–16. [35] Eubanks, V. 2018. Automating Inequality: How High-Tech Tools Profile,
[4] Andrejevic, M. 2007. Surveillance in the Digital Enclosure. The Police, and Punish the Poor. St. Martin's Press.
Communication Review. 10, 4 (Oct. 2007), 295–317. [36] Fernandez, L. and Matt, S.J. 2019. Bored, Lonely, Angry, Stupid. Harvard
[5] Andrejevic, M. 2011. The Work That Affective Economics Does. Cultural University Press.
Studies. 25, 4-5 (Sep. 2011), 604–620. [37] Fiesler, C. and Proferes, N. 2018. “Participant” Perceptions of Twitter
[6] Arnold, M. 1960. Emotion and Personality. Columbia University Press. Research Ethics. Social Media + Society. 4, 1 (Mar. 2018),
[7] Bakir, V. and McStay, A. 2017. Fake News and The Economy of Emotions. 205630511876336–14.
Digital Journalism. 6, 2 (Jun. 2017), 154–175. [38] Fjeld, J., Achten, N., Hilligoss, H., Nagy, A.C. and Srikumar, M. 2020.
[8] Barocas, S. and Selbst, A.D. 2016. Big Data's Disparate Impact. California Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-
Law Review. 104, 3 (2016), 671–732. based Approaches to Principles for AI. Technical Report #Berkman Klein
[9] Barrett, L.F. 2017. How Emotions Are Made: The Secret Life of the Brain. Center Research Publication No. 2020-1. Berkman Klein Center for Internet
Houghton Mifflin Harcourt. and Society.
[10] Barrett, L.F. 2006. Solving the Emotion Paradox: Categorization and the [39] Fridlund, A.J. 1994. Human Facial Expression: An Evolutionary View.
Experience of Emotion. Personality and Social Psychology Review. 10, 1 (Feb. Academic Press.
2006), 20–46. [40] Gates, K. 2011. Our Biometric Future: Facial Recognition Technology and the
[11] Barrett, L.F., Adolphs, R., Marsella, S., Martinez, A.M. and Pollak, S.D. 2019. Culture of Surveillance. New York University Press.
Emotional Expressions Reconsidered: Challenges to Inferring Emotion From [41] Ghassemi, M., Celi, L.A. and Stone, D.J. 2015. State of the art review: the data
Human Facial Movements. Psychological Science in the Public Interest. 20, 1 revolution in critical care. Critical care (London, England). 19, 1 (Mar. 2015),
(Jul. 2019), 1–68. 118–9.
[12] Ben Williamson 2019. Psychodata: disassembling the psychological, [42] Gould, D. 2010. On Affect and Protest. Political Emotions. J. Staiger, A.
economic, and statistical infrastructure of “social- emotional learning.” Journal Cvetkovich, and A. Reynolds, eds. Routledge. 18–44.
of Education Policy (Oct. 2019), 1–26. [43] Greene, D., Hoffmann, A.L. and Stark, L. 2019. Better, Nicer, Clearer, Fairer.
[13] Benjamin, R. 2019. Race after Technology: Abolitionist Tools for the New Jim HICSS 2019 (Jan. 2019), 2122–2131.
Code. John Wiley & Sons. [44] Greene, G. 2020. The Ethics of AI and Emotional Intelligence. Partnership on
[14] Birhane, A. and Guest, O. 2020. Towards decolonizing computational sciences. AI.
arXiv. [45] Hard Feelings — Inside Out, Silicon Valley, and Why Technologizing
[15] Boehner, K., Boehner, K., DePaula, R., Dourish, P. and Sengers, P. 2005. Emotion and Memory Is a Dangerous Idea: 2015.
Affect: From Information to Interaction. CC '05: Proceedings of the 4th [Link]
decennial conference on critical computing (New York, NY, 2005), 59–68. why-technologizing-emotion-and-memory-is-a-dangerous-idea. Accessed:
[16] Boehner, K., DePaula, R., Dourish, P. and Sengers, P. 2007. How Emotion is 2015-09-30.
Made and Measured. International Journal of Human-Computer Studies. 65, [46] Hardt, M. 1999. Affective Labor. boundary 2. 26, 2 (1999), 89–100.
(2007), 275–291. [47] Hassan, M.M., Alam, M.G.R., Uddin, M.Z., Huda, S., Almogren, A. and
[17] Boehner, K., Sengers, P. and Warner, S. 2008. Interfaces with the Ineffable: Fortino, G. 2019. Human emotion recognition using deep belief network
Meeting Aesthetic Experience on its Own Terms. ACM Transactions on architecture. Information Fusion. 51, (Nov. 2019), 10–18.
Computer-Human Interaction 15,3 (December 2008). [48] Hearables Will Monitor Your Brain and Body to Augment Your Life: 2019.
[Link] [Link]
[18] boyd, D. and Crawford, K. 2012. Critical Questions for Big Data. Information, monitor-your-brain-and-body-to-augment-your-life. Accessed: 2019-05-06.
Communication & Society. 15, 5 (Jun. 2012), 662–679. [49] Heise, D.R. 2007. Expressive Order. Springer.
[19] Brady, W.J., Wills, J.A., Jost, J.T., Tucker, J.A. and Van Bavel, J.J. 2017. [50] Hochschild, A.R. 2003. The Managed Heart: Commercialization of Human
Emotion shapes the diffusion of moralized content in social networks. Feeling. University of California Press.
Proceedings of the National Academy of Sciences. 114, 28 (Jul. 2017), 7313– [51] Hoey, J. Citizens, Madmen and Children: Equality, Uncertainty, Freedom and
7318. the Definition of State. SocArXiv 2021. [Link]
[20] Brandt, M. and Stark, L. 2018. Exploring Digital Interventions in Mental [52] Höök, K., Isbister, K., Westerman, S., Gardner, P., Sutherland, E., Vasalou, A.,
Health: A Roadmap. Interventions. A. Shaw and D.T. Scott, eds. Peter Lang. Sundström, P., Kaye, J. and Laaksolahti, J. 2010. Evaluation of Affective
167–182. Interactive Applications. Emotion-Oriented Systems. Springer Berlin
[21] Brodny, G., Kolakowska, A., Landowska, A., Szwoch, M., Szwoch, W. and Heidelberg. 687–703.
Wróbel, M.R. 2016. Comparison of selected off-the-shelf solutions for emotion [53] Insel, T.R. 2017. Digital Phenotyping: Technology for a New Science of
recognition based on facial expressions. HSI. (2016), 397–404. Behavior. JAMA. 318, 13 (Oct. 2017), 1215–1216.
[22] Bucher, T. 2016. The algorithmic imaginary: exploring the ordinary affects of [54] Isbister, K., Höök, K., Laaksolahti, J. and Sharp, M. 2007. The sensual
Facebook algorithms. Information, Communication & Society 20, 1 (Feb. evaluation instrument: Developing a trans-cultural self-report measure of
2016), 30–44. affect. International Journal of Human-Computer Studies. 65, 4 (Apr. 2007),
[23] Burrell, J. 2016. How the machine “thinks”: Understanding opacity in machine 315–328.
learning algorithms. Big Data & Society. 3, 1 (Feb. 2016), 205395171562251– [55] Izard, C.E. 1972. Patterns of Emotions: A New Analysis of Anxiety and
12. Depression. Academic Press.
[24] Cheney-Lippold, J. 2017. We Are Data: Algorithms and The Making of Our [56] Jacobs, A.Z. and Wallach, H. 2019. Measurement and Fairness. arXiv.
Digital Selves. New York University Press. [57] Jain, S.H., Powers, B.W., Hawkins, J.B. and Brownstein, J.S. 2015. The digital
[25] Chui, S. 2020. A Comparative Study of the Interpretations of Emojis in phenotype. Nature Publishing Group. 33, 5 (May 2015), 462–463.
Between U.S. and Chinese Users. International Journal of Literature and Arts. [58] James, W. 1884. What is an Emotion? Mind. 9, (1884), 188–205.
8, 3 (2020), 108–11. [59] Jobin, A., Ienca, M. and Vayena, E. 2019. The global landscape of AI ethics
[26] Clynes, M. 1989. Sentics: The Touch of the Emotions. Prism Press Ltd. guidelines. Nature Machine Intelligence. (Sep. 2019), 1–11.
[27] Crawford, K. et al. 2019. AI Now 2019 Report. AI Now Institute. [60] Johnson, D.G. 2018. AI, agency and responsibility: the VW fraud case and
beyond. AI & Society. 0, 0 (Jan. 2018), 0–0.

791
 FAccT’21, March 3–10, 2021, Virtual Event, Canada L. Stark and J. Hoey

[61] Karppi, T. 2018. Disconnect: Facebook's Affective Bonds. University of [96] Prinz, J.J. 2004. Introduction: Piecing Passions Apart. Gut Reactions: A
Minnesota. Perceptual Theory of Emotion. Oxford University Press. 1–11.
[62] Kerr, I. and McGill, J. 2007. Emanations, Snoop Dogs and Reasonable [97] Reece, A.G. and Danforth, C.M. 2017. Instagram photos reveal predictive
Expectations of Privacy. Criminal Law Quarterly. 52, 3 (2007), 392–431. markers of depression. EPJ Data Sci. 6, 1 (Jul. 2017), 1–12.
[63] Kim, D., Frank, M.G. and Kim, S.T. 2014. Emotional display behavior in [98] Reisenzein, R. 2009. Emotional Experience in the Computational Belief–
different forms of Computer Mediated Communication. Computers in Human Desire Theory of Emotion. Emotion Review. 1, 3 (Jun. 2009), 214–222.
Behavior. 30, (Jan. 2014), 222–229. [99] Ren, M., Nie, W., Liu, A. and Su, Y. 2019. Multi-modal Correlated Network
[64] König, A., Francis, L.E., Joshi, J., Robillard, J.M. and Hoey, J. 2017. for emotion recognition in speech. Visual Informatics. 3, 3 (Sep. 2019), 150–
Qualitative study of affective identities in dementia patients for the design of 155.
cognitive assistive technologies. Journal of Rehabilitation and Assistive [100] Revealed: 50 million Facebook profiles harvested for Cambridge Analytica in
Technologies Engineering. 4, (Jan. 2017), 205566831668503–15. major data breach: 2018.
[65] Kramer, A.D.I., Guillory, J.E. and Hancock, J.T. 2014. Experimental evidence [Link]
of massive-scale emotional contagion through social networks. Proceedings of facebook-influence-us-election. Accessed: 2018-03-18.
the National Academy of Sciences. 111, 24 (Jun. 2014), 8788–8790. [101] Rhue, L. 2018. Racial Influence on Automated Perceptions of Emotions. (Dec.
[66] Lazarus, R.S. 1991. Emotion and Adaptation. Oxford University Press. 2018), 1–11. [Link]
[67] Leahu, L., Schwenk, S. and Sengers, P. 2008. Subjective Objectivity: [102] Russell, J.A. 2003. Core affect and the psychological construction of emotion.
Negotiating Emotional Meaning. DIS '08. (February 25-27 2008), 425–434. Psychological Review. 110, 1 (2003), 145–172.
[68] Leys, R. 2017. The Ascent of Affect. University of Chicago Press. [103] Russell, J.A. 2009. Emotion, core affect, and psychological construction.
[69] Littlefield, M. and Johnson, J. 2012. The Neuroscientific Turn: Cognition & Emotion. 23, 7 (Nov. 2009), 1259–1283.
Transdisciplinarity in the Age of the Brain. University of Michigan Press. [104] Scarantino, A. and de Sousa, R. 2018. Emotion. The Stanford Encyclopedia of
[70] Lively, K.J. and Heise, D.R. 2004. Sociological Realms of Emotional Philosophy. E.N. Zalta, ed.
Experience. American Journal of Sociology. 109, 5 (Mar. 2004), 1109–1136. [105] Scheuerman, M.K., Paul, J.M. and Brubaker, J.R. 2019. How Computers See
[71] MacKenzie, D.A. 2016. An Engine, Not a Camera. The MIT Press. Gender. Proceedings of the ACM on Human-Computer Interaction. 3, CSCW
[72] MacKinnon, N.J. 2020. Affect Control Theory Applied to Morality. (Nov. 2019), 1–33.
[73] Malin, B. 2014. Feeling Mediated: A History of Media Technology and [106] Schlag, P. 1997. Law and Phrenology. Harvard Law Review. 110, (1997), 877–
Emotion in America. New York University Press. 921.
[74] Martin, E. 2007. Bipolar Expeditions. Princeton University Press. [107] Schneider, S.M. and Morris, E.K. 1987. A history of the term radical
[75] McStay, A. 2018. Emotional AI: The Rise of Empathic Media. SAGE. behaviorism: From Watson to Skinner. The Behavior Analyst. 10, 1 (1987),
[76] McStay, A. 2016. Empathic media and advertising: Industry, policy, legal and 27–39.
citizen perspectives (the case for intimacy). Big Data & Society. 3, 2 (Sep. [108] Schröder, T., Hoey, J. and Rogers, K.B. 2016. Modeling Dynamic Identities
2016), 205395171666686–11. and Uncertainty in Social Interactions. American Sociological Review. 81, 4
[77] McStay, A. and Pavliscak, P. 2019. Emotional Artificial Intelligence: (Jul. 2016), 828–855.
Guidelines for ethical use. [Link]. [109] Schuller, K. 2018. The Biopolitics of Feeling. Duke University Press.
[78] McStay, A. and Urquhart, L. 2019. “This time with feeling?” Assessing EU [110] Shank, D.B. 2010. An Affect Control Theory of Technology. Current Research
data governance implications of out of home appraisal based emotional AI. in Social Psychology. (2010), 1–13.
First Monday. (2019). [111] Shank, D.B. 2014. Technology and Emotions. Handbook of the Sociology of
[79] Meyer, M.N., Heck, P.R., Holtzman, G.S., Anderson, S.M., Cai, W., Watts, Emotions: Volume II. Springer Netherlands. 511–528.
D.J. and Chabris, C.F. 2019. Objecting to experiments that compare two [112] Solomon, R.C. 2003. What Is an Emotion?: Classic and Contemporary
unobjectionable policies or treatments. Proceedings of the National Academy Readings. Oxford University Press.
of Sciences. 43, (May 2019), 201820701–6. [113] Stark, L. 2019. Affect and Emotion in digitalSTS. digitalSTS: A Field Guide
[80] Nagel, T. 1979. The Fragmentation of Value. Mortal Questions. Cambridge for Science Technology Studies. J. Vertesi and D. Ribes, eds. 117–135.
University Press. 128–141. [114] Stark, L. 2018. Algorithmic Psychometrics and the Scalable Subject. Social
[81] Narayanan, A. and Vallor, S. 2014. Why software engineering courses should Studies of Science. 48, 2 (Apr. 2018), 204–231.
include ethics coverage. Communications of the ACM. 57, 3 (Mar. 2014), 23– [115] Stark, L. 2019. Facial recognition is the plutonium of AI. XRDS: Crossroads,
25. The ACM Magazine for Students. 25, 3 (Apr. 2019), 50–55.
[82] Ngai, S. 2002. “A Foul Lump Started Making Promises in My Voice”: Race, [116] Stark, L. 2018. Facial recognition, emotion and race in animated social media.
Affect, and the Animated Subject. American Literature. 74, 3 (2002), 571–602. First Monday. 23, 9 (Sep. 2018).
[83] Nissenbaum, H. 2010. Privacy in Context: Technology, Policy, and the [117] Stark, L. 2016. That Signal Feeling: Emotion and Interaction Design from
Integrity of Social Life. Stanford Law Books. Social Media to the “Anxious Seat.” Doctoral Dissertation, New York
[84] Olteanu, A., Castillo, C., Diaz, F. and Kiciman, E. 2016. Social Data: Biases, University.
Methodological Pitfalls, and Ethical Boundaries. (Dec. 2016), 1–44. [118] Stark, L. 2016. The emotional context of information privacy. The Information
[85] Papacharissi, Z. 2014. Affective Publics: Sentiment, Technology, and Politics. Society. 32, 1 (2016), 14–27.
Oxford University Press. [119] Stromfelt, H., Zhang, Y. and Schuller, B.W. 2017. Emotion-augmented
[86] Parisi, D. 2018. Archaeologies of Touch: Interfacing with Haptics from machine learning: Overview of an emerging domain. ACII. (2017), 305–312.
Electricity to Computing. University of Minnesota Press. [120] Terry, N.P. 2014. Big Data Proxies and Health Privacy Exceptionalism. 24
[87] Pasquale, F. 2015. Privacy, Autonomy, and Internet Platforms. Privacy in the Health Matrix 65 (2014), 1–45.
Modern Age The Search for Solutions. M. Rotenberg, J. Horwitz, and J. Scott, [121] The Science of “Inside Out”: 2015.
eds. 165–173. [Link]
[88] Pasquale, F. 2015. The Black Box Society. Harvard University Press. [Link]. Accessed: 2015-07-06.
[89] Physiognomy’s New Clothes: 2018. [122] Tomkins, S. and Izard, C.E. 1965. Affect, Cognition, and Personality:
[Link] Empirical Studies. Springer.
Accessed: 2018-02-03. [123] What happens when cars get emotional?: 2019.
[90] Piantadosi, S., Byar, D.P. and Green, S.B. 1988. The Ecological Fallacy. [Link]
American Journal of Epidemiology. 127, 5 (May 1988), 893–904. make-our-roads-much-safer. Accessed: 2021-01-17.
[91] Picard, R.W. 2000. Affective Computing. The MIT Press. [124] Williamson, B. 2017. Moulding student emotions through computational
[92] Picard, R.W. and Cosier, G. 1997. Affective intelligence — the missing link? psychology: affective learning technologies and algorithmic governance.
BT Technology Journal. 15, 4 (Oct. 1997), 150–161. Educational Media International. 10, 1 (Nov. 2017), 1–22.
[93] Picard, R.W. and Klein, J. 2002. Computers that recognise and respond to user [125] Wilson, E.A. 2010. Affect and Artificial Intelligence. University of Washington
emotion: theoretical and practical implications. Interacting with Computers. Press.
14, (2002), 141–169. [126] Wilson, E.A. 2015. Gut Feminism. Duke University Press.
[94] Picard, R.W. and Scheirer, J. 2001. The Galvactivator: A Glove that Senses [127] Yadollahi, A., Shahraki, A.G. and Zaiane, O.R. 2017. Current State of Text
and Communicates Skin Conductivity. Proceedings from the 9th International Sentiment Analysis from Opinion to Emotion Mining. ACM Computing
Conference on Human-Computer Interaction (New Orleans, LA, 2001), 1–6. Surveys. 50, 2 (Jun. 2017), 1–33.
[95] Posner, J., Russell, J.A. and Peterson, B.S. 2005. The circumplex model of [128] Zuboff, S. 2019. The Age of Surveillance Capitalism. PublicAffairs/Hachette.
affect: An integrative approach to affective neuroscience, cognitive [129] “Dehumanising, impenetrable, frustrating”: the grim reality of job hunting in
development, and psychopathology. Development and psychopathology. 17, 3 the age of AI: 2018.
(2005), 715–734. [Link]

792
 FAccT’21, March 3–10, 2021, Virtual Event, Canada L. Stark and J. Hoey

impenetrable-frustrating-the-grim-reality-of-job-hunting-in-the-age-of-ai.
Accessed: 2018-03-05.
[130] “I created Steve Bannon”s psychological warfare tool’: meet the data war
whistleblower : 2018. [Link]
war-whistleblower-christopher-wylie-faceook-nix-bannon-trump. Accessed:
2018-03-18.
[131] 2016. Letter on Proposed Changes to the Common Rule.
[132] 2018. The overly Social Network: Why the real villain in the Cambridge
Analytica story might be Facebook | CBC Radio. (2018).
[133] 2007. What is Emotion? Jerome Kagan, Ed. Yale University Press.

793