See discussions, stats, and author profiles for this publication at: [Link]

net/publication/359619265

Introduction to Socially Interactive Agents

Chapter · September 2021

DOI: 10.1145/3477322.3477323

CITATIONS READS

5 457

1 author:

Birgit Lugrin
University of Wuerzburg
190 PUBLICATIONS 2,378 CITATIONS

SEE PROFILE

All content following this page was uploaded by Birgit Lugrin on 08 April 2022.

The user has requested enhancement of the downloaded file.

 Introduction to
Socially Interactive Agents
Birgit Lugrin

Author note:
This is a preprint. The final article is published in
“The Handbook on Socially Interactive Agents”
by ACM books.

Citation information:
Lugrin, B. (2021). Introduction to Socially Interactive Agents. In B. Lugrin, C. Pelachaud, D. Traum (Eds.), Handbook on
Socially Interactive Agents – 20 Years of Research on Embodied Conversational Agents, Intelligent Virtual Agents, and Social
Robotics, Volume 1: Methods, Behavior, Cognition (pp. 1-18). ACM books.
DOI of the final chapter: 10.1145/3477322.3477324
DOI of volume 1 of the handbook: 10.1145/3477322

Correspondence concerning this chapter should be addressed to Birgit Lugrin, [Link]@[Link]

1 Introduction to
Socially Interactive Agents
Since the commercialization of graphical user interfaces in the late 1980s, the way humans
interact with computers has been dominated by interaction through windows, icons, menus,
pointers (WIMP) interfaces, with buttons that can be clicked and information that can be read
or watched in separate windows. The research discipline of Human-Computer Interaction
(HCI) is constantly developing new and creative systems that go beyond this traditional
interaction for a more intuitive usage, e.g. with technology such as touch interaction, virtual
reality, tangible computing and many more.
Taking a different approach to realize natural and intuitive interaction, the research area
of Socially Interactive Agents (SIAs) aims to develop artificial agents that can interact via
communication channels that come more natural to human interactants, by equipping the
interface with a body that interacts multi-modally by using verbal, para-verbal and non-verbal
behaviours. With it, communication styles that are known from human face-to-face interaction
can be transferred to the interaction with machines.
SIAs (see Figure 1.1 for examples) have been developed under different names in different
research fields such as Intelligent Virtual Agents, Embodied Conversational Agents, or Social
Robotics (see below for definitions of the respective terms). More than 20 years of research
and development in these fields have drastically advanced the state of the art. For this book,
we chose to use the term Socially Interactive Agents (or SIAs) as it includes both physical
and virtual embodiments, while highlighting their ability for social interaction, as well as the
need to realize socially intelligent, autonomous behaviours.

We define SIAs as follows:

Socially Interactive Agents (SIAs) are virtually or physically embodied agents that
are capable of autonomously communicating with people and each other in a
socially intelligent manner using multi-modal behaviours.
In order to interact with humans in a socially intelligent manner, underlying concepts such
as emotions, empathy or how to behave in a group are essential for SIAs to interpret. To be
part of the interaction, observed input must be reasoned about, and decisions to be taken upon
that resemble a cognitive process. The SIA’s (re)actions need to be externalized by natural
language, expressive speech and non-verbal behaviors.

1
 2 Chapter 1 Introduction toSocially Interactive Agents

Figure 1.1 Examples of Socially Interactive Agents: Intelligent Virtual Agents (left) and Social Robots
(right). SIAs in both figures are located in the same virtual vs. physical office space (reflected
reality) [Eckstein et al. 2019], used for various research in the Media Informatics Lab of
Wuerzburg University (left to right: two female agents and a male agent by Autodesk, partly
adapted by features such as clothing style, Pepper by SoftBank Robotics, Reeti by Robopec,
Nao by SoftBank Robotics)

1.1 Potential of SIAs

The right choice of interface is not a simple one. While traditional WIMP interaction is
certainly fast and well established, it is still rather well suited for simple, repetitive tasks, for
example in office work. Many of today’s challenging tasks have led to novel solutions, such as
sophisticated 3D interfaces to help visualize 3dimensional problems. Analogue in scenarios,
where social sensitivity and conversation are paramount, the natural communication with SIAs
might be best suited.
Thanks to extensive research, today prototypes including SIAs are used in many applica-
tion domains that are helpful for individuals or society, with SIAs serving as companions or
assistants in ageing support, health education, life-long learning, or training of specific skills.
On the long run, SIAs are envisioned to unobtrusively support humans in their daily lives.
Figure 1.2 illustrates that vision by extending a well-known humoristic illustration in HCI,
of how humans had to adapt for interaction with machines, by adding a future perspective of
how technology in the form of a SIA adapts to human-style interaction.
In some cases, using a SIA might even have advantages over a human communication
partner. For example, in a tutoring scenario with a SIA, an emotional distance can be kept and
a user might not feel embarrassment, e.g. to admit that he or she cannot read. In addition, a
training task can be repeated as often as liked, without the risk of annoying a human training
partner, or having to pay for each additional lesson, providing individualized sessions to social
groups that usually might not have access to private training. Also, there is a dichotomy
 1.1 Potential of SIAs 3

Figure 1.2 Vision: progressing from having to adapt to interact with technology, to a more natural
communication with SIAs that assist people in their everyday lives (based on [Zallinger
1965]. Humoristic extensions and silhouette versions of the Zallinger image have become
known as ”The March of Progress” that has achieved iconic status so that there are many
different versions of the March of Progress used today.).

between appearance and behavior for SIAs, allowing modification of background factors such
as age, gender, personality, or ethnic background separately from the implemented role and
communicative behavior. This can be useful to personalize a SIA to provide the best possible
solution for each user’s specific requirements or preferences.
Besides the many useful applications SIAs are (envisioned to be) employed in, they can
serve as a research paradigm. In perception studies, SIAs can serve as stimulus material. With
it, they can help learn more about humans, their judgements, preferences, or emotional reac-
tions to artificially created, yet very standardized variations of social situations. In interaction
studies, SIAs can serve as communication partners, allowing for high control over the experi-
ment, ensuring detailed consistent behavior over many sessions. That way, the social behavior
of humans, and the effects of different behaviors they are confronted with, can be studied.
A concern or fear that many researchers in the research area of SIAs are confronted with is
the conception that these agents might be developed to replace humans, in the workplace or
even in social relationships. It is very important to note here that the replacement of humans
is not, and has never been, a goal in the development or research on SIAs. On the contrary,
SIAs are developed to support humans and assist in situations where no human support can be
provided or is not desired, and to offer additional functionalities or support in social domains.
As they aim to provide a more human-like interface that is intuitive to understand and interact
with, they might be replacing other devices that might appear complicated to certain users for
certain tasks. We want to further highlight that, particularly since SIAs enter social domains,
development has to follow interdisciplinary approaches and methods, and needs to include,
besides the technical know-how, expertise in psychology, sociology and ethics.
 4 Chapter 1 Introduction toSocially Interactive Agents

1.2 Terminology
Since research on SIAs is manifold and researchers are coming from different disciplines and
research areas, a number of terms exist that can be found in the literature. In the following,
we aim to shed light on the terminology (in alphabetical order), and highlight their origin and
different foci, albeit you might find some of the definitions quite similar:
Agent ”An agent is anything that can be viewed as perceiving its environment through
sensors and acting upon that environment through actuators” [Russell and Norvig 2009].
This very classic and well known definition looks at agents from the perspective of Artificial
Intelligence (AI), highlighting the autonomy of the artificial entities. Those agents can be, but
are not necessarily, embodied. Examples include softbots, thermostats, robots or humans.
Avatar An avatar represents a game unit that is under the player’s control [Kromand 2007],
which is usually the graphical representation of the user in the virtual environment [Trepte and
Reinecke 2010]. Unfortunately, this term is often confused with virtual or robotic agents in
communities other than SIAs. Note that an avatar is not behaving or interacting autonomously
with a user, but representing the user in the virtual or real world. The term embodiment also
has a different meaning concerning avatars, and describes the physical process to substitute
(parts of) a person’s body with a virtual one by the deployment of virtual reality (VR) hard-
and software [Spanlang et al. 2014].
Embodied Conversational Agent ”Embodied conversational agents are computer-generated
cartoonlike characters that demonstrate many of the same properties as humans in face-to-
face conversation, including the ability to produce and respond to verbal and nonverbal
communication” [Cassell et al. 2000]. The term was defined by Cassell and colleagues in
their same named book on the topic in 2000. The authors highlight the importance of the
combination of the multi-modal interface, with a software agent and a dialogue system, to
assure natural conversation. While the original focus was on virtual embodiments, the term
also allows robotic embodiments, and is used in both fields.
Intelligent Virtual Agent ”Intelligent virtual agents are interactive digital characters that
exhibit human-like qualities and can communicate with humans and each other using natural
human modalities like facial expressions, speech and gesture. They are capable of real-time
perception, cognition, emotion and action that allow them to participate in dynamic social
environments” [IVA 2019]. This term focuses on communicative, digital characters and is
mainly used by researchers that are affiliated with the IVA conference series. An important
fact lies on the character’s intelligence that allows them to dynamically interact, as opposed
to scripted behavior.
Socially Assistive Robot Socially Assistive Robots were defined in [Feil-Seifer and Matarić
2005] as robots that share characteristics with assistive robots, in particular to provide assis-
tance to users, but are distinguished by their focus on social interaction while assisting people.
 1.2 Terminology 5

Socially Intelligent Agent ”The field of socially intelligent agents is characterized by agent
systems that show humanstyle social intelligence” [Dautenhahn et al. 2002]. The term was
coined by Dautenhahn in the late 1990’ and highlights the specific social intelligence of the
agent, relying on ”deep models of human cognition and social competence” [Dautenhahn
1998] that needs to comprise strongly interdisciplinary approaches. Different embodiments of
these agents are possible, virtual or robotic.

Socially Interactive Robot Socially interactive robots were defined as ”robots for which
social interaction plays a key role ” [Fong et al. 2003] in order to ”distinguish these robots
from other robots that involve “conventional” human–robot interaction, such as those used
in teleoperation scenarios” [Fong et al. 2003]. This term was defined after the definition of
socially intelligent agents, to highlight the need for social interaction.

Socially Interactive Agent The term socially interactive agents extends the term socially in-
teractive robot, by allowing virtual and physical embodiments. This term was used by the AA-
MAS (autonomous agents and multiagent systems) community and conference series, where
they are described as ”capable of interacting with people and each other using social com-
municative behaviors common to human-human interaction. Example applications include
social assistants on mobile devices, pedagogical agents in tutoring systems, characters in in-
teractive games, social robots collaborating with humans and multimodal interface agents for
smart appliances and environments”[AAMAS 2019].

Social Robot ”Social (or Sociable) robots are designed to interact with people in a natural,
interpersonal manner [...] They will need to be able to communicate naturally with people
using both verbal and non-verbal signals. They will need to engage us not only on a cognitive
level, but on an emotional level as well in order to provide effective social and task-related
support to people” [Breazeal et al. 2016]. Social robotics is distinguished from robotics
through its socially interactive focus with applications in domains such as education, ageing
support or entertainment. This term is predominantly used by the social robotics community
and the same named conference series and journal.

Virtual Character The term Virtual Character focusses on a virtual representation of a figure
along with its animations. ”Virtual characters in animated movies and games can be very
expressive and have the ability to convey complex emotions” [McDonnell et al. 2008]. Note,
that they do not necessarily have to be intelligent or interactive, c.f. characters of a movie.
Thus, the term is often used by researchers who focus on the character’s appearance, graphics,
animation or background story.

Virtual Human ”Virtual Humans are artificial characters who look and act like humans
but inhabit a simulated environment” [Traum 2008]. The term focusses on the human-like
appearance and behavior and is frequently used by US American authors and research groups.
 6 Chapter 1 Introduction toSocially Interactive Agents

Research on virtual humans often relies on highly realistic graphical representations of the
characters and their animations.
Please note, that the terms introduced above are the ones most commonly used. Other
variations, e.g. affective embodied agent, companion robot, conversational robot, relational
agent, social embodied agent, socially intelligent robot, socially intelligent virtual agent,
virtual agent, and so on, are also found in the literature and address similar research topics.
For the scope of this book, we use the term Socially Interactive Agents (SIAs) when we
talk about both kinds of embodiment, virtual or robotic. We chose this term as we think it
highlights the socially interactive nature as well as the intelligent background of the agent.
We use the term Intelligent Virtual Agent (or IVA) in instances where we discuss virtual
representations of SIAs solely. We use the term Social Robot (or SR) when we discuss robotic
representations of SIAs solely.

1.3 Origin and Embodiment

To understand why there is a large variety of terms, and why research in the fields of IVAs and
SRs might seem distinct sometimes, it is informative to have a look at their different origins.
IVAs originated from the idea of simulating human communication with the display of
human-like communication channels such as facial expressions or gestures, and became fea-
sible in the 1990s due to advances in computer graphics. With constant advances in computer
graphics and the integration of AI methods and cognitive modelling, the communicative abil-
ities and social behaviors of IVAs have constantly been driven further.
SRs on the other hand originated from robotics. Since robots are leaving industrial ap-
plications and are entering private households, a closer interaction with the user in a private
and social domain is imposed. Social behavior and skills that are acceptable for humans are
becoming a requirement (referred to as a ‘robotiquette’ in [Dautenhahn 2007]). Thus, SRs
are robots that are specifically designed to interact with people in an inter-personal manner,
including the need to recognize and generate verbal and non-verbal signals, e.g. [Breazeal
2002].
In simple terms, one could say that in IVA research the virtual body was introduced to
be able to simulate human behavior, while in SR research the physical body of the robot
was naturally there and needed to be adapted for human-like behavior when interpersonal
interaction was desired.
Despite their different origins, the fields of IVAs and SRs today follow the same goals and
are employed in similar domains. To a certain extent, they also share common underlying
technologies, such as text-to-speech systems, computer vision or emotion detention. Also the
theoretical background from psychology or the social sciences are shared for the computa-
tional modelling of cognitive processes such as empathy. Particularly in these areas, research
from IVAs and SRs can benefit greatly from one another.
 1.3 Origin and Embodiment 7

Other aspects are not as easily transferable from one field to the other. The key difference is
the environment that the SIA inhabits and with it whether or not they share the same physical
space with their human interaction partners. Particularly when it comes to human perception
of the SIA or the acceptance thereof, the different type of embodiment seems to play a key
factor. Also the translation from high-level behavior (e.g. show agreement) to the concrete
execution with the particular body part (e.g. nod head and smile) differs across embodiments.
Also, hybrid versions are available today, where a SR contains a display on the head that
shows a virtual face.

Either type of embodiment has certain advantages that might be a disadvantage of the other,
but does not necessarily have to be. Some of the characteristics are listed below:

Characteristics of a virtual embodiment:

• Appearance: The look of an IVA can be freely customized and adapted for different users,
applications or contexts.
• Animation: The virtual face and body can be animated very fine grained in a realistic
manner and show a large variety of emotional expressiveness.
• Acceptance: IVAs are often described as non-threatening. A ’safe setting’ can be created
through the separation of the environment that is inhabited by the human and the IVA.
• Duplication: Applications with IVAs can be duplicated easily and provided to many
users.
• Easy Access to Implementation: Since very good tools are available for free, students
and practitioners who want to get acquainted with the research area of SIAs can have
easy access to build their own IVA architecture or application. The only requirement to
get started is the availability of a computer.
• Easy Access to Use: Applications with IVAs can be implemented for usage with tra-
ditional computers or mobile phones. This way, the IVA can be deployed anywhere,
anytime on people’s private devices.

Characteristics of a physical embodiment:

• Appearance: With commercial SRs, the options to customize the appearance are limited
(e.g., by adding stickers or accessories). However, today’s opportunities with 3D printers
and single-board computers allow designing individual SRs at a rather low cost.
• Animation: The options to animate a SR is dependent on the particular model and its
individual degrees of freedom (e.g., whether it has limbs or an animate-able face). Due
to hardware limitations subtle emotional expressions might not be feasible.
 8 Chapter 1 Introduction toSocially Interactive Agents

• Acceptance: It has been widely reported that the physical presence of a SR has a positive
effect on the perception of users, and in particular their feeling of social presence, e.g.
[Breazeal et al. 2016].
• Mobility: The most dominant advantage of a robot’s physical body lies in its ability to
move around in the real world and conduct physical interaction with the environment. A
SR can, for example, provide services such as serving food or beverages. However, the
physical body also provides challenges, such as the risk of accidentally falling over.
• Physical Interaction: In addition to conversational interaction, physical interaction with
the human user is possible (e.g. by performing social touch). The shared space can
additionally be used for conversational purposes (e.g. to gain someone’s attention).
Despite the characteristics that are implied by the embodiment of a SIA, a number of
studies have directly compared physical and virtual embodiments to evaluate the outcomes of
similar interactions with users, see for example [Deng et al. 2019] for an overview. Mainly in
these comparisons, the virtual SIA is a direct transfer of a SR into a virtual representation of
the same robot. It seems as the physical embodiment of a SR outperforms a virtual one, both
in task performance and the perception of the users. However, results are more inconclusive,
if the concepts of physical presence and embodiment are separated, by either comparing
physically present SIAs to virtually present SIAs, or comparing physical SIAs with virtual
SIAs both presented on a screen [Li 2015].
While directly comparing virtual and physical representations of SRs is a valid research
paradigm that allows comparing between embodiments and the impact of physical presence
in very controlled settings, from a practical perspective, the design of a ’virtual social robot’
would not be beneficial. With it, most advantages of an IVA are out-ruled, and the virtual
representation is artificially bound to non-existing, virtual hardware limits. Advantages such
as subtle animations, duplication, or potential usage on smart-phones are neglected. To date,
the number of studies that compare three or more representations, or compare a state-of-the-
art SR against a state-oft-the-art IVA are rare.
It also needs to be noted that moderating factors such as the interaction scenario and task,
and the user’s perception of the SIA’s body-related capabilities seem to play a crucial role in
people’s ratings of the SIAs [Hoffmann et al. 2018]. The right choice of embodiment of a SIA
is thus highly complex and dependent on many factors such as the situational context, role of
the SIA, purpose of the application, or user’s preference.

1.4 Purpose of the Book

The fields of IVAs and SRs face similar research issues and challenges and are further
developed in universities and research facilities across the world. Research on IVAs and
SRs can benefit greatly from one another and have contributed to each other’s advancement
in the past. However, substantial work in both research fields is sometimes overlooked by
 1.5 Structure of the Book 9

researchers in the other area. This is partly due to the fact, that different wordings are used and
there exists a large number of journals and conferences that publish works on SIAs, making it
very difficult to maintain a good overview.
The interdisciplinary nature of SIA research also contributes to the very diverse venues
where you can find relevant findings on SIAs. While researchers from the cognitive sciences
bring expertise in underlying processes, communication, and interaction, computer scientists
bring expertise in conceptualizing computational models and implementation. Even within a
single discipline, approaches, methods, and wording can be used differently, thereby com-
plicating cooperation. In computer science, for example, many areas are involved in SIA re-
search, such as artificial intelligence, human-computer interaction, robotics, computer graph-
ics, or software engineering. Only through communication and research in interdisciplinary
teams the field can be advanced. This constitutes one major challenge by itself, as researchers
sometimes do not have enough insights into other areas (or even disciplines), and thus might
not appreciate each other’s work enough.
We hope that this handbook will help raise the visibility of the research in the fields
involved and further close the gap between the IVA and SR communities. At the same
time we hope that, in the future, reinventing the wheel can be avoided. This comprehensive
handbook on Socially Interactive Agents (SIAs) summarizes the research that has taken place
over the last 20 years. We are referring to this time period, since the first complete book
on Embodied Conversational Agents ([Cassell et al. 2000], see above) appeared in 2000,
although we are aware that research on this topic began earlier. By pointing out current
challenges and future directions in the various topics involved, we hope to help directing future
research and cooperation. In the book, we include views from an interdisciplinary perspective,
containing theoretical backgrounds from human-human interaction, their implementation in
computational models, their evaluation with human users, integration into applications, and
ethical implications.
In a structured and easily accessible way, the book (hopefully) provides a valuable source
of information on SIAs for research and education. Researchers in the research area of SIAs
will find it a valuable overview of the field. Teaching staff will benefit from the handbook
to structure courses for undergraduate or graduate students, and with it train the upcoming
generations of young researchers.
Particularly now, the public interest in SIAs is increasing. The book will also help profes-
sionals, and interested lay public readers, to get acquainted with this research area.

1.5 Structure of the Book

This handbook is divided into two volumes, including 28 chapters that are grouped in five
major parts, to cover the major topics in the area. For the book, we have relied on our
connections to both fields, IVAs and SRs, providing a collection of surveys, each written
by (an) acknowledged international expert(s) of their field.
10 Chapter 1 Introduction toSocially Interactive Agents

Each chapter provides a survey that summarizes the theoretical background, approaches for
implementation, history / overview of the topic, alongside with current challenges and future
directions. All the chapters discuss similarities and differences between IVAs and SRs and
highlight important work of both fields. Where applicable, the chapters will follow a common
structure to ensure internal consistency and facilitate understanding.

Volume 1
After this first chapter that introduces readers to the handbook, volume 1 starts with Part I
”Establishing SIA Research” that helps understand how research in this area is conducted
and discusses the impact thereof on individuals and society.
Chapter 2 ”Empirical Methods in the Social Science for Researching Socially Interactive
Agents”, by Astrid Rosenthal-von der Pütten and Anna M. H. Abrams, introduces the em-
pirical methodology from the social sciences that is necessary for SIA research, particularly
when it comes to research experiments including human participants.
Chapter 3 ”Social Reactions to Socially Interactive Agents and their Ethical Implications”,
by Nicole Krämer and Arne Manzeschke, looks at SIA research from a psychological and
ethical perspective. It points to numerous studies demonstrating that people (unconsciously)
react socially towards artificial entities, and that as soon as they display social cues, people
can also be manipulated or socially influenced.

Part II ”Appearance and Behavior”, deals with the impact of the looks of SIAs and the
various aspects of multi-modal behavior that need to be taken into account when convincing
SIAs behavior is modelled.
Chapter 4 ”Appearance”, by Rachel McDonnell and Bilge Mutlu, argues that compared
to voice assistants, embodied agents enable the use of appearance-based cues from human-
human interaction, such as mutual gaze, that are known to improve social outcomes. The
chapter shows that the appearance of an SIA can affect how people perceive, respond to, and
interact with it.
Chapter 5 ”Natural Language Understanding in Socially Interactive Agents”, by Roberto
Pieraccini, introduces natural language understanding as an essential part of any interactive
agent and highlights its complexity, particularly for SIAs that need to react to user initiated
interactions across various application areas.
Chapter 6 ”Building and Designing Expressive Speech Synthesis”, by Matthew Aylett,
Leigh Clark, Benjamin R. Cowan and Ilaria Torre, gives an overview of definitions, methods
and state-of-the art in expressive voices, and critically discusses when and where expressive
speech is beneficial.
Chapter 7 ”Gesture Generation”, by Carolyn Saund and Stacy Marsella, discusses the
complexity of communicative gestures and how they enhance communication in human-
 1.5 Structure of the Book 11

human conversation, and summarizes the research and their challenges in the transfer of this
complexity in the implementation with SIAs.
Chapter 8 ”Multimodal Behavior Modelling for Socially Interactive Agents”, by Catherine
Pelachaud, Carlos Busso and Dirk Heylen, extends the theme non-verbal behavior by adding
additional modalities such as gaze, smiles or social touch. Starting from introducing concepts
from the social sciences, the chapter has a strong focus on the different computational models
that can be employed for the implementation of multimodal behaviors.

Part III ”Social Cognition - Models and Phenomena” investigates internal processes
known from human cognition that are driving forces in human-human interaction, and demon-
strates how they are addressed in SIA systems.
Chapter 9 ”Theory of Mind and Joint Attention”, by Jairo Perez-Osorio, Eva Wiese
and Agnieszka Wykowska, introduces the two crucial mechanisms of social cognition, and
explains how they apply to the interaction between humans and SIAs from two angles:
evoking human social cognition, and modelling artificial social cognition.
Chapter 10 ”Emotion”, by Joost Broekens, focuses on the computational representation
of emotion and other related affective concepts such as mood, attitude, or appraisal and
highlights how SIAs can make constructive use of them.
Chapter 11 ”Empathy and Prosociality in Social Agents”, by Ana Paiva, Filipa Correia,
Raquel Oliveira, Fernando Santos and Patrı́cia Arriaga, focuses on empathy and in particular
on the related concept of prosociality (conducting positive and voluntary behavior that should
benefit others). With it, the authors provide a framework including the main variables needed
to design prosocial agents, for individual or dyadic interactions, or at the society level.
Chapter 12 ”Rapport Between Humans and Socially Interactive Agents”, by Jonathan
Gratch and Gale Lucas, introduces rapport (a fine grained emotional communicational in-
terplay) in the communication of humans and machines, by approaching it from a theoretical,
computational and empirical side, and demonstrating its benefits.
Chapter 13 ”Culture for Socially Interactive Agents”, by Birgit Lugrin and Matthias Rehm,
argues that implementing culture for SIAs can be beneficial not only to raise their acceptance
in certain user groups, but also to be able to teach about cultural differences, and foster cul-
tural diversity.

Volume 2
The second volume of this handbook starts with a preface that recaps the most important
aspects and terminology of its introduction chapter. Part IV ”Modelling Interactivity”
explains how interaction with human users or other SIAs is modelled, and how the many
detailed aspects of multimodal, multiparty, adaptive interactivity are implemented.
12 Chapter 1 Introduction toSocially Interactive Agents

Chapter 14 ”Interaction in Social Space”, by Hannes Högni Vilhjálmsson, deals with the
intricate social performance that inevitably takes place when SIAs and human users share the
same social space (virtual or physical), regardless of their explicit intentions to connect with
one another.
Chapter 15 ”Dialogue for Socially Interactive Agents”, by David Traum, introduces several
approaches to modelling the structure of extended verbal and multimodal interactions, with
an emphasis on how different kinds of embodiment impact the communication affordances
and requirements for SIA tasks.
Chapter 16 ”The Fabric of Socially Interactive Agents — Multimodal Interaction Archi-
tectures”, by Stefan Kopp and Teena Hassan, presents different SIA architectures and gives
an extensive overview on how SIAs can engage in dynamic and fluid social interaction, dis-
cussing different approaches to deal with multimodality and interactivity.
Chapter 17 ”Multiparty Interaction Between Humans and Socially Interactive Agents”, by
Sarah Gillet, Marynel Vázquez, Christopher Peters, Fangkai Yang and Iolanda Leite, looks
into SIAs that interact with a group of people for which the complex group dynamics need to
be understood, and highlights that the SIA can affect and even explicitly influence the group’s
dynamics.
Chapter 18 ”Adaptive Artificial Personalities”, by Kathrin Janowski, Hannes Ritschel and
Elisabeth André, focuses on how a SIA can automatically adapt its personality in accordance
with the user’s preferences, and with it make the interaction with them more enjoyable and
productive.
Chapter 19 ”Long-term Interaction with Relational Socially Interactive Agents”, by
Jacqueline M. Kory-Westlund, Cynthia Breazeal, Hae Won Park and Ishaan Grover, argues
that strong relationships support people in archiving their goals in various domains, and that
thus relational SIAs have the potential to scaffold humans in their long-term endeavors.
Chapter 20 ”Platforms and Tools for Socially Interactive Agent Research and Develop-
ment”, by Arno Hartholt and Sharon Mozgai, gives a practical introduction to the history
of SIA platforms and tools directing to state-of-the-art technical solutions that support the
development and implementation of SIAs.

Part V ”Areas of Application” gives an overview of the major domains in which SIAs
are employed, directing to systems and research findings, highlighting the benefits of SIAs to
individuals and society.
Chapter 21 ”Pedagogical Agents”, by H. Chad Lane and Noah L. Schroeder, introduces
work with SIAs in the domain of education, examining social aspects of teaching and learning
and summarizing empirical research with pedagogical agents.
Chapter 22 ”Socially Interactive Agents as Peers”, by Justine Cassell, describes work
that uses SIAs that are designed to work or play with children or teenagers at an eye-level,
 1.5 Structure of the Book 13

discussing the benefits of SIAs that look and act like peers rather than teachers, tutors, or
parents.
Chapter 23 ”Socially Interactive Agents for Supporting Aging”, by Moojan Ghafurian,
John Edison Munoz Cardona, Jennifer Boger, Jesse Hoey and Kerstin Dautenhahn, is centred
on work with SIAs located in the area of aging support that aim to improve older adults’
quality of life and wellbeing. The chapter provides methods and suggestions to address the
many challenges that arise when designing SIAs that should successfully assist the targeted
user group.
Chapter 24 ”Health-related Applications of Socially Interactive Agents”, by Timothy
Bickmore, addresses another area of major societal importance, and highlights the potential
of SIAs that have shown to have a positive impact on voluntary changes in health behavior.
Chapter 25 ”Autism and Socially Interactive Agents”, by Jacqueline Nadel, Ouriel Grynsz-
pan and Jean-Claude Martin, reviews work that uses SIAs to study or help improve the social
skills of people with autism spectrum disorder. The chapter highlights the improvements that
have been achieved throughout the last two decades and that, following a multi-disciplinary
approach, more can be expected in the future.
Chapter 26 ”Interactive Narrative and Story-telling”, by Ruth Aylett, introduces narra-
tive and storytelling as fundamental human capabilities, and outlines how SIAs are used in
character-or plot-based systems, highlighting the great challenge of interactivity in this do-
main.
Chapter 27 ”Socially Interactive Agents in Games”, by Rui Prada and Diogo Rato, dis-
cusses the complexity in which SIAs have been used in games, and introduces their different
roles alongside with their contributions to gameplay.
Chapter 28 ”Serious Games with Socially Interactive Agents”, by Patrick Gebhard, Dimitra
Tzovaltzi, Tanja Schneeberger and Fabrizio Nunnari, focuses on serious games that can partly
be seen as a means to an end to achieve certain goals in various domains (such as education or
health-behavior change) using specific methods from games and interactive narratives. Thus,
the chapter focuses on learning gain as well as individual experience during game play.
 14 Chapter 1 Introduction toSocially Interactive Agents

1.6 Further Readings

Since the research area of SIAs is interdisciplinary, and researchers approach it from different
angles and disciplines, a large number of books, conferences and journals present work on
SIAs. Below, we suggest further readings (in alphabetical order), but do not claim that the list
is complete.

A few books have appeared that focus on SIAs:

• C. Bartneck, T. Belpaeme, F. Eyssel, T. Kanda, M. Keijsers, S. Sabanovic, Human-

Robot Interaction – An Introduction, Cambridge University Press, 2019.
• C. Breazeal, Designing Sociable Robots, MIT Press, 2002.
• J. Cassell, J. Sullivan, and S. Prevost, Embodied Conversational Agents, The MIT
Press, 2000.
• K. Dautenhahn, A. Bond, L. Canamero, and B. Edmonds, Socially Intelligent Agents,
Springer, 2002.
• J. Gratch and S. Marsella, Social Emotions in Nature and Artifact, Oxford University
Press, 2014.
• N. Magnenat-Thalmann and D. Thalmann, Handbook of Virtual Humans, Wiley, 2007.

In addition, chapters on SIAs are part of broader handbooks:

• R. Calvo, S. D’Mello, J. Gratch, and A. Kappas, Handbook on Affective Computing,

Oxford University press, 2015.
Facial Expressions of Emotions for Virtual Characters, M. Ochs, R. Niewiadomski,
and C. Pelachaud
Expressing Emotion Through Posture and Gesture, M. Lhommet and S. Marsella
Emotion Modeling for Social Robots, A. Paiva, I. Leite, and T. Ribeiro
Preparing Emotional Agents for Intercultural Communication, E. André
Affect in Human-Robot Interaction, R. C. Arkin and L. Moshkina
Relational Agents in Health Applications: Leveraging Affective Computing to Pro-
mote Healing and Wellness, T. W. Bickmore
• B. Siciliano and O. Khatib, Springer Handbook on Robotics, Springer, 2016.

Cognitive Human–Robot Interaction, B. Mutlu, N. Roy, S. Sabanovic

Social Robotics, C. Breazeal, K. Dautenhahn, T. Kanda
Socially Assistive Robotics, M. Mataric, B. Scassellati
 1.6 Further Readings 15

Likewise a number of conferences addresses work related to SIAs: Please note that, as
opposed to some other research domains, in the area of SIAs, and computer science in general,
conferences and their proceedings are as important as (and sometimes even more important
than) journal papers. High quality conferences have acceptance rates of 15 % or lower. In the
domain of SIAs the following conferences are of relevance, albeit they strongly differ in their
acceptance rates):

• International Conference on Autonomous Agents and Multiagent Systems (AAMAS)

([Link] since 2002, proceedings by IFAAMAS, available by ACM
Digital Library.
• International Conference on Computer Animation and Social Agents (CASA)
([Link] since 2004, proceedings by ACM.
• International Conference on Human-Agent Interaction (HAI) ([Link]
since 2013, proceedings by ACM.
• International Conference on Human-Robot Interaction (HRI) ([Link]
since 2006, proceedings by ACM.
• International Conference on Intelligent Virtual Agents (IVA) ([Link]
since 1998, proceedings by ACM (Springer until 2017).
• International Conference on Robot and Human Interactive Communication (RO-
MAN) ([Link]
man), since 1992, proceedings by IEEE.
• International Conference on Social Robotics (ICSR) ([Link]
since 2010, proceedings by Springer.

A number of journals publishes work related to SIAs:

• ACM Transactions on Human-Robot Interaction ([Link] since

2012, ACM.
• ACM Transactions on Interactive Intelligent Systems ([Link]
since 2011, ACM.
• Autonomous Agents and Multi-Agent Systems ([Link]
since 1998, Springer.
• Computers in Human Behavior ([Link]
behavior), since 1985, Elsevier.
16 Chapter 1 Introduction toSocially Interactive Agents

• Frontiers in Robotics and AI, Section Human-Robot Interaction

([Link]
• IEEE Transactions on Affective Computing ([Link]
since 2010, IEEE.
• International Journal of Human-Computer Studies
([Link]
since 1994, Elsevier.
• International Journal of Social Robotics ([Link] since
2009, Springer.
• Journal on Multimodal User Interfaces ([Link] since
2007, Springer.

Acknowledgements
Many thanks to Catherine Pelachaud and Astrid Rosenthal-von der Pütten for their very
helpful comments on this introductory chapter!
Bibliography
AAMAS, 2019. International Conference on Autonomous Agents and Multiagent Systems (AAMAS):
Socially Interactive Agents Track. [Link]
C. Breazeal. 2002. Designing Sociable Robots. MIT Press.
C. Breazeal, K. Dautenhahn, and T. Kanda. 2016. Springer Handbook of Robotics, chapter Social
robotics, pp. 1935–1972. Springer.
J. Cassell, J. Sullivan, and S. Prevost. 2000. Embodied Conversational Agents. The MIT Press.
K. Dautenhahn. 1998. The art of designing socially intelligent agents: Science, fiction, and the human
in the loop. Applied Artificial Intelligence, 12(7-8): 573–617.
K. Dautenhahn. 2007. Socially intelligent robots: dimensions of human-robot interaction. Philosophical
transactions of the Royal Society of London. Series B, Biological sciences, 362(1480): 679–704. DOI:
[Link]
K. Dautenhahn, A. Bond, L. Canamero, and B. Edmonds, eds. 2002. Socially Intelligent Agents.
Springer.
E. Deng, B. Mutlu, and M. J. Mataric. 2019. Embodiment in socially interactive robots. Foundations
and Trends R in Robotics, 7(4): 251–356. ISSN 1935-8253. [Link]
DOI: 10.1561/2300000056.
B. Eckstein, F. Niebling, and B. Lugrin. 2019. Reflected reality: A mixed reality knowledge represen-
tation for context-aware systems. In 11th International Conference on Virtual Worlds and Games for
Serious Applications (VS-Games). accepted for publication.
D. Feil-Seifer and M. J. Matarić. 2005. Defining socially assistive robotics. In IEEE 9th International
Conference on Rehabilitation Robotics, pp. 465–468.
T. Fong, I. Nourbakhsh, and K. Dautenhahn. 2003. A survey of socially interactive robots. Robotics
and Autonomous Systems, 42: 143–166.
L. Hoffmann, N. Bock, and A. M. R. v.d. Pütten. 2018. The peculiarities of robot embodiment
- development, validation and initial test of the embodiment and corporeality of artificial agents
scale. In ACM/IEEE International Conference on Human-Robot Interaction (HRI’18). DOI:
[Link]
IVA, 2019. International Conference on Intelligent Virtual Agents (ACM IVA).
[Link]
D. Kromand. 2007. Avatar categorization. In Situated Play. The University of Tokyo.
J. Li. 2015. The benefit of beingphysically present: A survey of experimental works comparing
copresent robots, telepresent robots and virtual agents. Int. J. Human-Computer Studies, 77. DOI:
[Link]
R. McDonnell, S. Jörg, J. McHugh, F. Newell, and C. O’Sullivan. 2008. Evaluating the emo-
tional content of human motions on real and virtual characters. In Proceedings of the 5th sym-

17
 18 BIBLIOGRAPHY

posium on Applied perception in graphics and visualization (APGV 08), pp. 67–74. ACM. DOI:
[Link]
S. J. Russell and P. Norvig. 2009. Artificial Intelligence: A Modern Approach. Prentice Hall.
B. Spanlang, J.-M. Normand, D. Borland, K. Kilteni, E. Giannopoulos, A. Pomés, M. Gonzalez-Franco,
D. Perez-Marcos, J. Arroyo-Palacios, X. N. Muncunill, and M. Slater. 2014. How to build an
embodiment lab: achieving body representation illusions in virtual reality. Frontiers in Robotics
and AI, 1(9).
D. Traum. 2008. Talking to virtual humans: Dialogue models and methodologies for embodied
conversational agents. In I. Wachsmuth and G. Knoblich, eds., Modeling Communication with Robots
and Virtual Humans. Springer.
S. Trepte and L. Reinecke. 2010. Avatar creation and video game enjoyment: Effects of life-satisfaction,
game competitiveness, and identification with the avatar. Journal of Media Psychology: Theories,
Methods, and Applications, 22(4).
R. Zallinger. 1965. The road to homo sapiens (illustration). In F. C. Howel, ed., Early Man, pp. 41–45.
TIME-LIFE Books, New York.

View publication stats