AFFECTIVE COMPUTING

Complete Exam Notes — Weeks 1, 2 & 12

NPTEL | IIIT Delhi | Prof. Jainendra Shukla

EXAM STRATEGY — READ FIRST

• Week 1: Definitions, 6 Modalities, 3 App Types, Healthcare/Education apps, 3 Ethics
• Week 2: Emotion definition (4 parts), James' theory, Ekman 6 emotions, VAD/PAD, 4 brain
regions, Asymmetry exception (Anger=Left PFC), Physiology tables
• Week 12: 4 Ethical Considerations, Privacy Invasiveness levels, Anticipatory Computing 4
stages, Open Issues
• Memory Aids: SENSE-RECOGNIZE-RESPOND-INFLUENCE | D-E-P | M-P-D | PAAI | V-
B-R | E-P-P
WEEK 1 — FUNDAMENTALS OF AFFECTIVE COMPUTING
1. What is Affective Computing?
FORMAL DEFINITION (Picard 1997; Picard & Klein 2002)
The field of affective computing encompasses both the creation of and interaction with
machine
systems that SENSE, RECOGNIZE, RESPOND TO, and INFLUENCE emotions.

4-Keyword Breakdown (Exam Trick)

• 1. Sense: Capture raw data (camera, mic, wearable sensor, EEG).
• 2. Recognize: Classify data into an emotion label (happy, sad, stressed).
• 3. Respond: Take appropriate action (play music, recommend a break).
• 4. Influence: Actively modify the user's emotional state (therapy bot calming user).

• Key Person: Rosalind W. Picard — MIT Media Lab; author of 1997 book; founder of the field.

2. Historical Motivation — HAL 9000 (2001: A Space Odyssey, 1968)

• HAL could detect emotional cues (tone, hesitation, lip-reading) and make social decisions.
• Shows the idea of emotionally intelligent machines is DECADES old, pre-dating the formal field.
• Frames the central question: What should an emotion-aware AI be allowed to do?

3. Seven Capabilities of an Exciting AI Vision

If asked to list all 7 in order:
1. Speech — understand spoken language including prosody and tone.
2. Facial recognition — identify people and read facial expressions.
3. Lip reading — supplement speech recognition in noisy environments.
4. Systems monitoring — watch its own sensors and internal state.
5. Regulation and control — manage physical/digital systems safely.
6. Game play — strategic reasoning and planning.
7. Social skills — understand and respond to human emotion. ← AFFECTIVE COMPUTING piece

4. Affect Sensing
DEFINITION (Picard 1997)
Affect sensing refers to a system that can RECOGNIZE EMOTION by receiving data through
SIGNALS and PATTERNS.

• Hardware: Cameras, microphones, EEG caps, GSR/EDA sensors, PPG watches.

• Software: Signal-processing pipelines, feature extractors, ML classifiers → emotion labels.
• Key Idea: Each modality has a UNIQUE SIGNATURE — a characteristic pattern mapping to
emotional states.
5. Six Modalities of Affect-Sensing Systems ★ HIGH YIELD
MEMORIZE ALL SIX — Very Likely Exam Question
1. Facial activity — expressions, micro-expressions, action units (smile, brow raise).
2. Posture & Gesture — body position (slouched vs upright), hand/arm movements.
3. Hand tension — grip pressure, tremor; reflects anxiety or arousal.
4. Vocal — pitch, loudness, speech rate, jitter, shimmer. HOW it is said, not WHAT.
5. Textual — sentiment analysis of written language (NLP).
6. Physiological signals — EDA/GSR (sweat/arousal), EEG (brain activity), ECG, PPG,
respiration.

6. Three Application Types (Picard 1997) ★ HIGH YIELD

Memory Aid: D–E–P

Type Description Example

Detection Systems that DETECT the emotions of Webcam-based system inferring
the user. student confusion.
Expression Systems that EXPRESS what a human KISMET robot showing happy/sad
would perceive as an emotion. face.
Perception Systems that actually 'FEEL' an emotion. Open research question — most
(Philosophically controversial) debated category.

7. Healthcare Applications
7.1 Autism — Asperger Syndrome (AS) & High-Functioning Autism (HFA)
• Individuals with AS/HFA have difficulty reading emotions and self-monitoring behavior.
• SymTrend: Symptom tracking, charting, and reminding.
• Autism Track (HandHold Adaptive): Tracks behaviors: aggression, attentiveness, echolalia,
eye contact, hyperactivity, self-stimming.

7.2 Visharanti — Stress Management (Pipeline — Likely Diagram Question)

VISHARANTI PIPELINE
1. Wearable Device collects PPG (optical heart-rate) & GSR (galvanic skin response) data.
2. Classifier analyzes signals:
→ No Stress: positive acknowledgement (thumbs up) to user.
→ Stress: triggers the recommender path.
3. Recommendation System suggests an activity to reduce stress.
4. User Feedback on the activity is fed back to classifier — CLOSED LOOP system.

7.3 Post-Traumatic Stress Disorder (PTSD)

• Affective computing + VR used for EXPOSURE THERAPY.
 • StartleMart (Holmgard et al., 2013): Game that measures stress/startle reaction in PTSD
patients.
• Virtual Vietnam / Virtual Iraq / Virtual Afghanistan: Immersive VR environments for combat-
related PTSD therapy.

8. Education Applications
• EngageMe (Darnell, 2014): Collects skin-conductance data (EDA/GSR) + video from
classroom. Helps teacher reflect on engagement vs boredom.
• Subtle Stone (Balaam, 2009): Wireless squeezable ball. Students communicate affective state
to teacher in real time (e.g., 'I'm lost').
• China Wants to Raise Smarter Students — classroom cameras + EEG headbands to monitor
attention. Raises privacy/ethics concerns.

9. Other Applications
9.1 MACH — My Automated Conversation coacH
• Virtual-agent system acting as an INTERVIEW COACH.
• Uses computer vision (facial expression), speech analysis, and NLP.
• Feedback on: smiling, eye contact, filler words (um/uh), speaking rate.
• Also helps people with social-skill difficulties (e.g., autism).

9.2 KISMET
• Famous sociable robot from MIT Media Lab (Cynthia Breazeal).
• Category: EXPRESSION — movable eyes, ears, lips, eyebrows showing emotional states.
• Landmark system for human–robot affective interaction.

10. UX vs. Usability

Aspect User Experience (UX) Usability
Core Question How does the user FEEL about you? Ease of use?
Dimension Emotional, brand, aesthetic Practical / functional
Measurement Difficult — emotional & subjective Task time, error rate, completion rate
Goal Memorable, positive feeling Efficient, error-free interaction

• UX + Usability together = complete measure of product experience.

• Motivating question: 'How do you test for BOTH?' → leads to Advanced UX / affective testing.

11. Participatory Design & Acceptance Testing

• Participatory Design: Includes stakeholders (users) in early stages of design. Lends itself to
user-centered design innovation.
• PICTIVE (Muller, 1991): Plastic Interface for Collaborative Technology Initiatives through Video
Exploration. Users co-design using low-fidelity paper/sticky-note materials + video recording.

• Acceptance Testing (Beta Testing): End-user validates product for: Functionality, Usability,
Reliability, Compatibility. Typical method: self-reporting via focus groups or surveys.
12. Problems with Traditional Testing & Neuromarketing
Issues with Focus Groups / Surveys
• Users must articulate exactly how they feel — which is HARD.
• Participant bias — users answer what they think researcher wants to hear.
• Recall bias — memories of experience change over time; users reconstruct, not remember.
• Cannot capture unconscious or fleeting emotional reactions.

Solution — Neuromarketing
Application of affective computing / neuroscience to marketing.
• How do customers use their mind while responding to products?
• How does this impact decision making?
• Goal: gain more accurate measure of customers' functional mind beyond what they can
verbalize.

13. Ethical Considerations in Affective Computing (Week 1 Version)

Memory Aid: M–P–D
THREE ETHICAL ISSUES (Almost Certain Exam Question)
1. EMOTIONAL MANIPULATION — Is it ethical for computers to detect, recognize, then
modify behaviors?
Example: ad platform detects sadness and shows persuasive ads you would normally
resist.
Raises questions of consent and autonomy.

2. PRIVACY — Emotions, perhaps more so than thoughts, are ultimately personal and
private (Picard, 2003).
Who owns emotional data? Can it be used against you (insurance, employment)?

3. EMOTIONAL DEPENDENCY — Could routinely using these 'moral agents' create

codependence?
Risk: users rely on AI for emotional regulation instead of humans or own skills.
WEEK 2 — THEORY OF EMOTION & NEUROSCIENTIFIC
PERSPECTIVES
1. Two Core Questions of Emotion Psychology
• Q1: What ARE emotions themselves? — The nature of an emotion.
• Q2: How are emotions ELICITED or GENERATED? — What triggers them?
Every theory (James, Ekman, Russell, Mehrabian, Barrett, Norman) tries to answer one or both.

2. Formal Definition of Emotion (Kleinginna & Kleinginna, 1981) ★ HIGH

YIELD
DEFINITION — MEMORIZE 4 COMPONENTS
Emotion is a complex set of interactions among subjective and objective factors, mediated
by
NEURAL/HORMONAL systems, which can:
(a) AFFECTIVE EXPERIENCE — give rise to feelings of arousal, pleasure/displeasure.
(b) COGNITIVE PROCESSES — generate perceptual effects, appraisals, labelling.
(c) PHYSIOLOGICAL ADJUSTMENTS — activate widespread bodily changes.
(d) BEHAVIOR — often expressive, goal-directed, and adaptive.

KEY PHRASE: 'mediated by neural/hormonal systems' — emotion is NOT purely mental;

brain AND body involved.

3. Emotion Generation — James' Theory (1894) & Bidirectional Projections

FAMOUS QUESTION
Do we run from a bear because we are AFRAID, or are we AFRAID because we RUN?

James' Answer (Counter-Intuitive but Influential)

• We are afraid BECAUSE we run.
• Bodily responses (racing heart, tight stomach, sweaty palms, tense muscles) come FIRST.
• These physiological responses return to the brain as bodily sensations.
• The unique pattern of sensory feedback gives each emotion its unique quality.

Worcester (1893) — The Cognitive Caveat

• Fear is not directly caused by sense perceptions but by CERTAIN THOUGHTS to which they
give rise.
• A well-armed hunter seeing a bear might feel JOY. An ordinary man might feel CURIOSITY.

Bidirectional Projections (Brain ↔ Body)

• Brain → Body: via visceral EFFERENT pathways (brain tells body to react). Memory: Efferent
= Exits the brain.
 • Body → Brain: through AFFERENT feedback (body tells brain what it's doing). Memory:
Afferent = Arrives at brain.

3 Supporting Evidence Items — Often Asked

8. Laughter Yoga — laughter possible WITHOUT humor/thought; physical act alone triggers
emotion.
9. Strack et al. (1988) — Facial Feedback Hypothesis: voluntary contraction of facial muscles
contributes to emotional experience.
10. Levenson et al. (1990) — Pencil experiment: pencil in LIPS (preventing smile) → rated cartoons
LESS funny. Pencil in TEETH (mimicking smile) → rated cartoons MORE funny.

4. Embodied Cognition
Emotion is a multicomponent response to a significant stimulus characterized by brain and bodily
arousal and a subjective feeling state that elicits a tendency toward MOTIVATED ACTION.

3 Components
11. Significant stimulus — something that triggers the emotion.
12. Subjective feeling state — what it feels like.
13. Motivated action — tendency to act.

Exceptions (Where Components Are Missing)

• No significant stimulus → emotions without cause (unexplained anxiety).
• No subjective feeling state → unconscious emotions.
• No motivated action → e.g., sadness (feel it but do nothing).

5. Perceived vs. Induced Emotion (Gabrielsson, 2002)

Type Description Example
Perceived Emotion Emotion RECOGNIZED in the stimuli. I recognize this song is sad.
Induced / Felt Emotion Emotion actually EXPERIENCED by I actually FEEL sad hearing it.
listener.

• They CAN DIFFER — you might recognize a song as sad but not feel sad yourself.

6. Categorical (Discrete) Model — Darwin & Ekman ★ HIGH YIELD

• Origin: Darwin's evolutionary view (1872) — emotions are universal, adaptive products of
evolution.
• Closest to everyday life — a single word describes an affective state.
• In emotion recognition using classification methods, discrete models account for the largest part
of studies.

EKMAN & FRIESEN (1971) — SIX BASIC EMOTIONS

1. Happiness 2. Anger 3. Disgust 4. Sadness 5. Anxiety (Fear) 6. Surprise
 Later addition: CONTEMPT was proposed as a seventh pan-cultural facial expression.

Pros & Cons

• (+) Computationally easy — classification task.
• (−) Difficult to model RELATIONS between discrete states (how is 'happy' related to 'excited'?).
• (−) Inconsistency — researchers don't agree on which emotions are 'basic'.

7. Dimensional Models — VAD / PAD & Circumplex ★ HIGH YIELD

PAD / VAD MODEL (Russell & Mehrabian, 1977) — THREE DIMENSIONS
1. Pleasure–Displeasure (VALENCE / P) — pleasantness of the emotion.
2. Arousal–Non-Arousal (A) — intensity/energy of the emotion.
3. Dominance–Submissiveness (D) — controlling/dominant nature of the emotion.

NOTE: VAD and PAD refer to the SAME model. Valence = Pleasure.

Circumplex Model (2D — VA space, Dominance omitted)

Quadrant Arousal Valence Example Emotions
I High Positive Excited, Delighted, Happy
II High Negative Tense, Angry, Frustrated
III Low Negative Depressed, Bored, Tired
IV Low Positive Content, Relaxed, Calm

Pros of Dimensional Model

• (+) Overcomes limitation of relating affective states — provides measurable distance between
them.
• (+) Recent attention for both regression methods and discretized studies.

8. Why the Dominance Dimension Matters

PROBLEM IN 2D VA SPACE — Exam Trap
Fear → Negative valence, Low/High arousal
Anger → Negative valence, Low/High arousal
PROBLEM: Cannot differentiate them in 2D!

SOLUTION — Add D (Dominance):

Fear → Negative valence, Low/High arousal, SUBMISSIVE dominance.
Anger → Negative valence, Low/High arousal, DOMINANT (control).
 (+) Emotions that overlap in VA space become distinguishable in VAD space.

9. Three Problems in Traditional Affective Computing ★ ALMOST CERTAIN

EXAM Q
Memory Aid: E–P–P
THREE KEY LIMITATIONS
1. EMERGENCY vs PROCESS EMOTIONS:
Emergency emotions = stimulated by intense, instantaneous stimuli.
Process emotions = stimulated by accumulation of CONTINUED WEAK stimuli over time.
Traditional AC only handles emergency; process emotions are neglected.

2. EMERGENCY EMOTIONS NEED SPEED, NOT PRECISION:

They are quick, low-precision responses.
Computational complexity of precision-oriented AC is too high to handle them in real time.

3. EMOTIONS ARE PERSONALIZED:

Different individuals excited by the SAME stimulus can have DIFFERENT emotions.

10. Don Norman's Three Levels of Emotional Design ★ HIGH YIELD

Memory Aid: V–B–R = Visceral (first impression), Behavioral (during), Reflective (after)

Level What It Is Affective Computing Application

Visceral Quick reactions & judgements of stimuli. Understand user feelings to improve
In charge of emotional reactions; FIRST IMPRESSION. E.g., first time you
informs behavioral level. bought an iPhone.
Behavioral Level of most behaviors. Controlled by Understand emotions DURING tasks
reflective level but influenced by (success and failure). E.g., keyboard
visceral (subconscious). typing on MacBook.
Reflective Cognitive, intellectual level. Watches Understand POST-USAGE emotional
over behavioral level; NO direct experience; emotional bond. E.g.,
connection to visceral. recalling a roller-coaster ride.

• Flow: Sensory input → Visceral ↔ Behavioral ↔ Reflective → Motor output.

• CRITICAL: Reflective has NO direct link to Visceral level.
11. The Emotional Brain — Lobes Overview
Structure Key Functions
Frontal Lobe Movement, reasoning, behavior, memory, decision making,
personality, planning, judgment, initiative, inhibition, mood.
Parietal Lobe Telling right from left, calculations, sensations, reading, writing.
Occipital Lobe Vision.
Temporal Lobe Language comprehension, behavior, memory, hearing, EMOTIONS.
Cerebellum Balance, coordination, fine muscle control.
Brain Stem Breathing, blood pressure, heartbeat, swallowing.
Pituitary Gland Hormones, growth, fertility.

12. Four Key Emotion-Processing Brain Regions ★ MOST EXAMINABLE

MEMORIZE IN ORDER — Memory Aid: PAAI
1. PREFRONTAL CORTEX (PFC) — emotional experience and its REGULATION.
2. AMYGDALA — stimulus SALIENCE and MOTIVATIONAL SIGNIFICANCE.
3. ANTERIOR CINGULATE CORTEX (ACC) — SELECTION of stimuli for further
processing.
4. INSULA — FEELINGS and CONSCIOUSNESS.

13. Prefrontal Cortex (PFC) — Detailed

• Most anterior (front) part of the brain.
• Major role in higher-order cognitive activity, judgement, planning.
• Associated with emotional experience, motivation, and regulation.

• Right PFC: Role: emotional vulnerability and affective disturbance. Activity in the LEFT
hemisphere may provide a neurobiological marker of resilience (Begley & Davidson, 2012).
• Left PFC: Associated with emotion of ANGER (Harmon-Jones et al., 2010). Better framed as
APPROACH vs WITHDRAWAL motivation — NOT positive vs negative valence.
CRITICAL INSIGHT — WHY ANGER ACTIVATES LEFT PFC
Anger has NEGATIVE valence, but activates the LEFT PFC.
Because LEFT = APPROACH motivation. Anger is approach-motivated (you want to
CONFRONT).
This is the famous EXCEPTION to the standard left=positive / right=negative rule.

14. The Amygdala — Detailed

• Almond-shaped cluster of nuclei in the ANTERIOR MEDIAL TEMPORAL LOBE.
• General role: processing stimulus salience, motivational significance, arousal.
• Central role in negative emotions: fear and anxiety (Murphy et al., 2003).
• Also activated by: fear, disgust, sadness, anger, happiness, humor, sexually explicit images,
social emotions.
 • Non-negative activation: sexual arousal (Hamann et al., 2004).
TAKEAWAY
The amygdala is NOT only a 'fear center' — it responds to ANY emotionally salient stimulus.

15. Anterior Cingulate Cortex (ACC) — Detailed

• Key site for visceral regulations that helps resolve which sensory input is selected for further
processing (Lindquist et al., 2012).
• Activity observed during classical fear conditioning and instructed fear-based tasks (Etkin et al.,
2011).
• ACC activity is POSITIVELY correlated with sympathetic nervous system activity (fight-or-flight
arousal).

16. Insula — Detailed

• Plays key role in experimental and expressive aspects of internally generated emotion.
• Involved in general awareness of bodily sensations, affective feeling, and CONSCIOUSNESS
(Craig, 2009).
• Anatomical parts: AI (Anterior Insula) and PI (Posterior Insula).

17. Emotion and Brain Asymmetry ★ HIGH YIELD

GENERAL RULE (Coan et al., 2003)
Increased LEFT-Frontal Activity → POSITIVE feelings + APPROACH motivation.
Increased RIGHT-Frontal Activity → NEGATIVE feelings + WITHDRAWAL motivation.

EXCEPTION! (Harmon-Jones et al., 1998)

ANGER = LEFT brain bias — even though anger is a NEGATIVE emotion.
Why? LEFT codes APPROACH, not just positive valence.
Anger is approach-motivated (you want to confront the target).
EXAM TIP: If asked for the exception to left/right positive/negative mapping → answer is
ANGER.

18. Specificity of Emotions — Barrett's Challenge (2012)

• Ekman & Friesen (1975): basic emotions are characterized by specific facial expressions.
• Barrett (2012) — counter-claim: a SINGLE SET of facial actions can become DIFFERENT
emotional expressions in different CONTEXTS.
• Particularly true for: Surprise, Fear, and Anger — they share facial features and are
distinguished mainly by context.
• IMPLICATION for affective computing: you CANNOT rely solely on facial expressions; context
and multi-modal signals matter.

19. Physiological Signatures of Each Emotion ★ HIGH-YIELD TABLE

19.1 Fear
 • Facial: Eyebrows RAISED AND DRAWN TOGETHER, wide open eyes, tense lowered eyelids,
STRETCHED lips.
• Brain: Frontoparietal brain activation (Tettamanti et al., 2012).
• ANS: Broad pattern of sympathetic activation (Harrison et al., 2013).
• HRV: REDUCED Heart Rate Variability.
• Skin conductance: MORE numerous SC responses than anger (Stemmler et al., 2001).
• EMG: LARGER corrugator (brow muscle) activity than anger.

19.2 Anger
• Facial: LOWERED eyebrows DRAWN TOGETHER, tense lowered eyelids, PRESSED lips.
• Brain: LEFT FRONTAL PFC activated (Harmon-Jones et al., 2010). Literature is contradictory
(Wacker et al., 2008).
• HRV: NO CHANGE (Rainville et al., 2011).
• Response type: may elicit either an anger-MIRRORING response or a RECIPROCATING FEAR
response — psychophysiological responses differ across people.

19.3 Disgust
• Facial: RAISED UPPER LIP, WRINKLED nose bridge, RAISED cheeks.
• Core-disgust stimuli (dirty toilets) → UNCHANGED or DECREASED skin conductance.
• Body-boundary violating stimuli (mutilation scenes) → INCREASED skin conductance.

19.4 Sadness ★ EXAM TRAP — Two Sub-Types

Signal CRYING Sadness NON-CRYING Sadness
Heart Rate INCREASED REDUCED
HRV No change REDUCED
Skin conductance INCREASED REDUCED
Respiration — INCREASED

• Facial: RAISED INNER eyebrows, LOWERED lip corners.

• Brain: increased blood flow in ventral regions (Mayberg et al., 1999).

19.5 Happiness
• Facial: TENSED lower eyelids, RAISED cheeks, RAISED lip corners (Duchenne smile = full
genuine smile).
• Brain: medial prefrontal and temporo-parietal cortices (Tettamanti et al., 2012). Left PFC plays
important role.
• HRV: DECREASED HRV (Kreibig, 2010). BUT amusement and joy → INCREASED HRV.
• Longevity (Abel & Kruger, 2010): No smiles → 72.9 yrs. Partial smiles → 75.0 yrs. Duchenne
smiles → 79.9 yrs.

Quick Summary Table — All Five Emotions

Emotion Key Facial Features Key Physiological Marker
Fear Brows raised & drawn together; wide ↓HRV; ↑SC; ↑corrugator EMG
eyes; stretched lips
Emotion Key Facial Features Key Physiological Marker
Anger Brows lowered & drawn together; pressed No HRV change; Left PFC activation
lips (exception!)
Disgust Raised upper lip; wrinkled nose; raised ↓/unchanged SC (core); ↑SC (body-
cheeks boundary violations)
Sadness Raised INNER brows; lowered lip corners Crying: ↑HR, ↑SC; Non-crying: ↓HR,
↓HRV, ↓SC, ↑resp
Happiness Tensed lower eyelids; raised cheeks & lip ↓HRV; Left PFC; Duchenne smile =
corners 79.9 yrs longevity
WEEK 12 — ETHICAL CONSIDERATIONS (DETAILED)
Source: Dr. Jainendra Shukla, IIIT Delhi | Computer Science and Engineering

1. Motivation — The Picard (1997) Warning

OPENING QUOTE — Picard (1997)
"The fictional message has been repeated in many forms and is serious: a computer that
can express
itself emotionally will someday act emotionally, and if it is capable of certain behaviors, then
the
consequences can be tragic."

2. Four Core Ethical Considerations ★ HIGH YIELD

2.1 Privacy
• Emotions, perhaps more so than thoughts, are ultimately PERSONAL and PRIVATE (Picard,
2003).
• Emotional data is MORE intimate than typical personal data.
• Questions: Who owns the data? Who can access it? Can it be used against you (insurance,
employment)?

2.2 Emotional Dependency

• Could having users routinely use these MORAL AGENTS create CODEPENDENCE?
• Risk: users rely on AI for emotional regulation instead of humans or their own coping skills.
• Long-term effect on social/emotional development is unknown.

2.3 Emotional Manipulation

• Is it ethical for computers to DETECT, RECOGNIZE, and then attempt to MODIFY certain
behaviors?
• Even if 'motivated by good will' (Picard and Klein, 2002, p. 16) — intent does not resolve the
ethical question.
• Raises questions of consent, autonomy, and who benefits.

2.4 Building Relationships

• At what point will a person begin to VALUE affective technology and its well-being OVER that of
another human being?
• Risk of humans forming parasocial attachments to machines at the expense of real human
relationships.

3. Privacy Invasiveness Framework (Politou, Alepis, & Patsakis, 2017) ★

HIGH YIELD
Sensors and data types categorized by privacy invasiveness level:
 LOW Invasiveness MEDIUM Invasiveness HIGH Invasiveness
Accelerometer Apps Microphone
Battery WiFi Camera
Gyroscope Bluetooth Internet traffic
Charger jack GPS
Audio jack Calls
Ringer mode SMS
Light sensor E-mails
Screen on/off
Environmental sensors
Screen touch

• LOW: Physical/motion sensors that reveal nothing inherently personal.

• MEDIUM: Connectivity signals that can reveal social patterns.
• HIGH: Communications and location data that directly reveal identity, location, and private
interactions.

4. Anticipatory Mobile Computing (Pejovic & Musolesi, 2015) ★ HIGH YIELD

A 4-stage pipeline for proactive emotion-aware systems:

Stage Description Challenge Example

1. Sensing Collect smartphone Adaptive sensing; energy- Monitor user's voice as day
sensor data. efficient sampling; data progresses; regulate sampling
storage. rate by resource levels.
2. Inferring Extract features from Features & classifier Process user's voice: create
Context raw data. ML selection; scalable ML; Gaussian Mixture Model to
connects features balance between phone & identify voice and measure
with higher-level cloud processing. stress level.
concepts.
3. Predicting Build models of future Short- vs long-term Use personalized history of
Context events and predicted predictions for different behaviour to predict a health
user behaviour. purposes; incorporate data hazard (high stress due to
from multiple users. busy workday).
4. Intelligent Construct decision Learn from mistakes (RL); Reschedule user's meetings
Actioning framework based on curiosity vs accuracy — and locations to reduce the
past, current, and value of decision depends future level of stress.
future events. on how proactive it is.

5. Existing Approaches: Insufficiency

WHY CURRENT PRIVACY SOLUTIONS ARE INSUFFICIENT
• Existing approaches (cryptography, privacy-preserving data mining, anonymisation) are
 often insufficient in BOTH technological and operational terms (A. Kapadia, 2009).

• There is NO such thing as foolproof anonymisation — almost ALL information can be

defined
as 'personal' when combined with enough other relevant data (L. Sweeney et al., 2013).

Implication: technical solutions alone cannot protect emotional privacy.

6. Open Issues in Affective Computing Ethics ★ HIGH YIELD

Set 1 — Design & Deployment Questions
14. What sort of machines should be given affective capabilities?
15. What are the CONSEQUENCES of giving machines affective capabilities?
16. What should be the PURPOSE of developing an emotionally intelligent machine?
17. What kind of MORAL CONSTRAINTS should be considered while developing such a machine?
18. Should the machine simulate behaviour in SOCIALLY ACCEPTABLE settings or based on a
CRITICAL MORAL STANDPOINT?

Set 2 — Deeper Philosophical & Technical Questions

19. Is there DECEPTION inherent in affective computing? Is such deception ethical?
20. Does the machine enable human beings to function in the manner they CHOOSE to represent
themselves, or does it LOOK THROUGH the person? (i.e., does it expose concealment inherent
to self-representation?)
21. Should DESTRUCTIVE MACHINES be given emotional capabilities?
22. What ethical aspects should a DESIGNER include in developing emotionally intelligent
machines?

7. Targeted Advertising — Discussion ★ LIKELY DISCUSSED IN EXAM

KEY DISCUSSION QUESTIONS
• Should personal EMOTIONS be used for marketing? (Based on the emotional states of the
person)
• What about VULNERABLE POPULATIONS? (e.g., people with depression, anxiety,
children)
• How will targeted advertising CHANGE if a system becomes emotionally intelligent?
• How can we deal with it? (Regulation? Consent mechanisms? Opt-out?)

Why This Is Ethically Problematic

• Emotionally targeted ads exploit users at their most vulnerable moments (e.g., showing comfort
food ads to someone detected as sad).
• Children and people with mental illness are especially vulnerable to emotion-based
manipulation.
• Current ad regulation frameworks were not designed for emotion-aware systems.
MASTER QUICK REVISION — ALL WEEKS
WEEK 1 — ONE-PAGE CHEAT SHEET
DEFINITION: Sense, Recognize, Respond to, Influence emotions (Picard 1997).
7 AI CAPABILITIES: Speech | Facial recognition | Lip reading | Systems monitoring |
Regulation | Game play | Social skills.
6 MODALITIES: Facial | Posture/Gesture | Hand tension | Vocal | Textual | Physiological
(EDA, EEG).
3 APP TYPES (D-E-P): Detection | Expression | Perception.
HEALTHCARE: SymTrend, Autism Track (AS/HFA); Visharanti (stress, PPG+GSR);
StartleMart, Virtual Iraq (PTSD).
EDUCATION: EngageMe (Darnell 2014 — GSR) | Subtle Stone (Balaam 2009 — squeeze
ball).
OTHER: MACH (interview coach) | KISMET (sociable robot — Expression type).
UX = How user feels | Usability = Ease of use. Advanced UX = affective computing +
neuromarketing.
PICTIVE (Muller 1991) = participatory design with paper/sticky notes + video.
3 ETHICS (M-P-D): Manipulation | Privacy (Picard 2003) | Dependency.

WEEK 2 — ONE-PAGE CHEAT SHEET

DEFINITION (Kleinginna & Kleinginna 1981): 4 parts — (a) Affective experience, (b)
Cognition/appraisal, (c) Physiology, (d) Behavior. Mediated by neural/hormonal systems.
JAMES (1894): We are afraid BECAUSE we run — body first, then emotion.
FACIAL FEEDBACK: Strack (1988) + pencil experiment (Levenson 1990). Laughter yoga =
physical → emotional.
PERCEIVED ≠ INDUCED (Gabrielsson 2002).
6 BASIC EMOTIONS (Ekman & Friesen 1971): Happiness, Anger, Disgust, Sadness,
Fear/Anxiety, Surprise. (+Contempt later).
VAD/PAD (Russell & Mehrabian 1977): Valence × Arousal × Dominance. Circumplex = drop
D → 2D.
D dimension resolves Fear vs Anger overlap: Fear = Submissive, Anger = Dominant.
3 PROBLEMS (E-P-P): Emergency vs Process emotions | Precision too slow for emergency
| Emotions are personalized.
NORMAN 3 LEVELS (V-B-R): Visceral (first impression) | Behavioral (during) | Reflective
(post-use, bond). Reflective has NO direct link to Visceral.
4 BRAIN REGIONS (PAAI): PFC (experience+regulation) | Amygdala (salience) | ACC
(selection) | Insula (feelings+consciousness).
ASYMMETRY: LEFT = positive/approach | RIGHT = negative/withdrawal. EXCEPTION:
ANGER → LEFT (approach-motivated, not valence-based).
BARRETT (2012): Facial expressions are context-dependent. Can't rely on face alone →
need multimodal.
PHYSIOLOGY: Fear=↓HRV,↑SC | Anger=Left PFC,no HRV change | Disgust=↑/↓SC by
stimulus | Sadness=Crying vs Non-Crying differ | Happiness=↓HRV, Duchenne=79.9yrs.

WEEK 12 — ONE-PAGE CHEAT SHEET

4 ETHICAL ISSUES: Privacy | Emotional Dependency | Emotional Manipulation | Building
Relationships.
PRIVACY INVASIVENESS: Low (accelerometer, gyro, screen) | Medium (apps, WiFi,
Bluetooth) | High (mic, camera, GPS, calls, SMS, email).
ANTICIPATORY MOBILE COMPUTING (Pejovic & Musolesi 2015): 4 stages — Sensing →
Inferring Context → Predicting Context → Intelligent Actioning.
EXISTING APPROACHES INSUFFICIENT: Cryptography, anonymisation are not foolproof
(Kapadia 2009; Sweeney 2013).
OPEN ISSUES: Deception in AC? Machine seeing through self-representation? Destructive
machines + emotions? Designer ethics?
TARGETED ADVERTISING: Emotion-based targeting of vulnerable populations is ethically
problematic.

KEY REFERENCES FOR ALL WEEKS

Author(s) Year Key Contribution
Picard, R.W. 1997 Affective Computing — founding book; definition of AC;
affect sensing.
Picard, R.W. 2003 Affective Computing challenges; Privacy — emotions are
personal and private.
Picard & Klein 2002 Extended definition to include interaction; 'motivated by
good will'.
Kleinginna & Kleinginna 1981 Formal 4-component definition of emotion.
James 1894 Bodily response first theory of emotion.
Darwin 1872 Evolutionary view of emotions as universal, adaptive.
Ekman & Friesen 1971/1975 Six basic emotions; universal facial expressions; FAPCs.
Russell & Mehrabian 1977 VAD/PAD 3-factor theory of emotion dimensions.
Gabrielsson 2002 Perceived vs induced emotion.
Barrett 2012 Context-dependence of facial expressions — specificity
challenge.
Coan et al. 2003 Frontal EEG asymmetry: LEFT=positive/approach,
RIGHT=negative/withdrawal.
Harmon-Jones et al. 1998/2010 Anger = Left PFC (exception to valence rule).
Strack et al. 1988 Facial feedback hypothesis.
Levenson et al. 1990 Pencil experiment; facial feedback evidence.
Costafreda et al. 2008 Amygdala meta-analysis (385 PET+fMRI studies).
Author(s) Year Key Contribution
Craig 2009 Insula role in bodily awareness, feeling, consciousness.
Abel & Kruger 2010 Smile intensity predicts longevity (Duchenne = 79.9 yrs).
Kreibig 2010 HRV and emotion-specific autonomic patterns.
Don Norman — Emotional Design: Why We Love (or Hate) Everyday
Things — 3 levels.
Pejovic & Musolesi 2015 Anticipatory mobile computing — 4-stage pipeline.
Politou, Alepis & Patsakis 2017 Privacy invasiveness levels of mobile sensors.
Kapadia 2009 Insufficiency of existing privacy approaches (opportunistic
sensing).
L. Sweeney et al. 2013 No foolproof anonymisation — data combination =
personal.
Muller, M.J. 1991 PICTIVE — participatory design technique.
Darnell, S.S. 2014 EngageMe — classroom affect sensing.
Balaam et al. 2009 Subtle Stone — affective communication in classroom.
Holmgard et al. 2013 StartleMart — stress detection for PTSD.