NPTEL
Experimental Methodology
Dr. Jainendra Shukla
jainendra@[Link]
Computer Science and Engineering| Human-Centered Design
Agenda
1. Affect Elicitation
2. Experimental Methodology
NPTEL
1. Institutional Review Board (IRB)
3. Research and Development Tools
1. Demo: PsychoPy
NPTEL
Affect Elicitation
Dataset of emotional expressions
1. Acted or posed Expressions obtained by asking individuals,
NPTEL
often actors, to portray emotions
i. Easy to collect
ii. Ecological validity of data is a concern
2. Naturalistic display of emotions
i. Ecologically valid
ii. Notoriously difficult to collect
3. Induced Expressions, in which emotional responses are
elicited via some stimulus;
i. Middle road
Emotion Elicitation
1. Passive or perception based
NPTEL
• Individuals observe stimuli, such as film clips, images or music that
are designed to evoke particular feelings or methods
2. Active or Expression based
• Individuals are instructed to perform particular behaviours that
might naturally evoke different emotions, such as posing facial
muscles, adopting body postures or interacting with other people.
Passive Methods: Images
1. The presentation method is standardized such that all the
individuals have the same viewing experience;
NPTEL
• E.g. the images could be presented for 10 seconds on a computer screen
that is at a fixed distance; with a constant screen resolution, screen
brightness, and image size (Monkaresi et. al, 2012).
2. The images can be selected on the basis of which emotion should be
elicited from a database of standard images.
• International Affective Picture System (IAPS)
• Genova Affective Picture Database (GAPED)
Image Databases
NPTEL
Source - Emotional Design: International affective picture system (IAPS)
Passive Methods: Images
a. Advantages:
i. Noninvasive
NPTEL
ii. Easily accessible
iii. Relatively simple setup
b. Disadvantages:
i. Strength of emotion is lower
ii. Emotional reactions are short and transient
iii. Lack of Personalization:
1. Individuals with cocaine addiction react differently to pleasant, unpleasant
and neutral IAPS images (Asensio et. al, 2010)
iv. Etc.
c. Utilizing images may be useful for reactive modalities such as facial
expressions, physiological signals but not for productive modalities
(text or gestures etc.)
Passive Methods: Film Clips
1. For comparison purposes, a neutral baseline film is generally shown
prior to the presentation of each emotional clip.
NPTEL
• neutral clip [12 seconds] -> emotional clip [1-2 minute] -> self-assessment
phase
2. Short clips or long clips?
3. Physical situation should be standardized (Rottenberg et. al, 2007):
• E.g. 20 inch monitor about 5 feet from the participant
4. For each target emotion, 1 to 2 short clips as homogenous as possible
5. The flims can be selected online from various sources, e.g:
• [Link]
Passive Methods: Sample
NPTEL
Source - [Link]
Passive Methods: Film Clips
1. Advantages:
NPTEL
a. They capture the attention well
b. Can elicit higher intensity emotions and more complex emotional states
(Rottenberg et. al, 2007)
c. Also useful in studying the emotion latency, rise time , duration and offset
2. Disadvantages:
a. Ecological validity is in question
b. Individuals may have seen the films from which the clips are taken
c. Not ideal for obtaining data from productive modalities.
• Gestures or verbal behaviour
Active Methods:Behavioral Manipulation
1. Individuals are instructed to adopt particular muscular configurations
NPTEL
or behavioral patterns that have been associated with emotional
expressions, such as contracting or relaxing facial muscles or
exaggerating natural emotional expressions (Ekman et. al, 1993, 2007).
2. Directed Facial Action Task (Ekman et. al, 2007)
a. Method for eliciting emotion through the manipulation of facial expressions
3. Researchers often ask participants to think or write about an instance(s)
from their past when they experienced an emotion such as anger (e.g.,
Siedlecka et al., 2015).
Active Methods: Facial Action Task
NPTEL
Source: Dhruv Verma, Sejal Bhalla, Dhruv Sahnan, Jainendra Shukla, and Aman Parnami. 2021. ExpressEar: Sensing Fine-Grained
Facial Expressions with Earables. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 3, Article 129 (Sept 2021), 28 pages.
Active Methods:Behavioral Manipulation
1. Disadvantages:
NPTEL
a. Ecological validity: Can one really say that the emotions elicited
through behavioural manipulation are more “pure”?
b. One needs to know the physical behaviour associated with target
emotions.
i. Ekman's work does not include more complex emotions such as frustration,
confusion, engagement etc.
ii. Limits the range of target emotions
c. Poses or actions required can be difficult or complicated.
2. Advantages:
a. Useful for collecting reactive expressions
b. The effects are strong if physical behaviour associated with target
emotions are known precisely.
Active Methods: Social Interaction
NPTEL
• Advantages:
a. Elicited emotions
are more realistic
and natural.
b. Easy to elicit
emotions that are
otherwise hard to
elicit, such as anger,
guilt.
Source:
[Link]
[Link]
NPTEL
Experimental Methodology
IRB for Human Research
● Research Involving Human Subjects: Institutional Review Board (IRB)
● IRB is a review committee established to help protect the rights and
NPTEL
welfare of human research subjects.
● Documents for IRB:
● A draft or an abstract of the proposal
● A clear description of the research methodology, or the experimental design.
● A copy of the "informed" consent form(s).
● A description of how confidentiality/anonymity will be protected.
● A description of the risks and benefits to the subjects.
● A copy of any recruitment document (advertisement flyers/invitation
letters/invitation emails).
● Data collection instruments the survey, or sensors, description of the
experimental research method, or the list of questions or interview protocols.
Criteria for IRB Approval
1. Risks to subjects are minimized or
reasonable.
NPTEL
2. Selection of subjects is equitable.
3. Informed consent is sought.
4. Plans for monitoring the data
collected.
5. Adequate provisions to protect the
privacy of subjects and to
maintain the confidentiality of
data.
6. Additional safeguards to protect
vulnerable population.
Source: [Link]
Experimental Design - I
NPTEL
Experimental Design - II
NPTEL
NPTEL
Research and Development
Tools
Category of Tools
● Data Collection
NPTEL
● Data Annotation
● Signal Analysis
● Affect Classification
● Affect Expression
Data Collection
● PsychoPy
● OpenSesame
NPTEL
● Experiment Wizard
● SuperLab
● E-Prime
● Presentation
● DMDX
● Paradigm
● In-house
Psychopy
NPTEL
Source: [Link]
Labelling and Data Annotation
● Genova Emo Wheel ● Video:
● Self-Assessment Mankins (SAM) ○ ELAN
NPTEL
● Feeltrace ○ Cowlog
● Tracetools ○ ANVIL
○ Gtrace
● Audio
○ ChronoViz
○ ELAN
○ Wavesurfer ● Text
○ Praat ○ Whissell Dictionary of Affect
○ Speechalyzer in Language
● In-house
SAM Mankins
NPTEL
Source: Morris, J.D. (1995). Observations: SAM: The Self-Assessment Manikin An Efficient Cross-Cultural Measurement Of
Emotional Response 1. Journal of Advertising Research.
Signal Processing and Analysis
● Audio:
○ Praat
NPTEL
○ OpenEAR
○ OpenSmile
○ Wavesurfer
● Physiological Signals
○ EEGLab, PyEEG
○ ELAN
○ AuBT
● Image/Video
○ OpenCV, OpenPose
○ Kinect SDK
○ MediaPipe
● In-house
MediaPipe
NPTEL
Source: [Link]
Data Mining
• WEKA
• AutoML
NPTEL
• SPSS
• PRTools
• MATLAB Arsenal
• R
• LibSVM, SVMlight
• HTK
• SAS
• RapidMiner
• SciPy
WEKA
NPTEL
Source: [Link]
Affect Expression
● MARY
● GRETA
NPTEL
● Festival
● VHML
● SmartBody
● FaceFX
● Xface
● Horde3D
● Haptek
● ROS
● ICT Virtual Human Toolkit
● SOAR
● ACT-R
ICT Virtual Human Toolkit
NPTEL
Source: [Link]
References
● [Link]
NPTEL
-methods%[Link]
● Research Methods in Psychology
● Review of Human Studies Methods in HRI and
Recommendations
● Chapter 27, OHAC
● Development and validation of Image Stimuli for Emotion
Elicitation (ISEE): A novel affective pictorial system with
test-retest repeatability
● It's the Effect Size, Stupid: What effect size is and why it is
important