NPTEL
Multimodal Emotion Recognition
Dr. Jainendra Shukla
jainendra@[Link]
B-403, R & D Block
Computer Science and Engineering
Motivation
● Multiple modalities -> richer representation -> more accurate
inference and expression.
NPTEL
● Each modality is expected to provide unique information,
○ Redundancy.
● Challenges (D’Melo and Kory, 2012):
○ Classifiers and fusion methods that better capture the relationships
between different modalities.
○ Affective corpora that contains adequate samples of synchronized
expressions.
○ Improvements are still often relatively modest.
Multimodal Affect Recognition
● Underlying relationships and correlation between feature sets in
different modalities and affect dimensions
NPTEL
○ Feature Selection
● How different affective expressions influence each other.
○ Uniqueness
● How much information each of them provides about the expressed
affect.
○ Modality Selection
Data Collection
● The need for corpora and databases that contain spontaneous and
subtle.
NPTEL
● Databases containing contextual descriptions synchronized with
other modalities.
● Affective stimuli should be labeled simultaneously for all the
modalities available to the coder (e.g. audio + video) rather than
considering the single modalities (e.g. audio/video) separately.
Feature Extraction
● Variable sampling frequency
○ Video processing: 25 FPS | GSR: 16 Hz | EEG: 128 Hz etc.
NPTEL
● Synchronization is required
○ To unite the feature information or
○ To come to a decision at a certain moment in time
● Feature Selection: Optimize the feature space individually per
information stream followed by a combined feature selection
(Schuller et al., 2008).
○ I. Often highly correlated information should be reduced individually per
modality
○ II. Secondary optimization process can lead to further improvements
reducing cross-modal redundancy
Stream Fusion
NPTEL
Early Fusion
● Concatenating the the features from
multiple cues into one feature vector.
NPTEL
● Becomes more challenging as the number of
features increases and when the features are
of very different nature.
● Synchronization is of utmost importance.
Late Fusion
● Feature and time dependency are abstracted.
NPTEL
● Each classifier process its own data stream
and the multiple sets of outputs are
combined at a later stage to produce the final
results.
● Soft level: a measure of confidence is
associated with the decision
● Hard level: the combining mechanism
operates on single hypothesis decisions.
Slow Fusion
● Assumption of conditional independence
between modalities and cues in decision level
NPTEL
fusion can result in loss of information.
● Exploit the correlations between the
modalities while relaxing the requirement of
the synchronization.
SEMAINE
● Sensitive Artificial Listener
(SAL): A multimodal dialogue
NPTEL
system with the social
interaction skills needed for a
sustained conversation with a
human user.
● Aims to engage the user in a
dialog and create an emotional
workout by paying attention to
the user’s non-verbal
expressions and reacting
accordingly.
○ E.g. Nodding and smiling.
Multimodal Affect Recognition
● Real-time Multimodal Data
○ Audio-visual data obtained from cameras and microphones
Data segmentation and analysis
NPTEL
●
○ Frame/Millisecond/Second/Minute level
○ Trade-off
■ Real-time prediction requires smaller window
■ Reliable prediction accuracy requires longer-term monitoring.
○ Appropriate unit of analysis is context dependent
● Video was recorded at 49.979 fps at a spatial resolution of 780 x
580 pixels and 8 bits per sample, while audio was recorded at 48
kHz with 24 bits per sample.
● Audio-video signals were synchronised with an accuracy of 25 µs.
SEMAINE Database
● Spontaneous data capturing the audiovisual interaction between a
human and an operator undertaking the role of an avatar with
NPTEL
four personalities:
○ Prudence, who is even tempered and sensible;
○ Poppy, who is happy and outgoing;
○ Spike, who is angry and confrontational; and
○ Obadiah, who is sad and depressive.
● All interactions were annotated by 2 to 8 raters in the four
dimensions in continuous time and continuous value.
● Dimensions are Arousal, Expectation, Power, and Valence.
○ Four dimensions account for most of the distinctions between everyday
emotion categories (Fontaine, J., et. al., 2007).
● {vai, vei, vpi, vvi} for every rater i and dimension a/e/p/v.
Audio-Visual Emotion Challenge
Uni and multi-modal challenge based on acoustic, linguistic and
video cues
NPTEL
Data and Annotations
● To attain binary labels,
○ Average value of each dimension over all raters was computed, resulting
in a set of continuous time, real valued variables {va, ve, vp, vv}
NPTEL
○ Mean of these average ratings over all interactions in the dataset was
computed, resulting in the scalar values: {𝜇ai, 𝜇ei, 𝜇pi, 𝜇vi}
○ The binary labels {ya, ye, yp, yv} ∈ {±1} are then found by
thresholding vjt > 𝜇j for each dimension j at every frame t.
● Video stream was chunked in two ways:
○ Frame level: Per frame for the video only taks
○ Word level: Per word for the audio and audio-visual tasks
Feature-level fusion
NPTEL
Decision-level fusion
NPTEL
Audio/visual emotion challenge
(Wöllmer, M., 2013)
NPTEL
Results: No Feature Selection
NPTEL
Discussion: No FS
● For arousal, the best WA of 68.5 % is obtained for acoustic
features only, showing that audio is the most important modality
NPTEL
for assessing arousal (M. Wollmer, et. al., 2010).
● The classification of expectation seems to benefit from including
visual information as the best WA (67.6 %).
● Power is best classified via speech-based features. However, for
unidirectional modeling WA significantly increases from 64.7 %
to 66.2 % when using linguistics in addition to audio features
● For valence, the inclusion of video information helps, leading to a
WA of 69.8 %.
Results: CFS Feature Selection
NPTEL
Discussion: FS
● For most settings, CFS does not significantly improve the average
weighted accuracy.
NPTEL
● However, for recognition based on video only, CFS leads to a
remarkable performance gain, increasing the mean WA from 60.4
% to 65.8 % for unidirectional LSTM networks.
Audio/visual emotion challenge
NPTEL
Discussion: AVEC Challenge
● Audio features lead to the best result for arousal classification.
● For classification of expectation using facial movement features,
NPTEL
the obtained WA of 68.6 % is higher than what is reported for
other techniques.
● For power, audiovisual classification with Latent-Dynamic
Conditional Random Fields as proposed in (G. Ramirez, et. al.,
2011) outperformed.
● For valence, the audio features lead to the highest accuracy (70.2
%) (A. Sayedelahl, et. al., 2011).
References
1. D'Mello, S., & Kory, J. (2012, October). Consistent but modest: a
meta-analysis on unimodal and multimodal affect detection accuracies
NPTEL
from 30 studies. In Proceedings of the 14th ACM international
conference on Multimodal interaction (pp. 31-38).
2. Schuller, B., Wimmer, M., Arsic, D., Moosmayr, T., & Rigoll, G. (2008).
Detection of security related affect and behaviour in passenger
transport. In Ninth Annual Conference of the International Speech
Communication Association.
3. [Link]
4. Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G. (2013).
LSTM-Modeling of continuous emotions in an audiovisual affect
recognition framework. Image and Vision Computing, 31(2), 153-163.
References
1. Fontaine, J., K.R., S., Roesch, E., Ellsworth, P.: The world of emotions is
not two-dimensional. Psychological science 18(2), 1050 – 1057 (2007)
2. M. Wollmer, A. Metallinou, F. Eyben, B. Schuller, S. Narayanan,
NPTEL
Context-sensitive multimodal emotion recognition from speech and
facial expression using bidirectional LSTM modeling, in: Proc. of
Interspeech, Makuhari, Japan, 2010, pp. 2362–2365.
3. G. Ramirez, T. Baltrusaitis, L. P. Morency, Modeling latent
discriminative dynamic of multi-dimensional affective signals, in: Proc.
of First International Audio/Visual Emotion Challenge and Workshop
(AVEC 2011) held in conjunction with ACII, Memphis, Tennessee, USA,
2011, pp. 396–406.
4. A. Sayedelahl, P. Fewzee, M. Kamel, F. Karray, Audio-based emotion
recognition from natural conversations based on co-occurrence matrix
and frequency domain energy distribution features, in: Proc. of (AVEC
2011) held in conjunction with ACII, Memphis, Tennessee, USA, 2011