Case Study: Emotional Virtual Personal
Assistant
Dr. Jainendra Shukla
jainendra@[Link]
Computer Science and Engineering| Human-Centered Design
1
Agenda
1. Motivation and Challenges
2. Methods and Techniques
3. Ethical Concerns
4. Interaction with Dr. Aniket Bera (Purdue University, USA)
2
Motivation and Challenges
3
Requirement Specifications
4
Source - [Link]
Use Cases
● Mental health and therapy: An Emotional Virtual Personal Assistant
could be used to help people with mental health disorders or emotional
struggles.
● Customer service and support: An Emotional Virtual Personal
Assistant could be used in customer service and support industries to
provide a more personalized and empathetic experience to customers.
● Education: An Emotional Virtual Personal Assistant could be used in
education to help students learn more effectively.
● Entertainment: An Emotional Virtual Personal Assistant could be used
in entertainment to create more engaging and interactive experiences.
● Workplace productivity: An Emotional Virtual Personal Assistant could
be used in the workplace to help employees manage stress, improve
communication, and increase productivity. 5
● What else???
Challenges
● Accuracy: Variability in emotional expression across individuals, cultures,
and contexts.
● Real-time processing: Must be able to operate in real-time or near real-
time to provide a seamless user experience.
● Multi-language support: Supporting multiple languages and cultural
contexts can be challenging.
● Bias and Discrimination: Emotion recognition technology may reflect the
biases of the individuals who create and train the models, potentially leading
to unfair treatment of certain groups.
● Privacy Concerns: Collecting and analyzing speech data for emotion
recognition raises privacy concerns, particularly if the data is collected without
the user's knowledge or consent.
● What else???
6
Requirement Specifications
1. The system should be able to recognize emotions in speech and
distinguish at least the basic emotions of happy, sad, angry, and neutral.
2. The system should be able to process natural language input in voice
format.
3. The system should be able to understand user intents and generate
appropriate responses based on the user's input and recognized
emotions.
4. The speech emotion recognition and virtual assistant components
should be able to operate in real-time or near real-time.
5. The system should protect user data, including speech data and
personal information.
6. The system should support multiple languages and be able to operate in
7
different cultural contexts.
Method
8
Skelton Code
import speech_recognition as sr
# Initialize the speech recognition and language identification objects
r = [Link]()
language_identifier = ...
# Initialize the emotional virtual personal assistant components
virtual_assistant = ...
emotion_recognition = ...
# Initialize the context object
context = ...
# Start the conversation loop
while True:
with [Link]() as source: # Prompt the user to provide voice input
audio = [Link](source) 9
Skelton Code
# Determine the language of the user's input
language = language_identifier.identify(audio)
# Recognize the user's emotions from the speech input
emotion = emotion_recognition.recognize(audio)
# Process the user's natural language input to understand their intent and context
input_text = r.recognize_google(audio, language=language)
intent = virtual_assistant.process_input(input_text, emotion, context)
# Generate an appropriate response based on the user's input, recognized emotion, and context
response_text = virtual_assistant.generate_response(intent, emotion, context)
# Output the response in voice format
response_audio = [Link](response_text, language=language)
response_audio.play()
# Update the context object based on the current interaction
[Link](intent, emotion) 10
End to end life cycle
11
Source - [Link]
Data
● A dataset of audio recordings and corresponding emotion labels, such as the
Berlin Database of Emotional Speech (EmoDB) or the Ryerson Audio-Visual
Database of Emotional Speech and Song (RAVDESS)
● Text data labeled with emotion categories, such as the Affective Text dataset
or the EmotionLines dataset.
● Example of the training data:
● Audio signal: "I am so happy to be here!"
Emotion label: happy
● Audio signal: "I'm feeling really sad today"
Emotion label: sad
● Audio signal: "I'm so angry with you right now"
Emotion label: angry
● Audio signal: "I don't really care one way or the other"
12
Emotion label: neutral
Skelton Code
13
Explore: Features
● Relevant features for emotion recognition, such as:
● Mel Frequency Cepstral Coefficients (MFCCs)
● Prosody features (pitch, loudness, speech rate, etc.)
● Spectral features (spectral centroid, spectral flux, etc.)
● Time-domain features (zero-crossing rate, energy, etc.)
● Feature selection can be used to reduce the dimensionality of the
feature space and improve the performance of the emotion recognition
model:
● Principal Component Analysis (PCA)
● Linear Discriminant Analysis (LDA)
● Mutual Information-based feature selection
● Recursive Feature Elimination (RFE)
● It's important to balance the number of features used with the
complexity of the model, to avoid overfitting. 14
Skeleton Code
15
Model and Register
● Define the machine learning model architecture.
● Compile the model by selecting the loss function, optimizer, and metrics
for evaluation.
● Train the model on the training set using the fit() function.
● Evaluate the model on the validation set using the evaluate() function.
● Tweak the model architecture, hyperparameters, or dataset if needed
and repeat steps 4 and 5 until satisfactory performance is achieved.
● Test the final model on the testing set to get an unbiased estimate of its
performance.
● Save the trained model and its weights using the save() function.
● Register the saved model with a model registry, such as TensorFlow
Serving or MLflow, for deployment.
16
Skeleton Code
17
Deployment
● Determine that the Raspberry Pi has sufficient processing power,
memory, and storage to run the model.
● Choose the TensorFlow Lite framework for deployment, which is
compatible with the Raspberry Pi and TensorFlow, the machine
learning library used to build the model.
● Optimize the model by converting it to a TensorFlow Lite format and
quantizing it to reduce its size and computational requirements.
● Write code in Python to load the optimized model onto the Raspberry Pi
and interface with its microphone and speaker.
● Test the deployed model on the Raspberry Pi by running it with
different audio inputs and comparing the predicted emotions with the
actual emotions.
18
Skelton Code
19
Ethical Concerns
20
Ethical Concerns - I
● Privacy concerns: Emotional virtual agents may collect personal data
which could be used for commercial or other purposes without the
user's consent.
● Emotional manipulation: Emotional virtual agents could be
designed to manipulate users' emotions, behaviors, or beliefs without
their knowledge or consent.
● Bias and discrimination: There is a risk that the emotional
responses of virtual agents may be influenced by bias or prejudice,
resulting in discrimination against certain groups of people.
● Unintended consequences: Emotional virtual agents may have
unintended consequences on users' emotional well-being, such as
addiction or dependence on the virtual agent.
21
Ethical Concerns - II
● Liability: Emotional virtual agents may cause harm or have
unintended consequences, leading to potential legal liability for
developers or manufacturers.
● Deception: Emotional virtual agents may be designed to appear as if
they have emotions, but in reality, they are just algorithms. This could
be considered deceptive if users believe that the virtual agent is
genuinely experiencing emotions.
● Social isolation: Emotional virtual agents could lead to social
isolation, as people may rely on them for emotional support rather than
engaging in human interactions.
22
Sample Scenarios
1. A company creates an EVA that collects voice patterns to provide
tailored emotional support to users.
a. However, the company does not obtain users' consent to collect this personal
data, raising privacy concerns.
2. A virtual reality therapy platform creates an EVA that helps patients
with post-traumatic stress disorder (PTSD) relive traumatic experiences
in a controlled environment.
a. However, the use of the virtual assistant may trigger severe emotional
responses in some patients, worsening their symptoms.
3. A company creates an EVA for customer service that uses human-like
speech and empathy to provide personalized support.
a. However, the virtual assistant is programmed to promote certain products or
services over others, raising concerns about the fairness of the
recommendations. 23
Interaction with Dr. Aniket
Bera on
24