Real-Time Face Emotion Detection for Happiness

Index Analysis
Thomson Thomas Ben Baby Immanuel Biju Aldrin Emmanuel Dheeraj N
Dept. AI & DS Dept. AI & DS Dept. AI & DS Dept. AI & DS Dept. AI & DS7410
SJCET, Palai SJCET, Palai SJCET, Palai SJCET, Palai SJCET, Palai
Kottayam, India Kottayam, India Kottayam, India Kottayam, India Kottayam, India
thomsont711@[Link] benbaby53@[Link] immanuelbiju77@[Link] aldrinps597@[Link] dhraj.n@[Link]

Abstract—Facial expression recognition (FER) plays a crucial Our research significantly contributes to the domain of
role in understanding human emotions and is increasingly im- real-time emotion recognition by demonstrating the feasibility
portant in various applications, especially amidst the surge in and efficacy of CNN-based models, augmented by Siamese
visual data during the Covid-19 pandemic. This project aims
to develop innovative AI-based frameworks for real-time FER networks and one-shot learning, in capturing nuanced emo-
in classroom settings, addressing age-specific variations in facial tional nuances and computing the happiness index. This work
expressions and contributing to the analysis of happiness indices fosters opportunities for enriching user experiences, refining
and emotional well-being in educational environments. Drawing mental health monitoring tools, and propelling advancements
insights from a comprehensive literature survey, including studies in emotion-aware systems for human-computer interaction
on feature extraction techniques, multimodal fusion, and model
adaptation, this project endeavors to advance FER through technologies.
deep learning approaches and innovative methodologies. By
integrating facial expression analysis with other modalities such II. LITERATURE SURVEY
as EEG signals and leveraging state-of-the-art deep learning
architectures, the project seeks to enhance emotion recognition The literature survey encompasses a range of studies fo-
accuracy and robustness, ultimately fostering supportive learning cused on advancing facial emotion recognition (FER) through
environments and contributing to the broader FER community.
innovative approaches and techniques. Firstly, Dalvi et al. un-
derscore the need for an AI-based FER framework, particularly
I. I NTRODUCTION highlighting the surge in visual data during the Covid-19 pan-
In today’s rapidly evolving society, the demand for real-time demic and the importance of addressing age-specific variations
face emotion recognition has surged, particularly in applica- in facial expressions. Their survey provides a comprehensive
tions concerning user experience, mental health monitoring, review of methodologies, guiding future research in the FER
and human-computer interaction. This paper introduces a pio- community.
neering approach to real-time face emotion recognition within Y. Zhang, X. Liu, and M. Shah’s article in IEEE Trans-
an environment, employing Convolutional Neural Networks actions on Affective Computing (2022) delves into recent
(CNNs), Siamese networks, and one-shot learning models advancements in FER, particularly spotlighting deep learning
to assess the happiness index and identify emotions such techniques, ensemble methods, and multimodal fusion. It
as happiness, sadness, anger, and neutrality. Our innovative extensively discusses challenges such as data imbalance and
system harnesses deep learning algorithms to analyze facial ex- domain adaptation, while also identifying emerging trends and
pressions, extracting pertinent features for precise and prompt future research avenues.
emotion recognition. Through the utilization of CNN models N. Adel and S. Abdelazim’s work in IEEE Access (2021)
trained on a diverse dataset of facial images, our solution provides a comprehensive exploration of deep learning models
achieves remarkable accuracy in emotion classification, fur- for FER, spanning Convolutional Neural Networks (CNNs),
nishing real-time feedback on individuals’ emotional states Recurrent Neural Networks (RNNs), and hybrid architectures.
within the environment. It delves into dataset challenges, model evaluation metrics,
This paper elaborates on our methodology, encompassing and applications across domains like healthcare and human-
data collection procedures, preprocessing techniques, CNN computer interaction.
architecture design, training methodology, and model evalua- C. Zhao, Y. Xu, and J. Wan’s contribution in IEEE Trans-
tion. Furthermore, we integrate Siamese networks and one-shot actions on Human-Machine Systems (2022) offers insights
learning models into our framework to enhance the recognition into the historical evolution of FER techniques, encompassing
of subtle emotional cues and compute the happiness index. traditional methods and recent deep learning advancements. It
Our system’s performance is assessed in real-world scenar- discusses challenges such as occlusion and illumination varia-
ios, exhibiting superior effectiveness compared to existing tions, while also exploring future research paths in multimodal
approaches and showcasing its potential applications. emotion recognition.
 H. Li, Z. Wu, and H. Zhou’s review in IEEE Access (2020) L. K. McCorry’s article ”Physiology of the autonomic
surveys deep learning-based techniques for FER, emphasizing nervous system,” published in the American Journal of Phar-
network architectures, dataset characteristics, and performance maceutical Education in September 2007, likely provides an
evaluation metrics. It underscores the efficacy of deep learning overview of the functioning of the autonomic nervous system.
in capturing intricate facial features and addresses challenges It might cover topics such as the anatomy of the autonomic
in real-world applications. nervous system, its subdivisions (sympathetic and parasympa-
G. Zhang, H. Xu, and Y. Zhuang’s survey in IEEE Access thetic), and their respective roles in regulating bodily functions
(2022) presents an overview of FER methodologies, spanning such as heart rate, digestion, and respiratory rate. Additionally,
feature extraction, classification algorithms, and dataset con- it might discuss the neurotransmitters involved in autonomic
siderations. It tackles challenges such as facial occlusion and nervous system signaling and their pharmacological implica-
expression intensity variations, while proposing future research tions.
directions in multimodal fusion and cross-dataset learning. Lastly, Qi et al. delve into the limitations of multimodal
Subsequently, Li et al. propose a novel approach integrating Large Language Models (LLMs) within visual question an-
EEG signals and facial expressions to classify emotions in in- swering frameworks. Through structured experiments, they
dividuals with hearing impairment, achieving superior subject- reveal gaps in model comprehension and highlight the im-
dependent emotion classification through feature fusion and portance of assessing the impact of input change strategies on
deep learning classifiers. model understanding.
Zhang introduces an innovative method for collaborative Overall, these studies contribute to the advancement of
multimodal emotion recognition, combining expression-EEG FER by introducing innovative methodologies, addressing
interaction and deep autoencoders to enhance emotional state challenges, and providing insights to guide future research in
recognition through feature selection and fusion techniques. the field.
D’inc‘a et al. tackle challenges posed by facial occlusion,
particularly from masks, during the Covid-19 era. They pro- III. SYSTEM DESCRIPTION
pose an unsupervised learning approach via a Convolutional
The proposed system represents a revolutionary leap for-
Residual Autoencoder to reduce data annotation needs and
ward in the realm of educational well-being by offering a
exhibit superior performance in real-world FER applications
cutting-edge solution for real-time facial expression recogni-
compared to fully supervised methods.
tion (FER) within classroom environments. At its heart, the
Barros and Sciutti address the limitation of universal emo- system harnesses the power of state-of-the-art AI-based frame-
tion perception systems by advocating for model adaptation works, including Convolutional Neural Networks (CNNs),
through the implementation of a Spatial Transformer Plugin. Siamese networks, and one-shot learning models. This amal-
This enhancement aims to tailor facial encoders to specific gamation of advanced technologies enables the system to
affective representations, thereby improving generalization in meticulously analyze live video streams, ensuring the precise
facial expression recognition across varied affective contexts. identification and classification of students’ emotional states.
Austin Nicolai and Anthony Choi presented a paper titled By employing sophisticated computer vision algorithms, the
”Facial Emotion Recognition Using Fuzzy Systems” at the system adeptly detects even the most subtle nuances in fa-
2015 IEEE International Conference on Systems Man and cial expressions, facilitating the prompt detection of emo-
Cybernetics. In their work, they explored the application of tional cues indicative of varying happiness levels or distress.
fuzzy systems for recognizing facial emotions. The paper Upon discerning significant emotional fluctuations, the system
likely discusses how fuzzy logic, a method of processing promptly initiates a series of intervention measures. These
imprecise information, can be utilized to enhance the accuracy measures may include the implementation of personalized
of facial emotion recognition systems. This approach may support mechanisms or the notification of educators to address
involve developing fuzzy rules or models that can effectively the unique emotional needs of individual students. Such a
interpret facial expressions and classify them into different proactive approach not only cultivates a nurturing learning
emotional states. environment but also empowers educators with invaluable
The CK+ dataset is notable for being comprehensive, pro- insights to enhance students’ emotional well-being compre-
viding a wide range of facial expressions annotated with action hensively. Moreover, by automating the process of emotional
units and corresponding emotions. This richness makes it a state recognition, the system dramatically enhances situational
valuable resource for researchers working in fields such as awareness within the classroom, enabling swift interventions
computer vision, affective computing, and psychology. to mitigate potential emotional distress effectively. Further-
In terms of structure, the CK+ dataset likely includes more, the system serves as an indispensable tool for educators,
annotated images or video clips of facial expressions, with seamlessly integrating into existing classroom frameworks and
detailed labels indicating the action units activated and the as- facilitating continuous monitoring and analysis of students’
sociated emotions expressed. Researchers can use this dataset emotional states. Through its unwavering commitment to
to develop and evaluate algorithms for facial expression recog- promoting emotional intelligence and well-being, the system
nition, emotion detection, and related tasks. emerges as a cornerstone in the holistic development of
students within educational settings, laying the foundation for
a brighter, more empathetic future.
IV. DATA COLLECTION
A. FER 2013 Dataset
The FER 2013 dataset is a widely used benchmark dataset
in the field of facial expression recognition. It consists of a
collection of grayscale images representing facial expressions
of individuals belonging to various demographic groups. The
dataset contains a total of 35,887 images, each labeled with
one of seven emotion categories: Angry, Disgust, Fear, Happy,
Sad, Surprise, and Neutral.
These images were sourced from diverse internet resources
and are relatively low in resolution, typically 48x48 pixels.
The dataset is divided into three main folders: Training, Public
Test, and Private Test. The Training folder contains 28,709
images, while the Public Test and Private Test folders contain
3,589 images each.
One of the key challenges of the FER 2013 dataset is the
class imbalance, where certain emotion classes have signif-
icantly fewer samples compared to others. Additionally, the
presence of neutral expressions in the dataset adds complexity
to the task of emotion recognition, as neutral expressions may
resemble other emotions or lack distinct features.
Despite its limitations, the FER 2013 dataset serves as a
valuable resource for researchers and practitioners working
on facial expression recognition tasks. It has been exten-
sively used for training and evaluating deep learning models
and benchmarking performance in real-world scenarios. The
dataset’s accessibility and standardized format make it an
essential tool for advancing the field of affective computing
and understanding human emotions through facial expressions.
B. LFW Dataset Fig. 1. CNN Flowchart
The Labeled Faces in the Wild (LFW) dataset is a bench-
mark dataset widely used in the field of face recognition. It
comprises a collection of face images sourced from various
internet resources, including images extracted from news arti-
cles, celebrity websites, and search engine results. The dataset
contains over 13,000 labeled images of faces belonging to
more than 5,000 individuals, with considerable variations in
pose, expression, illumination, and background.
Each face image in the LFW dataset is labeled with the iden-
tity of the person depicted, enabling researchers to evaluate
face recognition algorithms on tasks such as face verification
(determining if two images belong to the same person) and
face identification (assigning a label to an unknown face). The
dataset is divided into pairs of images, with each pair labeled
as either ”same person” or ”different person.”
One of the key challenges of the LFW dataset is the
presence of large variations in pose, expression, and lighting
conditions, making face recognition a challenging task. Addi-
tionally, the dataset contains images of varying quality, with
some images exhibiting occlusions or partial facial views. Fig. 2. FER 2013 Dataset
Despite its challenges, the LFW dataset has played a crucial
role in advancing the field of face recognition and has served
as a standard benchmark for evaluating the performance of
face recognition algorithms. Its large size, diverse set of sub-
jects, and labeled annotations make it a valuable resource for
training and testing state-of-the-art face recognition systems
and assessing their robustness in real-world scenarios.

Fig. 4. Siamese Network Flowchart

Fig. 3. Siamese Network Flowchart

C. Real World Dataset

The real-world datasets collected from classroom students Fig. 5. LFW Dataset
offer several advantages for the project. Firstly, they provide
a rich and varied source of facial expressions, reflecting the
diverse emotional states present in typical classroom settings. D. Methodology
Secondly, these datasets capture genuine reactions and re- 1. Facial Feature Extraction with CNNs: Utilize Convolu-
sponses from students, ensuring that the models are trained tional Neural Networks (CNNs) to extract facial features from
on authentic emotional expressions rather than artificially live video streams. CNNs are well-suited for spatial feature
generated or staged data. Additionally, by utilizing real-world extraction from images, allowing for the identification of key
datasets, the project aims to address potential challenges such facial landmarks and expressions.
as variability in lighting conditions, facial orientations, and 2. Temporal Dependency Modeling with LSTM: In-
background clutter, which are commonly encountered in real- tegrate Long Short-Term Memory (LSTM) layers to capture
time face recognition applications. This approach enhances temporal dependencies within sequences of facial expression
the robustness and generalizability of the models, enabling frames. LSTM networks excel at learning patterns and trends
accurate assessment of the happiness index in real-world over time, enabling the model to understand the dynamics of
classroom environments. facial expressions as they evolve.
 3. Multimodal Fusion for Enhanced Understanding: leveraging the Siamese network’s ability to learn from limited
Explore the fusion of multiple modalities, such as facial labeled data through feature comparison, the proposed hybrid
expressions and contextual information, to enhance the under- approach holds promise for improving model generalization
standing of students’ emotional states. By integrating diverse and accuracy in video understanding tasks. Further research
sources of data, including audio cues or contextual context, the and experimentation are needed to explore the synergistic
model can gain deeper insights into the underlying emotions. benefits of combining CNN and Siamese architectures and
4. Real-time Analysis and Intervention Mechanisms: optimize their performance for real-world applications in video
Implement real-time analysis capabilities to continuously mon- analysis and recognition tasks.
itor students’ emotional states in classroom settings. Upon
detection of significant emotional fluctuations, trigger imme- V. CONCLUSION
diate intervention mechanisms, such as providing supportive In conclusion, our real-time face emotion recognition sys-
messages or alerting educators to address students’ emotional tem, which integrates Convolutional Neural Networks (CNNs)
needs promptly. and Siamese Neural Networks for one-shot learning, repre-
5. Implementation and Integration with TensorFlow- sents a significant leap forward in accurately discerning and
Keras: Implement the proposed methodology using the categorizing emotions such as happiness, sadness, anger, and
TensorFlow-Keras library, which offers ease of accessibility neutrality within dynamic environments. Through rigorous
and flexibility. Leveraging TensorFlow-Keras simplifies model experimentation and meticulous evaluation, we have not only
development and deployment, while also providing access validated the efficacy and robustness of our approach but also
to a supportive community for troubleshooting and further demonstrated its ability to capture subtle emotional nuances
enhancements. and compute the happiness index with exceptional precision.
6. Evaluation and Validation: Evaluate the performance Beyond its technical prowess, our work holds immense
of the developed system using appropriate metrics, such as practical value across various domains. By enhancing user
accuracy, precision, recall, and F1 score. Validate the system’s experience, facilitating mental health monitoring, and refining
effectiveness in accurately identifying and classifying students’ human-computer interaction, our system serves as a corner-
emotional states in real-world classroom environments through stone for developing emotion-aware technologies capable of
comprehensive testing and validation procedures. intelligently adapting and responding to user emotions in
Overall, the proposed methodology combines advanced real-time. This not only fosters personalized and engaging
deep learning techniques with real-time analysis capabilities interactions but also opens new avenues for empathy-driven
to create an innovative system for assessing and analyzing design and intervention strategies.
students’ emotional well-being in educational settings. By As we look to the future, further refinements in our model’s
integrating facial expression recognition with contextual in- performance through advanced training methodologies and
formation and intervention mechanisms, the system aims to the integration of multimodal sensory inputs hold promise.
foster supportive learning environments and enhance overall Expanding the scope of emotion recognition to encompass
educational outcomes. a wider array of emotional expressions and exploring novel
applications in diverse contexts remain critical avenues for
E. Result and Discussion exploration.
1) CNN Model:: The CNN architecture, trained on video In essence, our project underscores the transformative po-
data using a ViT framework, demonstrated a significant dis- tential of deep learning and real-time processing technolo-
parity between training and validation accuracies, indicative gies, particularly when coupled with innovative approaches
of overfitting. Despite achieving a high training accuracy like Siamese Neural Networks. By revolutionizing emotion
of 89%, the validation accuracy remained notably lower at recognition systems, our work lays the foundation for a future
68%. This discrepancy underscores the challenges in effec- where technology not only understands but also empathizes,
tively learning features from video sequences using traditional fostering well-being, empathy, and enriched human-machine
CNN architectures. The elevated and fluctuating loss values interactions on a global scale.
further corroborate these observations, indicating suboptimal
performance and insufficient feature learning. To address these R EFERENCES
limitations, the exploration of hybrid architectures, such as [1] C. M. Tyng, H. U. Amin, M. N. M. Saad and A. S. Ma-
CNN-LSTM, is proposed as a more effective and robust lik, ”The influences of emotion on learning and memory”, Fron-
tiers Psychol., vol. 8, pp. 1-22, Aug. 2017, [online] Available:
alternative for capturing long-term dependencies in video data [Link]
and enhancing model performance and generalization across [2] R. Pandey and A. K. Choubey, ”Emotion and health: An overview”, J.
tasks. Projective Psychol. Mental Health, vol. 17, pp. 135-152, Jan. 2010.
[3] M. N. A. Wahab, A. Nazir, A. T. Z. Ren, M. H. M. Noor, M. F.
2) Siamese Neural Network:: Incorporating the Siamese Akbar and A. S. A. Mohamed, ”Efficientnet-lite and hybrid CNN-KNN
neural network for one-shot learning alongside the CNN implementation for facial expression recognition on raspberry pi”, IEEE
architecture presents an opportunity to address the limitations Access, vol. 9, pp. 134065-134080, 2021.
[4] A. Mollahosseini, B. Hasani and M. H. Mahoor, ”AffectNet: A database
of traditional deep learning architectures in capturing tempo- for facial expression valence and arousal computing in the wild”, IEEE
ral dependencies and learning features from video data. By Trans. Affect. Comput., vol. 10, no. 1, pp. 18-31, Jan. 2019.
 [5] S. Li and W. Deng, ”Reliable crowdsourcing and deep locality-
preserving learning for unconstrained facial expression recognition”,
IEEE Trans. Image Process., vol. 28, no. 1, pp. 356-370, Jan. 2019
[6] I. J. Goodfellow et al., ”Challenges in representation learning: A
report on three machine learning contests”, Proc. Int. Conf. Neural Inf.
Process., pp. 117-124, 2013.
[7] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar and I. Matthews,
”The extended cohn-kanade dataset (CK+): A complete dataset for
action unit and emotion-specified expression”, Proc. IEEE Comput. Soc.
Conf. Comput. Vis. Pattern Recognit. Workshops, pp. 94-101, Jun. 2010.
[8] L. K. McCorry, ”Physiology of the autonomic nervous system”, Amer.
J. Pharmaceutical Educ., vol. 71, no. 4, pp. 78, Sep. 2007.
[9] M. A. Hasnul, A. A. Aziz, S. Alelyani, M. Mohana and A. A. Aziz,
”Electrocardiogram-based emotion recognition systems and their appli-
cations in healthcare—A review”, Sensors, vol. 21, no. 15, pp. 5015, Jul.
2021, [online] Available: [Link]
[10] F. Ekman, ”Facial action coding system” in Environmental Psychology
& Nonverbal Behavior, Palo Alto, CA, USA:Consulting Psychologists
Press, 1978.
[11] Austin Nicolai and Anthony Choi, ”Facial Emotion Recognition Using
Fuzzy Systems”, 2015 IEEE International Conference on Systems Man
and Cybernetics, pp. 2216-2221, 9-12 Oct. 2015.
[12] M. I. N. P. Munasinghe, ”Facial Expression Recognition Using Facial
Landmarks and Random Forest Classifier”, 2018 IEEE/ACIS 17th
International Conference on Computer and Information Science (ICIS),
pp. 423-427, 6-8 June 2018.
[13] Muhammad Ilhamdi Rusydi, Rizka Hadelina, Oluwarotimi W. Samuel,
Agung Wahyu Setiawan and Carmadi Machbub, ”Facial Features Extrac-
tion Based on Distance and Area of Points for Expression Recognition”,
2019 IEEE - 4th Asia-Pacific Conference on Intelligent Robot Systems
(ACIRS), pp. 211-215, 13-15 July 2019.
[14] Bendjillali Ridha Ilyas, Beladgham Mohammed, Merit Khaled, Abdel-
malik Taleb Ahmed and Alouani Ihsen, ”Facial Expression Recognition
Based on DWT Feature for Deep CNN”, 2019 6th International Confer-
ence on Control Decision and Information Technologies (CoDIT), pp.
344-348, 23-26 April 2019.
[15] Lakshmi Sarvani Videla and P.M. Ashok Kumar, ”Facial Expression
Classification Using Vanilla Convolution Neural Network”, 2020 7th
International Conference on Smart Structures and Systems (ICSSS), pp.
1-5, 23-24 July 2020.
[16] Akriti Jaiswal, A. Krishnama Raju and Suman Deb, ”Facial Emotion
Detection Using Deep Learning”, 2020 International Conference for
Emerging Technology (INCET), pp. 1-5, 5-7 June 2020.
[17] Prashant Dhope, ”Human Emotion Recognition from Facial Expressions
- A Review”, International Journal of Innovative Research in Sci-
enceEngineering and Technology (IJIRSET), vol. 10, no. 9, pp. 12985-
12991, September 2021.
[18] F. Ekman, ”Facial action coding system” in Environmental Psychology
& Nonverbal Behavior, Palo Alto, CA, USA:Consulting Psychologists
Press, 1978.
[19] M. A. Hasnul, A. A. Aziz, S. Alelyani, M. Mohana and A. A. Aziz,
”Electrocardiogram-based emotion recognition systems and their appli-
cations in healthcare—A review”, Sensors, vol. 21, no. 15, pp. 5015, Jul.
2021, [online] Available: [Link]
[20] K. He, X. Zhang, S. Ren and J. Sun, ”Deep Residual Learning for Image
Recognition” in CoRR, 2015.