Text-Based Emotion Recognition Using Deep Learning
Text-Based Emotion Recognition Using Deep Learning
Research Article
Text-Based Emotion Recognition Using Deep Learning Approach
Received 29 April 2022; Revised 22 May 2022; Accepted 1 July 2022; Published 23 August 2022
Copyright © 2022 Santosh Kumar Bharti et al. This is an open access article distributed under the Creative Commons Attribution
License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is
properly cited.
Sentiment analysis is a method to identify people’s attitudes, sentiments, and emotions towards a given goal, such as people,
activities, organizations, services, subjects, and products. Emotion detection is a subset of sentiment analysis as it predicts the
unique emotion rather than just stating positive, negative, or neutral. In recent times, many researchers have already worked on
speech and facial expressions for emotion recognition. However, emotion detection in text is a tedious task as cues are missing,
unlike in speech, such as tonal stress, facial expression, pitch, etc. To identify emotions from text, several methods have been
proposed in the past using natural language processing (NLP) techniques: the keyword approach, the lexicon-based approach, and
the machine learning approach. However, there were some limitations with keyword- and lexicon-based approaches as they focus
on semantic relations. In this article, we have proposed a hybrid (machine learning + deep learning) model to identify emotions in
text. Convolutional neural network (CNN) and Bi-GRU were exploited as deep learning techniques. Support vector machine is
used as a machine learning approach. The performance of the proposed approach is evaluated using a combination of three
different types of datasets, namely, sentences, tweets, and dialogs, and it attains an accuracy of 80.11%.
text sentences, we have preprocessed the data. Then put the The rest of the article is organized as follows: Section 2
data in different types of ML and DL models. presents the literature survey on emotion detection. The
According to Ekman [1], emotions are categorized into proposed scheme is explained in Section 3. Section 4 draws
6 different types, as follows: joy, sadness, fear, surprise, results and analysis of the proposed work. Finally, Section 5
anger, and disgust. Further, emotion is also described in concludes the article with future directions.
various forms, such as love, optimism, etc., as shown in
Figure 1 [2]. Facial expressions, gestures, speech, and text 2. Literature Review
generally express the mood and emotions of a human
being. Unlike facial expression and speech recognition, a Several studies have used various techniques to detect
text sentence loses the ability to define itself because it is emotions from text [3–7]. It will show that which is the best
tasteless. Because of the complexity and ambiguity of the model and gives us a higher accuracy.
text, it is a difficult task to find out the emotions of that text. Seal et al. [4] have performed emotion detection with a
It becomes a difficult task to recognize the emotion of a keyword-based approach mainly focused on phrasal verbs.
given text as each word can have a different meaning and They used ISEAR [5] data, preprocessed the data, and then
morphological form. applied the keyword-based approach. They discovered
In recent times, researchers have proposed various several phrasal verbs that should have been associated with
methods to detect the emotions of the text, such as keyword- emotion terms but were not, and so they built their own
based, lexical affinity, learning-based, and hybrid models [3]. database. They recognized phrasal verbs and keywords
In the beginning, they introduced a rule-based approach that synonymous with various emotions and categorized them
consisted of two approaches, namely, lexical affinity-based using their database. They did, however, achieve a much
and keyword-based. Later on, a new approach came into higher accuracy of 65%, but they were unable to address the
existence, i.e., the learning-based approach. This method was researcher’s existing issues, such as an insufficient list of
more accurate and gave better results. In a learning-based emotion keywords and a lack of respect for word semantics
approach, different models are used to detect emotion. Many in meaning. The work by Alotaibi [7] has worked on a
researchers have also started combining the approaches and learning-based approach. He has used the ISEAR [5] da-
making them hybrid in the search for high accuracy. As per tabase for emotion detection. Then, using classifiers like
the study, deep learning models show better accuracy than Logistic Regression, K-Nearest Neighbour (KNN), XG-
machine learning models for large sizes of text or data. But Boost, and Support Vector Machine (SVM), he preprocessed
for small data, machine learning gives us better accuracy. and trained the data. According to him, all other classifiers
Still, none of the approaches gave a complete solution to poorly performed as compared to Logistic Regression. Fi-
detect the emotion from a given text. nally, he said that the deep learning technique would help to
There were many limitations in the existing solutions, improve the model.
such as that they did not have a list of all the emotions. The Xu et al. [8] P. Xu et al. [12] has proposed an Emo2Vec
existing lists have an inadequate vocabulary of words in the method that encodes emotional semantics into vector form.
lexicon, disregarded words, semantics-based context, low They have trained Emo2Vec on a multitask learning
extractions of contextual information from the given sen- framework by using smaller and larger datasets (smaller
tences, do not perform well for detecting some specific datasets such as ISEAR, WASSA, and Olympic). It shows
emotions; weak context information extraction, loose se- that their results are better than those of Convolution Neural
mantic feature extraction, less computational speed, ignored Network (CNN), DeepMoji embedding, and more. They
relations between features, an inadequate amount of data, have utilized their work on emotion analysis, sarcasm
and a high number of misclassifications. Some models were classification, stress detection, etc. Finally, the model
not suited well for frequently occurring emojis, weak se- Emo2Vec, when combined with Logistic Regression and
mantic information extraction, and the structure of the GloVe, can achieve more competitive results. Ragheb et al.
sentence. It differs from model to model. There were many [9] W. Ragheb et al. [13]worked on detecting emotions from
limitations in this system that were fulfilled by previous textual conversations through the help of learning-based
researchers. The proposed model has fulfilled many of the model. Their data comprises 6 types of emotions that Paul
existing limitations. Ekman has [1] described. In their methods, two phases of
Emotion detection is one of the big advantages of hu- encoder and classification are present. After the data is
man-machine interaction as a nonliving thing can sense or collected, it is tokenized and passed to an encoder, which
feel like a human being. Our proposed model can detect then passes it on to Bi-LSTM units that have been trained
emotions from text sentences that are tasteless as they do not using average stochastic gradient descent (ASGD). To avoid
have any tone or expression. Many researchers have worked over-fitting, they have applied dropouts between the LSTM
on a single dataset. But we have worked on three datasets units. Then, to focus on specific emotion-carrying conver-
which include the textual form of simple sentences, tweets, sations, a self-attention mechanism was used. The data was
and dialogs to detect emotions. Our text-based emotion classified into its respective categories through the help of a
recognition model can be implemented on any system. For dense layer and a SoftMax activation. The model showed an
business potential, this model can help to find emotions F1 score of 75.82%.
from customer reviews, services, give security for social Suhasini and Srinivasu [10] M. Suhasini and B. Srinivasu
media users, and many others. [14]used a learning-based approach in which machine
Computational Intelligence and Neuroscience 3
love sion
se ce submis
ism
re ptan
optim
n ity acce
trust
joy sion
interest ehen
appr
ecstasy
ion
anticipation
fear
rat
mi
ss
sivene
ad
aggres v igilance terror
awe
rage amazement
anger
se
annoyance
pri
sur
loathing grief distraction
ss
pt disgust
contem
ne
roval
sad
disapp
pe
ns
om
ive
bored s e
remor
ne
ss
Figure 1: Various types of Emotions.
learning classifiers were used to detect or classify emotions. generate generalized and personalized recommendations
They have used KNN and Naive Bayes (NB) for the detection for users based on their Twitter activity [4].
of emotions using tweets in the Sentiment 140 corpus. They
compared the accuracy of the NB to that of the KNN, finding
that the NB was 72.06% compared to the KNN’s accuracy of
3. Proposed Scheme
55.50%. The model’s drawbacks are that they have low This section describes the proposed system that consists of
extractions of contextual information in the given sentences. data collection and data preprocessing. After preprocessing,
Hasan et al. [11] M. Hasan et al. [15] used the supervised the data will move forward as an input to both ML and DL
machine learning method and an emotion dictionary in their models. In ML, the preprocessed data will be given as the
proposed model for recognizing emotions from the text. To input to ML classifiers and will show the results of all ML
carry out emotion classification, they performed two tasks, classifiers. Furthermore, it will select the best ML model that
first offline and then online. Through the help of emotion- gives the highest accuracy. In DL, the data is converted into
labeled text from Twitter and other classifiers, an offline vector form and given as an input to the DL models. Before
model was developed for emotion classification. The training that, we used a pretrained word vector to make the word
dataset was built by preprocessing the data. Then, the online embedding matrix and add the embedded layer to the DL
approach classified streaming content of tweets in real time model [18]. After performing on individual models, we
using the model developed in the offline approach [12]. Their combine the two best DL models based on accuracy and F1
model had a 90% overall accuracy. score. Combining them will give us the latent vector and it
Rodriguez et al. [13] use emotion analysis to identify will be given as an input to the best ML model for the
hate speech on social media. Their aim with this research prediction of emotions. Finally, it will select the best ac-
was to locate and analyse the unstructured data of selected curacy of all the ML, DL, and hybrid models. The pipelined
social media posts that intend to spread hate in the diagram of our proposed model is shown in Figure 2.
comment sections. Cao et al. [14] exploited machine and
deep learning approaches to evaluate emotion in textual
data. They also highlight the issues and challenges re- 3.1. Dataset Description. The data is taken from three dif-
garding emotion detection in text. Acheampong et al. [15] ferent datasets: ISEAR, WASSA, and Emotion-stimulus,
F. A. Acheampong [20] surveyed the concept of emotion which have text and emotions as the attributes. These
detection (ED) from texts and highlighted the main ap- datasets consist of three different types of text: normal
proaches adopted by researchers in the design of text-based sentences, tweets, and dialogs.
ED systems. Navarrete Verma [16] P. Nandwani and R. The International Survey on Emotion Antecedents and
Verma [21] described the process used to create an emotion Reactions (ISEAR) database [5] was built over a number of
lexicon enriched with the emotional intensity of words and years in the 1990s by a vast community of psychologists from
focused on improving the emotion analysis process in texts all over the world under the direction of Wallbott and
[13]. Sailunaza and Alhajj [17] K. Sailunaz and R. Alhajj Scherer. They experienced 7 types of emotions (joy, anger,
[22] used Twitter data to detect emotion and sentiment guilt, sadness, disgust, fear, and shame). According to a
from text. They exploited sentiment and emotion scores to cross-cultural survey conducted in 37 countries across five
4 Computational Intelligence and Neuroscience
surprise
TfidfTransformar
CountVectorizer
joy
ML Classifier
1.5% (212)
fear
Output
0)
Input
119
sadness Emotions
%(
Text
8.2
25.2% (3659)
Pipeline
21.4% (3109)
Figure 4: Machine Learning model to detect emotions from text.
22.04% (3195)
GRU
disgust 21.6% (3112)
anger
Conv1d
Input Output
Text Emotions
Flatten
Global Layer
Max_pooling
Embedding Layer
CNN
Conv1d Global Flatten
Max_pooling Layer
CNN + GRU
SVM
Input Output
Bi-GRU Flatten
Text Layer Emotions
Bi-GRU
Embedding
Layer
Figure 8: Hybrid model to detect emotions from text.
our proposed model. After feature extraction, the em- learning. In the individual results of deep learning models,
bedding layer of size (18210, 300) will be input for the CNN and Bi-GRU performed well. Similarly, in the machine
CNN model shown in Figure 7. The training vector will learning algorithms, SVM performed well [29]. There-
be input into the CNN model to predict the emotions for fore, we choose the best classifiers from both categories
the data [27]. to improve the result. In the hybrid model, we combined
the ML and DL models, as shown in Figure 8. So, we
combined the two best deep learning models, which give
3.5. Hybrid Model. The proposed hybrid model combines the best accuracy and F1 score. After getting the best DL
deep learning and machine learning algorithms to predict models, the latent vector was given as an input to the best
emotions. The overall system diagram is shown in Figure 8. ML models, which predicts emotions as it shows high
Deep learning consists of CNN and Bi-GRU, and machine accuracy [30].
learning consists of an SVM classifier. It starts with input
datasets, which are fed into the word embedding layer, i.e.,
word2vec. After getting the embedding vector, it needs to 4. Results and Discussion
be fed into both the deep learning algorithms, namely,
CNN and Bi-GRU. From CNN and Bi-GRU models, we We have performed many experiments using various
have removed the last layer, and so they will act as encoders methods to get the best accuracy for our proposed model.
[28]. Emotion classification with a machine learning approach, a
Furthermore, both of these encoders will generate a deep learning approach, and our hybrid model approach on
latent vector for the given input embedding vector. Lastly, the multitext dataset consisting of sentences, tweets, and
these latent vectors will be concatenated and will be fed to dialogs. Three datasets are used for performing these
the SVM classifier. The SVM classifier will predict the experiments.
emotion of these input texts. First, the text was given as an input to the pipeline, which
The proposed hybrid model gives improved results in then converts text into a vector. These vectors were used to
terms of accuracy and F1 score due to the selection of train the ML Classifier. The accuracy which is listed in
classification models in both deep learning and machine Table 2 is from the machine learning approach.
Computational Intelligence and Neuroscience 7
Table 2: Evaluation matrix for ML Classifiers of 82.39, a recall of 80.40, an F1 score of 81.27, and an
ML classifier Precision Recall F1 score Accuracy
accuracy of 80.11%.
In the future, we will try more potential classifiers or
SVM 81.45 78.36 79.67 78.97
ensemble techniques to improve the results. In the deep
RF 79.42 75.66 77.02 76.25
NB 61.75 51.41 49.61 68.94 learning approach, we may get the combination of CNN, Bi-
DT 72.48 69.70 70.94 69.42 GRU, and LSTM to improve the results. Additionally, we
will work on the structure of text sentences and some of the
regional languages. Moreover, in this digital world, people’s
Table 3: Evaluation matrix for with DL model. usage of sending text messages, uploading tweets, and
writing online reviews of products have been in great use and
Deep learning Precision Recall F1 score Accuracy demand. Therefore, by having a lot of data, we can make a
GRU 78.37 78.94 78.65 78.02 real-time text-based emotion recognition model to find the
Bi-GRU 80.62 79.64 80.09 79.46 emotions or moods of the people.
CNN 82.12 79.92 80.76 79.32
Data Availability
Table 4: Evaluation matrix for our hybrid model. The data that support the findings of this study are available
on request from the corresponding author.
Hybrid Precision Recall F1 score Accuracy
CNN + Bi-GRU + SVM 82.39 80.40 81.27 80.11 Conflicts of Interest
The authors declare that they have no conflicts of interest.
Second, the features were extracted using the pretrained
word vector, and the embedding matrix of size (18210, 300)
was the input layer for the DL model. The padded vector was
Acknowledgments
trained on the DL model. Table 3 shows the accuracy of the The authors thank the Deanship of Scientific Research at
DL models. King Khalid University for funding this work through the
Finally, our hybrid model is a combination of both ML research group program under grant number R. G. P. 2/217/
and DL models. Both the DL model CNN and Bi-GRU are 43.
combined, and the latent vector from them is an input vector
for the SVM model for training the data. Table 4 represents References
the accuracy of our proposed hybrid model.
As with the basic model of ML and DL, we get better [1] P. Ekman, “Basic emotions,” Handbook of cognition and
results, but they are not the best results. The ML approach emotion, vol. 98, no. 45-60, p. 16, 1999.
will give the best accuracy for different types of emotions, [2] R. Plutchik, “The nature of emotions,” American Scientist,
and the same for the DL approach. So, by combining the vol. 89, no. 4, p. 344, 2001.
models, we get the highest accuracy. [3] C. R. Chopade, “Text based emotion recognition: a survey,”
Emotion detection from text is one of the most chal- International Journal of Science and Research, vol. 2, no. 6,
pp. 409–414, 2015.
lenging and important tasks as it does not have any ex-
[4] D. Seal, U. K. Roy, and R. Basak, “Sentence-level emotion
pression of emotions and the structure of text sentences. detection from text based on semantic rules,” Information and
Researchers are trying hard to get a complete solution in this Communication Technology for Sustainable Development,
field, but they all have failed. But they have found the best Springer, Singapore, pp. 423–430, 2020.
solution for facial emotion expression and speech emotion [5] A. A. Alnuaim, M. Zakariah, P. K. Shukla et al., “Human-
recognition. Still, there is a mystery in this field. computer interaction for recognizing speech emotions using
multilayer perceptron classifier,” Journal of Healthcare En-
gineering, vol. 2022, Article ID 6005446, 12 pages, 2022.
5. Conclusion and Future Direction [6] D. Singh, V. Kumar, M. Kaur, M. Y. Lee, and H.-N. Lee,
“Screening of COVID-19 suspected subjects using multi-
In this paper, proposed a text-based emotion recognition crossover genetic algorithm based dense convolutional neural
model. The proposed model is a combination of deep network,” IEEE Access, vol. 9, pp. 142566–142580, 2021.
learning and machine learning approaches. This proposed [7] S. M. Mohammad and F. Bravo-Marquez, “WASSA-2017
hybrid approach uses the combination of three datasets, Shared Task on Emotion Intensity,” 2017, [Link]
namely, ISEAR, WASSA, and the Emotion-Stimulus dataset. abs/1708.03700.
The proposed model has many advantages, as it can work on [8] P. Xu, A. Madotto, C. S. Wu, J. H. Park, and P. Fung,
“Emo2vec: learning generalized emotion representation by
multitext sentences, tweets, dialogs, keywords, and lexicon
multi-task training,” 2018, [Link]
words of emotions that can be easily detected. According to [9] W. Ragheb, J. Azé, S. Bringay, and M. Servajean, “Attention-
the ML classifier, SVM gives the highest accuracy of 78.97%. based modeling for emotion detection and classification in
In the DL method, the Bi-GRU model achieves the highest textual conversations,” 2019, [Link]
accuracy of 79.46%, and the CNN model achieves the highest [10] M. Suhasini and B. Srinivasu, “Emotion detection framework
F1-score of 80.76. The hybrid model has achieved a precision for twitter data using supervised classifiers,” Data Engineering
8 Computational Intelligence and Neuroscience
and Communication Technology, Springer, Singapore, Language and Technology (CLT), pp. 87–94, IEEE, Lahore,
pp. 565–576, 2020. Pakistan, February 2020.
[11] M. Hasan, E. Rundensteiner, and E. Agu, “Automatic emotion [27] S. Garcı́a, J. Luengo, and F. Herrera, Data Preprocessing in
detection in text streams by analyzing twitter data,” Inter- Data Mining, Springer International Publishing, Cham,
national Journal of Data Science and Analytics, vol. 7, no. 1, Switzerland, 2015.
pp. 35–51, 2019. [28] E. M. Onyema, P. K. Shukla, S. Dalal, M. N. Mathur,
[12] A. S. Rajawat, P. Bedi, S. B. Goyal et al., “Fog big data analysis M. Zakariah, and B. Tiwari, “Enhancement of patient facial
for IoT sensor application using fusion deep learning,” recognition through deep learning algorithm: ConvNet,”
Mathematical Problems in Engineering, vol. 2021, Article ID Journal of Healthcare Engineering, vol. 2021, Article ID
6876688, 16 pages, 2021. 5196000, 8 pages, 2021.
[13] A. Rodriguez, Y. L. Chen, and C. Argueta, “FADOHS: [29] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient
framework for detection and integration of unstructured data estimation of word representations in vector space,” 2013,
of hate speech on facebook using sentiment and emotion [Link]
analysis,” IEEE Access, vol. 10, pp. 22400–22419, 2022. [30] T. M. Mitchell, “Machine learning and data mining,” Com-
[14] L. Cao, S. Peng, P. Yin, Y. Zhou, A. Yang, and X. Li, “A survey munications of the ACM, vol. 42, no. 11, pp. 30–36, 1999.
of emotion analysis in text based on deep learning,” in [31] D. Ghazi, D. Inkpen, and S. Szpakowicz, “Detecting emotion
Proceedings of the 2020 IEEE 8th International Conference on stimuli in emotion-bearing sentences,” Computational Lin-
Smart City and Informatization (iSCI), pp. 81–88, IEEE, guistics and Intelligent Text Processing, Springer, Cham,
Guangzhou, China, December 2020. pp. 152–165, 2015.
[15] F. A. Acheampong, C. Wenyu, and H. Nunoo-Mensah, [32] F. M. Alotaibi, “Classifying text-based emotions using logistic
“Textbased emotion detection: a,” Engineering Reports, vol. 2, regression,” VAWKUM Transactions on Computer Sciences,
no. 7, Article ID e12189, 2020. vol. 7, no. 1, pp. 31–37, 2019.
[16] A. S. Navarrete, C. Martinez-Araneda, C. Vidal-Castro, and [33] A. A. Alnuaim, M. Zakariah, C. Shashidhar et al., “Speaker
C. Rubio-Manzano, “A novel approach to the creation of a gender recognition based on deep neural networks and
labelling lexicon for improving emotion analysis in text,” The ResNet50,” Wireless Communications and Mobile Computing,
Electronic Library, vol. 39, 2021. vol. 2022, Article ID 4444388, 13 pages, 2022.
[17] K. Sailunaz and R. Alhajj, “Emotion and sentiment analysis [34] H. KaushikKaushik, D. Singh, and
from Twitter text,” Journal of Computational Science, vol. 36, H. KaurAlshazlyZaguiaHamam, “Diabetic retinopathy diag-
Article ID 101003,, 2019. nosis from fundus images using stacked generalization of deep
[18] K. Cho, B. Van Merriënboer, C. Gulcehre et al., “Learning models,” IEEE Access, vol. 9, pp. 108276–108292, 2021.
phrase representations using rnn encoder-decoder for sta- [35] Y. Pathak, P. K. Arya, and K. V. Arya, “Deep bidirectional
tistical machine translation,” 2014, [Link] classification model for COVID-19 disease infected patients,”
1078. IEEE/ACM Transactions on Computational Biology and Bio-
[19] J. X. Chen, D. M. Jiang, and Y. N. Zhang, “A hierarchical informatics, vol. 18, no. 4, pp. 1234–1241, 2021.
bidirectional GRU model with attention for EEG-based
emotion classification,” IEEE Access, vol. 7, pp. 118530–
118540, 2019.
[20] A. Motwani, P. K. Shukla, and M. Pawar, “Novel framework
based on deep learning and cloud analytics for smart patient
monitoring and recommendation (SPMR),” Journal of Am-3
bient Intelligence and Humanized Computing, 2021.
[21] P. Kim, “Convolutional neural network,” in MATLAB Deep
Learning, pp. 121–147, A press, Berkeley, CA, 2017.
[22] A. A. Alnuaim, M. Zakariah, A. Alhadlaq et al., “Human-
computer interaction with detection of speaker emotions
using convolution neural networks,” Computational Intelli-
gence and Neuroscience, vol. 2022, p. 16, Article ID 7463091,
2022.
[23] D. Singh, M. Kaur, M. Y. Jabarulla, H.-N Lee, and H.-N. Lee,
“Evolving fusion-based visibility restoration model for hazy
remote sensing images using dynamic differential evolution,”
in IEEE Transactions on Geoscience and Remote
Sensingvol. 60, , pp. 1–14, IEEE, 2022.
[24] D. Acharya, “Comparative analysis of feature extraction
technique on EEG-based dataset,” in Soft Computing for
Problem Solving, A. Tiwari, K. Ahuja, A. Yadav, J. C. Bansal,
K. Deep, and A. K. Nagar, Eds., vol. 1392, Singapore, Springer,
2021.
[25] P. Nandwani and R. Verma, “A review on sentiment analysis
and emotion detection from text,” Social Network Analysis
and Mining, vol. 11, no. 1, p. 81, 2021.
[26] N. Azam, B. Tahir, and M. A. Mehmood, “Sentiment and
emotion analysis of text: a survey on approaches and re-
sources,” in Proceedings of the 7th International Conference on