A Comparative Study of Sentiment Analysis Using
Different Machine Learning Techniques on US
Airline Twitter Data
Ram Srivastava Aaditya Varshney
Student of Computer Engineering Student of Computer Engineering
NMIMS’ MPSTME Shirpur NMIMS’ MPSTME Shirpur
Shirpur, India Shirpur, India
ramsrivastava4321@[Link] [Link]@[Link]
Abstract: In today's highly competitive business extract meaningful insights manually. Sentiment Analysis
landscape, customer satisfaction has become a crucial provides a powerful solution by enabling the
factor for business success. Companies invest substantial
resources—both financial and human—into strategies
aimed at understanding and meeting customer needs.
However, due to the limitations of manual analysis in identification and classification of opinions within textual
handling diverse customer demands, many organizations data. The primary objective of Sentiment Analysis is to
struggle to achieve high levels of customer satisfaction. determine the polarity of text—whether it is positive,
This often results in customer attrition and increased negative, or neutral [4]. This approach helps decision-
marketing expenditures. A promising solution to this makers track shifts in public sentiment regarding products,
challenge is Sentiment Analysis, a technique that services, activities, or even social and political entities. By
integrates Natural Language Processing (NLP) and leveraging Sentiment Analysis, businesses can enhance their
Machine Learning (ML) to extract valuable insights offerings, while political and social organizations can
from public opinions on various topics, products, and optimize their strategies based on public perception. Most
services. By leveraging online data, businesses can gain a sentiment analysis datasets are sourced from social media
deeper understanding of customer sentiment. Our platforms and stored in structured files. However, challenges
research aims to determine the most effective approach arise when dealing with large, imbalanced, and multi-class
for sentiment analysis on a large, imbalanced, multi-class datasets. In this study, we utilized the Twitter US Airline
dataset. The results indicate that Long Short-Term Sentiment dataset, a real-world dataset containing diverse
Memory (LSTM) approach yield the highest accuracy of airline-related opinions. To process and analyze the data, we
91%, making them the most effective techniques in our applied Natural Language Processing (NLP) [15]
study. techniques, including Long Short-Term Memory (LSTM),
for text vectorization. We then employed Machine Learning
Keywords--- Sentiment Analysis, Natural Language (ML) classification algorithms—Random Forest, and
Processing, LSTM, Random Forest, XGBoost, Machine
XGBoost—to classify sentiment polarity. Finally, we
Learning
compared these algorithms and NLP techniques to identify
the most effective approach for sentiment analysis.
I. INTRODUCTION
Customer satisfaction reflects consumers' perceptions of
products, services, and organizations. Research has II. LITREATURE REVIEW
consistently shown that product or service quality [1] and In recent years, sentiment analysis has gained considerable
customer happiness are critical factors influencing business traction, especially in the context of social media, where
performance [2]. To maintain a competitive edge, businesses platforms like Twitter provide valuable, real-time data for
must deeply understand customer expectations and ensure understanding public opinion. The study by Md. Taufiqul
their satisfaction, fostering long-term engagement. Haque Khan Tusar and Md. Touhidul Islam focuses on a
Additionally, effective customer relationship management comparative evaluation of sentiment analysis using Natural
plays a crucial role in sustaining business growth. A study Language Processing (NLP) and various machine learning
conducted in [3] examined service quality and customer techniques applied to US airline Twitter data. Their work
satisfaction in the top 14 U.S. airlines from 2007 to 2011. builds upon a substantial body of literature that explores
The findings indicated that the airline industry has faced both the methodological aspects and application areas of
challenges in delivering exceptional services and meeting sentiment analysis.
the diverse needs of consumers. A significant portion of
customer feedback on social media and other platforms Earlier studies such as those by Suchanek and Králová [1]
exists in an unstructured format, making it difficult to and Safariena and Shamsudin [2] examined the broader
impact of customer satisfaction on business performance.
Baker [3] focused specifically on the airline industry, Figure 1. Workflow of Sentiment Analysis using NLP and
comparing service quality perceptions between traditional Machine Learning
and low-cost carriers, which closely aligns with the domain
addressed by Khan Tusar and Islam.
From a technical perspective, several studies have evaluated A. Data Collection
different machine learning models for sentiment The data originally came from CrowdFlower’s Data for
classification. Wongkar and Angdresey [5], as well as Everyone library. Contributors scraped Twitter data of the
Mandloi and Patel [6], used algorithms like Naive Bayes and travelers who traveled through six US airlines in February
other supervised learning techniques for analysing Twitter 2015. They provided the data on Kaggle as a dataset, named
data. Naresh and Krishna [8], and Jayamalini et al. [10], Twitter US Airline Sentiment [13] under the CC BY-NCSA
provided comparative analyses of machine learning 4.0 license. The dataset has around 14640 records and 15
methods, which mirrors the approach taken in the final attributes. It contains whether the sentiment of the tweets in
study. this set was positive, neutral, or negative for six US airlines
services. Fig.2 shows the frequency of polarity in the
Kumar et al. [7] and M.V.K. et al. [9] specifically focused dataset.
on airline-related tweets, adding contextual relevance. Other
studies, like Elhadad et al. [12], explored multilingual B. Data Preprocessing
sentiment analysis, broadening the linguistic scope of such A tweet can contain various symbols (!, #, @, etc.),
research. numbers, punctuation, or stop-words. Stop-words mean
which words don’t comprise any sentiment. Such as he, she,
Overall, the final paper synthesizes insights from these the, is, that. These are noisy data for Sentiment Analysis. So,
works and provides a targeted, comparative evaluation of we have cleaned the data for further processing by removing
sentiment analysis techniques, adding valuable knowledge punctuation, number, symbol, converting all the characters
specific to airline customer feedback via Twitter. into lowercase.
III. METHODOLOGY C. Classification
There are the steps for our approaches: We have used the Train-Test-Split technique to divide the
dataset into 75% for training and 25% for testing. Then
1) Collecting dataset to train and test ML Classifier.
applied different classification algorithms of Supervised
2) Pre-processing the dataset for subsequent processing. Machine Learning on training data to train Machine
3) Converting textual data into vector form using NLP. Learning Classifiers and tested with testing data. Applied
algorithms are Random Forest, XGBoost and LSTM.
4) Dividing the dataset into training and testing groups.
IV. RESULT AND DISCUSSION
Then train the ML Classifier with training data and We evaluated the performance of our approaches using
predict the polarity of testing data. Fig.1 depicts the standard classification metrics: Accuracy, Precision, Recall,
workflow of Sentiment Analysis using NLP and different and F1-Score. Since the dataset was imbalanced, relying
Machine Learning techniques. solely on accuracy could be misleading. Therefore, we also
calculated the weighted averages of Precision, Recall, and
F1-Score to account for class imbalance, ensuring a more
balanced evaluation. Weighted averaging assigns importance
to each class based on its support (number of true instances).
The formulas used for these metrics are standard and widely
accepted in classification tasks.
Formulae used:
Precision: TP / (TP + FP)
Recall: TP / (TP + FN)
F1-Score: 2 * Precision * Recall / (Precision + Recall)
Accuracy: TP + TN / TP + TN + FP + FN
The summary of Accuracy, Precision, Recall, and F1-
Score matrices found from applied Machine Learning
classification algorithms and NLP techniques. Where LSTM
provides the highest accuracy of 91%.
Table I. Result of Random Forest and XGBoost
Algorithms
Precision Recall F1- Support
Score
Negative 0.84 0.98 0.90 3697
Neutral 0.85 0.57 0.68 1204 Various Machine Learning Based Logistic Regression 74.10%
Social Media Sentiment Analysis Random Forest 70.90%
Positive 0.89 0.69 0.78 917 and Opinion Mining Approaches
(2020) [10]
Accuracy 0.85 5818 A Study on The Performance of Support Vector Machine 66.59%
Above table showcases the summary of the analysis Supervised Algorithms for Random Forest 49.67%
Classification in Sentiment
conducted by Random Forest and the XGBoost Algorithms Analysis (2019) [11]
Table II. Result Analysis of LSTM Algorithm Sentiment Analysis of Arabic and Multinomial Naive Bayes 70.00%
English Tweets (2019) [12] Logistic Regression 74.00%
Accurac Loss Validatio Validation
y n Loss A Comparative Study of Support Vector Machine 77.00%
Sentiment Analysis Using NLP Logistic Regression 77.00%
Accuracy and Different Machine Learning
Techniques on US Airline Twitter
Epoch 0.68 0.75 0.83 0.43 Data [13]
1
The approaches of this paper Random Forest and XGBoost 85%
Epoch 0.86 0.37 0.87 0.35 LSTM 91%
2
Epoch 0.91 0.24 0.89 0.32
3 The above table in graphical format:
Epoch 0.94 0.17 0.91 0.31
4
Epoch 0.95 0.13 0.91 0.30
5
Above table showcases the summary of the analysis
conducted by LSTM Algorithm
Figure 3. Comparative analysis of Sentiment Analysis
Algorithms
V. CONCLUSION
In this paper, we have implemented various Machine
Learning classification algorithms and NLP techniques on a
large, imbalanced, multi-classed, and real-world dataset to
analyse sentiment. Our best approaches provide 91%
accuracy with the LSTM (Long Short-Term Memory)
algorithm. In the future, we would like to apply more
Figure 2. Comparison with base paper advanced techniques to increase accuracy and will also try to
build a generalized and robust model for similar datasets.
Above graph compares findings and accuracy with [13]
Alongside LSTM, we experimented with models such as
Table III. Comparative Study of different Sentiment Naive Bayes, SVM, and Random Forest, comparing their
Analysis Algorithms performances using evaluation metrics like Accuracy,
Precision, Recall, and F1-Score. Preprocessing techniques
Title and Year Algorithm and Accuracy such as tokenization, stop-word removal, and vectorization
Sentiment Analysis Using Naive Support Vector Machine 63.99% using TF-IDF and word embeddings were also applied to
Bayes Algorithm of The Data enhance model performance. We also handled data
Crawler: Twitter (2019) [5] imbalance using weighted metrics and sampling techniques.
Twitter Sentiments Analysis Support Vector Machine 74.60% Future directions include experimenting with transformer-
Using Machine Learning Methods based models like BERT, integrating sentiment lexicons, and
(2020) [6] exploring ensemble approaches. These enhancements aim to
Sentiment Analysis for Airline Support Vector Machine 74.24% improve model generalization, handle noisy real-world data
Tweets Utilizing Machine more effectively, and achieve better scalability across various
Learning Techniques (2021) [7]
domains.
An Efficient Approach for Support Vector Machine 68.00%
Sentiment Analysis Using
Machine Learning Algorithm
(2020) [8] REFERENCES
Collaborative Classification Support Vector Machine 65.59% [1] P. Suchanek and M. Kr ´ alov ´ a, “Effect of customer satisfaction on ´
Approach for Airline Tweets Logistic Regression 77.42% company performance,” Acta Univ. Agric. Silvic. Mendel. Brun., vol.
Using Sentiment Analysis (2021) Random Forest 75.29% 63, no. 3, pp. 1013–1021, 2015.
[9] [2] Safariena Ilias and Mohd Farid Shamsudin, “Customer Satisfaction
and Business Growth”, JUSST, vol. 2, no. 2, 2020.
A Comparative Analysis of Support Vector Machine 50.00%
[3] D. M. A. Baker, “Service quality and customer satisfaction in the [10] K. Jayamalini, M. Ponnavaikko, and J. Kothandan, “A comparative
airline industry: A comparison between legacy airlines and low-cost analysis of various machine learning based social media sentiment
airlines,” Am. J. Tour. Res., vol. 2, no. 1, 2013. analysis and opinion mining approaches,” Adv. Math., Sci. J., vol. 9,
[4] F. Alattar and K. Shaalan, ”Using Artificial Intelligence to no. 11, pp. 10195–10209, 2020.
Understand What Causes Sentiment Changes on Social Media,” in [11] P. B. Sunitha, S. Joseph, and P. V. Akhil, “A study on the
IEEE Access, vol. 9, pp. 61756-61767, 2021. performance of supervised algorithms for classification in sentiment
[5] M. Wongkar and A. Angdresey, “Sentiment analysis using naive analysis,” in TENCON 2019 - 2019 IEEE Region 10 Conference
Bayes algorithm of the data crawler: Twitter,” in 2019 Fourth (TENCON), 2019.
International Conference on Informatics and Computing (ICIC), 2019, [12] M. K. Elhadad, K. F. Li, and F. Gebali, “Sentiment analysis of Arabic
pp. 1–5. and English tweets,” in Advances in Intelligent Systems and
[6] L. Mandloi and R. Patel, ”Twitter Sentiments Analysis Using Computing, Cham: Springer International Publishing, 2019, pp. 334–
Machine Learninig Methods,” 2020 International Conference for 348.
Emerging Technology (INCET), 2020, pp. 1-5. [13] Md. Taufiqul Haque Khan Tusar and Md. Touhidul Islam “A
[7] G. Ravi Kumar, K. Venkata Sheshanna, and G. Anjan Babu, Comparative Study of Sentiment Analysis Using NLP and Different
“Sentiment analysis for airline tweets utilizing machine learning Machine Learning Techniques on US Airline Twitter Data” in
techniques,” in International Conference on Mobile Computing and ICECIT conference
Sustainable Informatics, Cham: Springer International Publishing, [14] Kaggle US Airlines Dataset,
2021, pp. 791–799. [Link]
[8] A. Naresh and P. Venkata Krishna, “An efficient approach for [15] Ahamad, R., Mishra, K.N. “Exploring sentiment analysis in
sentiment analysis using machine learning algorithm,” Evol. Intell., handwritten and E-text documents using advanced machine learning
vol. 14, no. 2, pp. 725–731, 2021. techniques: a novel approach.” J Big Data 12, 11 (2025).
[9] M. V. K. [Link], “Collaborative classification approach for airline [Link]
tweets using sentiment analysis,” Turk. J. Comput. Math. Educ.
(TURCOMAT), vol. 12, no. 3, pp. 3597–3603, 2021.