Abstract

Sentiment Analysis know as a field of artificial intelligence which is subfield consists of

Natural Language Processing. It has become an important aspect/technique because it is
concerned with the emotions and tones of reviews and post, classifying them into neutral,
positive or negative. In the highly competitive usage of social media platforms, sentiment of
user has an effective role in increasing the use of these applications by providing targeted
content. Traditional methods used in the analysis often provide less accurate results of user
views, meaning it requires a more optimized technique. This research is basically a need for
understanding the sentiments like neutral, positive or negative of user across the platform of
twitter, providing the marketing companies to make decisions informed helping in the
promotion of product, improvement in the market competitions and much more. However,
this analysis has several challenges when it come to use the Natural Language Processing due
to complexities in the dataset. To address all these, we are developing a model which use the
advanced transformer-based models like BERT and RoBERTa focusing on the betterment of
the sentiment analysis using the pre-processed data and analyzing the factors of accuracy of
an algorithm. Therefore, we can assess the research project and utilize it in the practical
world.
Keywords: Sentiment Analysis, Natural Language Processing, Twitter, Neutral, Positive,
Negative, BERT, RoBERTa
Aims
The aims of this research are divided into three sections; firstly, it is to create and enhance the
Natural Language Processing algorithm which is responsible for the tweets on which
sentiment analysis will be applied with the help of advanced transformers models that will
provide efficiency for the sentiments to be applied on the tweets of Twitter. Secondly, figure
out the requirements to develop advanced transformer models, significantly the dynamic
fused models for effectively capturing the sentiments which are expressed in the tweets. And
in last, analyzing the integration of transformer-based architectures containing the contextual
embeddings and representation of sentiments (neutral, positive or negative) being determined
in the twitter tweets.
Objectives
Stated are the objectives, required to complete the proposed research:
 Investigating the advanced transformer-based models, when applied on the sentiments
of twitter tweets.
 Selecting a suitable architecture of transformer-based techniques like BERT,
ReBERTa and GPT.
 Gathering the Dataset and applying the process of cleaning.
 Taking the pre-processed dataset and implementing the architecture to be used for the
sentiment analysis.
 Training the model on the dataset and retesting it, to get the accurate result for
enhancing the sentiment analysis.
 Exploring the factors and attributes which are contributing in effectiveness of results
of sentiments that are expressed in Twitter tweets.
  Comparing the results and compiling the factors to determine how semantic
representations improve the process of emotion categories (neutral, positive or
negative) present in the tweets of twitter.

Justification
The proposed research addresses the growing effects and need for accurate sentiment analysis
on the Twitter platform, which we know is a platform where people provide their posts and
comments containing their concerns, opinions and emotions in vast amount. With the help of
advanced transformer-based natural language processing techniques, we can effectively
improve the sentiment analysis models. Now a days, every social media platform is providing
content on the bases of user attraction with the help of their content and connections, in
which twitter holds an important role. Twitter contains several unique characteristics, such as
informal language, limitations in length of text, which pose a challenge for the traditional
methods of sentiment analysis, making it difficult in improving the techniques. With the help
of this analysis, several factors can provide positive information for several applications,
including opinions of public, market research and management of perception of brands. The
proposed research not only contributes in advancing the natural language processing field but
also provide effective solutions for the social media platforms.
Literature Review
The project's literature review delivers a technique to apply similar techniques or related
sectors in order to apply past finds and experiences in order to reach the desired outcomes in
a good and efficient approach. Such data additionally increases the probability of
achievement of the objectives and aims.

Sentiment Analysis Background

Three types are used to review research on sentiment analysis: hybrid, lexicon-based, and
machine-learning techniques. Language characteristics and well-known ML algorithms are
used in machine learning (ML) techniques. Sentiment dictionaries are the foundation of
dictionary-based and corpus-based approaches (LB) that use statistical or semantic techniques
to determine sentiment polarity. In emotion dictionaries, hybrid approaches that combine the
two approaches are frequently employed and are crucial to the majority of methods.

A study that used a variety of techniques, such as Naive Bayes (NB) and SVM, provided an
answer to the Arabic text categorization based on comments obtained from Saudi Arabian
tweets in particular. The study looks on binary classes that are positive and negative. The
results of the study show that although the authors used two feature extraction methods, NB
with TF-IDF is 80% and with BTO is 80%, they also used the terms in Inverse Document
Frequency (TF-IDF) and Binary-Term Occurrence (BTO). SVM is 88% when using TF-IDF
and 87% when using BTO, respectively.

(H. Al-Rubaiee, R. Qiu and D. Li, "Identifying Mubasher software products through sentiment analysis
of Arabic tweets," 2016 International Conference on Industrial Informatics and Computer Systems
(CIICS), Sharjah, United Arab Emirates, 2016, pp. 1-6, doi: 10.1109/ICCSII.2016.7462396)
In a parallel study, the effectiveness of three classifiers-LR, KNN, and DT—for SA of tweets
in Arabic was investigated. In the course of the research, 2 extraction of features methods
were employed: TF-IDF and Binary-Term Occurrence (BTO). However, they only
considered both positive and negative classifications. The study's findings demonstrate that
decision-tree can attain up to 90% accuracy.

N. K. Bolbol and A. Y. Maghari, "Sentiment Analysis of Arabic Tweets Using Supervised Machine
Learning," 2020 International Conference on Promising Electronic Technologies (ICPET), Jerusalem,
Palestine, 2020, pp. 89-93, doi: 10.1109/ICPET51420.2020.00025.

In a different study, the primary goal was to examine Amazon evaluations of electronic
devices and create a predictive model that could effectively categorise reviews as either good
or negative. Three well-liked text pre-processing methods were compared by the researchers
during the study: Word2Vec, Bag of Words, and Term Frequency-Inverse Document
Frequency. Using techniques like SVM, DT, RF, Naïve Bayes, and Multi-Layer Perceptron,
they additionally discovered the best predictive model. With an accuracy of 90%, the MLP
algorithm generated the best categorization results in the majority of situations.

M. Hawlader, A. Ghosh, Z. K. Raad, W. A. Chowdhury, M. S. H. Shehan and F. B. Ashraf, "Amazon

Product Reviews: Sentiment Analysis Using Supervised Learning Algorithms," 2021 International
Conference on Electronics, Communications and Information Technology (ICECIT), Khulna,
Bangladesh, 2021, pp. 1-6, doi: 10.1109/ICECIT54077.2021.9641243.

This study's primary goal was to assess patron perceptions of cafés and restaurants in Saudi
Arabia's Qassim region. The breakdown of sentiment categories, such as both positive and
negative, was the primary focus of the study, which used the TF-IDF extraction of features
approach. When these techniques were compared to five different classifiers (Logistic
Regression, Support Vector Machine, Naïve Bayes, K-Nearest Neighbours, and Random
Forest), SVM outperformed the other classifiers with the greatest accuracy values (90%).

L. M. Alharbi and A. M. Qamar, "Arabic Sentiment Analysis of Eateries’ Reviews: Qassim region Case
study," 2021 National Computing Colleges Conference (NCCC), Taif, Saudi Arabia, 2021, pp. 1-6,
doi: 10.1109/NCCC49330.2021.9428788.

Transformer Techniques for Sentiment Analysis

Transformers are known as Deep neural networks which identify contextual linkages in
sequential data by using a self-attention process. Transformer models are superior to
conventional neural networks and recurrent neural network versions like LSTM (Long Short-
Term Memory) in that they can handle long-term dependencies between input sequence
pieces and enable parallel processing.
Saidul Islam, Hanae Elmekki, Ahmed Elsebai, Jamal Bentahar, Nagat Drawel, Gaith Rjoub,
Witold Pedrycz,
A comprehensive survey on applications of transformers for deep learning tasks,
Expert Systems with Applications,
Volume 241,
2024,
122666,
ISSN 0957-4174, As such, researchers studying artificial intelligence have given
Transformer-based models a lot of attention. This is because of their extraordinary potential
and outstanding achievements, which go beyond NLP tasks to cover a variety of fields,
including as artificial intelligence, as well as the growing Internet of Things.

Transformer-based architectures have made it possible to represent the query and the
document in low-dimensional density vector spaces for the context visualisation of text data.
With the help of these vectors, which are acquired as insertions of fixed sizes, text
comprehension is improved. In this work, we combined the more advanced text recognising
features of the transformer-based BERT, as the model with an expression embedding-based
query extension model to develop a pipeline for efficiently obtaining documents from a broad
search field.
Amol P. Bhopale, Ashish Tiwari,
Transformer based contextual text representation framework for intelligent information
retrieval,
Expert Systems with Applications,
Volume 238, Part C,
2024,
121629,
ISSN 0957-4174,
[Link] independently encoded the query and the
document to fine-tune a deep semantic matching model and learn the contextual
representations. The BERT architecture, that independently creates complex vector
representations for documents and queries, serves as the foundation for the encoder model.

Traditional Approaches for Sentiment Analysis

A study found that, we could accurately predict sentiments, we might extract views from the
web and determine the preferences of online customers. These insights could be useful for
marketing or economic analysis, for leveraging a company's strategic advantage, or for
identifying security threats and cyber risk. We provide an intuitive search-enhanced Markov
blanket framework that can effectively capture word dependencies and offer a suitable
vocabulary for sentiment extraction. Based on computations performed on two groups, our
method outperforms several cutting-edge feature selection and sentiment prediction
algorithms in terms of yielding comparable or superior predictions about sentiment
orientations while also being able to find a restricting set of predictive features.

T. Parlar, E. Saraç and S. A. Özel, "Comparison of feature selection methods for sentiment analysis
on Turkish Twitter data," 2017 25th Signal Processing and Communications Applications Conference
(SIU), Antalya, Turkey, 2017, pp. 1-4, doi: 10.1109/SIU.2017.7960388.
keywords: {Twitter;Sentiment analysis;Data mining;Ant colony optimization;Micromechanical
devices;Support vector machines;sentiment analysis;feature selection;text classification},

Words are typically treated as discrete, atomic symbols in NLP systems. The model can make
use of sparse information about the relationships between the individual symbols. In this
article, we propose ConvLstm, a neural network architecture based on pre-trained word
vectors that combines Long Short-Term Memory (LSTM) and Convolutional Neural
Network (CNN). In our tests, ConvLstm uses long short-term dependencies (LSTM) to
capture long-term dependencies in a phrase sequence and lessen the reduction of specific
local information in CNN's pooling layer. We test the suggested model using the Stanford
Sentiment Treebank (SSTb) and the IMDB sentiment datasets. Empirical findings
demonstrate that ConvLstm performed comparably on sentiment evaluation tasks with less
parameters.

Shehu, H.A., Tokat, S. (2020). A Hybrid Approach for the Sentiment Analysis of
Turkish Twitter Data. In: Hemanth, D., Kose, U. (eds) Artificial Intelligence and
Applied Mathematics in Engineering Problems. ICAIAME 2019. Lecture Notes on
Data Engineering and Communications Technologies, vol 43. Springer, Cham.
[Link]

Using document classification techniques, Turkish Twitter feeds that were obtained via the
Twitter API were examined in this study for positive or negative sentiment contexts. Machine
learning algorithms like SVM, the Naive Bayes algorithm, Multinomial Naive Bayes, and
KNN have all been the subject of experiments. The Bag of Words and N-Gram models are
two distinct models from which the characteristics expressed in vector space are taken. The
impact of classification techniques has been examined in relation to the experimental
outcomes.

G. M. Demirci, Ş. R. Keskin and G. Doğan, "Sentiment Analysis in Turkish with Deep Learning," 2019
IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA, 2019, pp. 2215-2221,
doi: 10.1109/BigData47090.2019.9006066.

Sentiment Analysis Used for twitter

Feature selection techniques that aid in identifying most valuable features are typically used
to improve classification performance. In this study, we use the Maximum Entropi Modelling
classification technique to analyse the efficacy of four feature selection methods: Chi-square,
Gain of Information, Query Expansion Ranking, and Ant Colonies Optimisation using
Turkish Twitter dataset. As a result, the impact of feature selection techniques on Turkish
Twitter data sentiment analysis performance is assessed. According to experimental data,
alternative conventional feature selection techniques for sentiment analysis are not as
effective as Query Expansion Ranking and Ant Colony Optimisation.
R. Velioğlu, T. Yıldız and S. Yıldırım, "Sentiment Analysis Using Learning Approaches Over Emojis
for Turkish Tweets," 2018 3rd International Conference on Computer Science and Engineering
(UBMK), Sarajevo, Bosnia and Herzegovina, 2018, pp. 303-307, doi: 10.1109/UBMK.2018.8566260.

To represent tweets, the bag-of-words approach is first used as a basic and effective
technique. Classifiers like Logistic Regression, Machine Learning, Naive Bayes, Support
Vector Machines, and Decision Trees are then applied to these tweets. Second, for the
sentiment analysis task, we represented tweets into word n-grams using fast Text. The
outcomes demonstrate that there isn't a discernible difference between both models. For the
classification of binary data, fast Text achieves 78% and a linear regression classification
obtains 75% F1-score; however, for classification with multiple classes, fast Text achieves
60% and Linear Regression obtains 56% F1-score.
M. Meral and B. Diri, "Sentiment analysis on Twitter," 2014 22nd Signal Processing and
Communications Applications Conference (SIU), Trabzon, Turkey, 2014, pp. 690-693, doi:
10.1109/SIU.2014.6830323.

The unprocessed and disorganised raw data from social media platforms cannot be used to
produce sufficient outcomes. A sentiment assessment has been carried out in this study by
gathering information from Twitter. In order to do this analysis, an intelligent system was
built utilising machine learning techniques like Support Vector Machine, Random Forest, and
Naïve Bayes, and the results were compared.
Ariel Hasell (2021) Shared Emotion: The Social Amplification of Partisan News on
Twitter, Digital Journalism, 9:8, 1085-
1102, DOI: 10.1080/21670811.2020.1831937

In order to determine what kind of political news is most frequently shared on the internet
and how emotions in news content can affect political news sharing, this study looks at over
278,000 tweets as well as retweets across 20 news organisations. The findings imply that, in
comparison to non-partisan news, partisan news sources are shared on Twitter more
frequently and that their material is able to convey emotion. In general, social media
amplifies partisan and emotive news media content disproportionately, which could have
serious repercussions for those who depend upon it for political news information.
S. A. El Rahman, F. A. AlOtaibi and W. A. AlShehri, "Sentiment Analysis of Twitter Data," 2019
International Conference on Computer and Information Sciences (ICCIS), Sakaka, Saudi Arabia,
2019, pp. 1-4, doi: 10.1109/ICCISci.2019.8716464.

ISSN 1877-0509,

Hybrid Approaches for sentiment analysis:

Deep learning has improved to the point where advanced models based on deep learning can
perform better in sentiment analysis by combining auxiliary knowledge. Based on masking
language models and transformers structures, BERT is a strong pre-trained model of
language that generates deep bidirectional encoder visualisations of characters.

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019, May 24). Bert: Pre-training of
deep bidirectional Transformers for language understanding.

It has demonstrated outstanding sentiment analysis performance by using a lexicalized area

ontology for collecting domain knowledge and BERT for word embedding. When combined
with the neural attention model design, this results in a hybrid sentiment analysis solution.

Marco Pota, Mirko Ventura, Hamido Fujita, Massimo Esposito,

Multilingual evaluation of pre-processing for BERT-based sentiment analysis of tweets,
Expert Systems with Applications,
Volume 181,
2021,
115119,
ISSN 0957-4174,
[Link]
The particular project of sentiment analysis of twitter is examined in this research. Several of
the most sophisticated techniques currently in use require specifically pre-training BERT on
useful corpora, especially when it comes to sentiment analysis of tweets in various languages.
Though it is still up for debate, pre-training BERT-based models on large, well-formed text
corpora has proven to be effective in demonstrating the potential of these models. On the one
hand, it is evident that a uniquely specialised technique is required to analyse tweets.

The vote algorithm has been applied to three classifiers: SVM, Bagging, and Naive Bayes.
The SVM's parameters have been optimised for usage as a standalone classifier. The
outcomes of the experiments shown that the effectiveness of these multi classifier systems is
increased by meta classifiers, and on sentiment analysis datasets, the accuracy of individual
classifiers is improved by multiple classifier systems.
Parlar, T., Özel, S.A. & Song, F. QER: a new feature selection method for
sentiment analysis. Hum. Cent. Comput. Inf. Sci. 8, 10 (2018).
[Link]

The suggested method outperformed Support Vector Machines and Naive Bayes, which was
said to be the most effective particular classifier for these datasets.

Neural networks are designed to overcome for the absence of training data through the
incorporation of external knowledge. Our method identifies aspect-opinion pairs and
determines their sentiment orientations by using context features that are taken from
evaluation sentences and additional information that is obtained from a sentiment knowledge
network. This allows our model to operate better with less training corpora and produce more
thorough sentiment analysis results.
Fang Chen, Yongfeng Huang,
Knowledge-enhanced neural networks for sentiment analysis of Chinese reviews,
Neurocomputing,
Volume 368,
2019,
Pages 51-58,
ISSN 0925-2312,
[Link] We test our method on a dataset of Chinese
vehicle ratings. The knowledge-enhanced artificial neural networks frequently outperform the
traditional models, according to experimental results.

Two datasets, one for the first and one for the second testing phase, have been
identified: 2500 and 1100 from the derived data from each class having an equal distribution.
Based on experimental findings, support vector machines are more effective in categorising
negative and unbiased stemmed data than random forests algorithm in categorising positive
stemmed data. Consequently, a hybrid approach combining random forests and support
vector machines in an ordered manner has been developed and applied to determine the data's
outcome. Lastly, the first and second datasets have been used to test the implemented
procedures. It has been noted that the created hybrid approach achieves a precision of as
much as 85% and 80% on both the first and second dataset, respectively, while both random
forest algorithms and support vector machines could not obtain an efficiency of up to 75% on
that first and 70% on the second dataset.

Cagatay Catal, Mehmet Nangir,

A sentiment classification model based on multiple classifiers,
Applied Soft Computing,
Volume 50,
2017,
Pages 135-141,
ISSN 1568-4946,
[Link]
([Link]

Methodology
The term section methodology relates to the collection of tasks needed to fulfil the objectives
of the research study. These processes comprise data collection, dataset collection, pre-
processing, research evaluation, and algorithm architecture creation.

Dataset Collection and Description

In the proposed research, data gathering in an important and crucial task, because it
comprises of stages to gather the dataset which is most suitable to tackle the problem. Data
Collection has several outcomes, verification of research, making better decisions and
making sure that the findings are relevant.
In our research, we used the dataset which was focused on detecting the cyberbullying with
the help of advanced techniques, also with the datasets of tweets that is publicly available.
The dataset we are using is comprised of values that are Unique which is 46,017, data which
is related to religion is 17%, when it relates to age it is about 17% also and in the end the data
which comes under the category of Other is equal to 60% which is around 31,702 data
instances.
Bert:
Unlike standard language models that employ a unidirectional method (whether it's from
right to left or right to left), BERT utilises a bidirectional technique that allows it to fully
comprehend the multiple meanings of words in their specific settings. BERT receives
extensive pre-training on a vast amount of unstructured textual data. This model can then be
optimised for specific Natural Language Processing (NLP) tasks by leveraging its
fundamental unsupervised learning-acquired language understanding using smaller labelled
datasets.
Sentiment analysis, recognised entities, and question answering are just a few of the natural
language processing techniques that have been revolutionised by BERT's remarkable
precision in language analysis. Its range of applications may be used for a variety of tasks,
including chatbots, virtual assistants, and machine translation. BERT is a very useful tool for
NLP research and applications because of its capacity to comprehend phrase contexts and
carry out a variety of activities.
By simultaneously analysing each side of the situation at the same time, BERT has the ability
to capture a word's whole meaning in its context, unlike prior models that only considered the
left or right context of the word. As a result, BERT is capable of managing intricate and
perplexing linguistic phenomena like co-reference, multilingualism, and long-distance
partnerships.
RoBERTa:
An effective deep learning model in natural language processing (NLP) is called RoBERTa.
based upon the BERT, RoBERTa is a development from Facebook AI Research (FAIR).
However, the reason this ground-breaking model stands out is that it has gone through several
thoughtful modifications and enhancements that have further boosted its performance in a
variety of NLP tasks.
The initial step in utilising RoBERTa for sentiment analysis of Twitter data is to gather a
labelled dataset of tweets. The learned RoBERTa model has been loaded and tweaked
following the preprocessing of the data to accurately identify sentiment characteristics in
Twitter language. The model undergoes training and validation tweaking in order to optimise
its performance. New, unlabeled tweets are submitted to sentiment analysis utilising the
trained RoBERTa model after testing on a test set to ensure accurate sentiment predictions.
Post-processing techniques can be used to visualise the data and show sentiment trends over
time.
In order to enhance the model's capacity to represent the nuances of sentiment as expressed in
tweets, it is essential to tackle problems unique to Twitter language, like slang and emoji. The
representativeness and quality of the training data, together with the fine-tuning process, are
essential to the success of sentiment analysis using RoBERTa.
Workflow
Capable outcomes have been obtained when BERT and RoBERTa are used for twitter
sentiment analysis. For the analysis of sentiment across each of the classes (neutral, positive
and negative), these models have demonstrated good accuracy, precision, recall, and F1-
score. Scientists are able to learn more about the public's views on a variety of issues, such as
political thought, health care, and social problems, thanks to Twitter data. Transformer-based
models are helpful for sentiment analysis of twitter tweets since they can capture semantics
and context, such as BERT and RoBERTa. Irony, humour, and sarcasm are examples of
subtle tweet language that are frequently observed in internet data. The link between context
and semantics has been identified by BERT and RoBERTa.
Workplan
To complete the project, time of fifteen weeks is provided, and to organize everything in the
given workplan is helpful. The work plan which follows lists the tasks accomplished as well
as the span of time required to finish each within the allocated period. An overview of the
workplan which will be carried out is provided below.
 Research and Planning: Planning and research in a project is an important step which
includes defining the scope of project, conducting review of literature and with the
help of all these findings selecting the appropriate technique.
 Collection of Dataset and its Pre-processing: As mentioned above it is also a crucial
step in research, which helps in selection of Dataset, cleaning the data and doing
analysis of data with the help organizing it.
 Model Architecture and Training: In this step, the transformer architecture will be
used in training, testing and doing validation, pre-processed dataset will be used in the
training process.
 Model Training and Testing Fine tuning the models of transformer techniques,
evaluating and analyzing the results of model and document it.
 Model Fine Tuning: It is used to fine tune the results of validation, by optimizing the
hyper-parameters and testing it again with the unseen dataset.
 Finalization of Thesis: To prove all the findings, it is documented in a presented way
including all the findings and results gathered in the research project.

Planning and Research

Defining Project Scope Done

Conducting Literature Review Done Week 1

Selecting Appropriate
Done
Algorithms

Dataset Collection and Processing

Selecting Sentiments Type on

Done
Twitter

Cleaning Dataset Done

Week 2 - 4
Data Analysis Done

Organizing Dataset Done

Model Architecture and Tuning

Defining Transformer Done Week 5 - 7

Architecture
 Splitting of Train/Test/Valid
Done
Splitting

Train model on pre-processed

Done
dataset

Model Training and Testing

Fine Tuning Transformers

Pending
Model

Training and Evaluting Model

Pending Week 8 - 9
on Dataset

Documenting Pending

Model Fine Tuning

Fine Tuning Model on

Pending
Validation Results

Optimizing Model Pending Week 10 – 12

Test Model on unseen Dataset Pending

Thesis Finalization

Finalizing Thesis Pending

Week 15
Presentation Pending