INTELLIGENT RECOMMENDATION

SYSTEM USING TRANSFORMER

NETWORK FOR SENTIMENTS IN E-COMMERCE

A PROJECT REPORT

Submitted by

DILIPKUMAR. S (731619205013)

KRITHICK SANJAY. K (731619205027)

MANOJKUMAR. E (731619205031)

SURYA. S. K (731619205052)

In partial fulfillment for the award of the degree

of

BACHELOR OF TECHNOLOGY
IN
INFORMATION TECHNOLOGY

K S R INSTITUTE FOR ENGINEERING AND TECHNOLOGY,

TIRUCHENGODE

ANNA UNIVERSITY: CHENNAI 600 025

MAY 2023
 ANNA UNIVERSITY: CHENNAI 600 025

BONAFIDE CERTIFICATE

Certified that project report “INTELLIGENT RECOMMENDATION

SYSTEM USING TRANSFORMER NETWORK FOR SENTIMENTS IN
E-COMMERCE” is the bonafide work of “DILIPKUMAR S (731619205013),
KRITHICK SANJAY K (731619205027), MANOJKUMAR E
(731619205031) and SURYA S K (731619205052)” who carried out the project
work under my supervision.

SIGNATURE SIGNATURE
Dr. L. SELVAM, M.E., Ph.D., Mr. N. KANNAN, M.E.,
HEAD OF THE DEPARTMENT, SUPERVISOR,
Department of Information Technology, Assistant Professor,
K S R Institute for Engineering and Department of Information Technology,
Technology, K S R Institute for Engineering and
Tiruchengode - 637215. Technology,
Tiruchengode - 637215.

Submitted for the Project Viva-Voce examination held on _________________

Internal Examiner External Examiner

 ACKNOWLEDGEMENT
Behind every achievement lays an unfathomable sea of gratitude to those
who actuated it, without them it would never have come into existence. To them
we lay the word of gratitude imprinted within ourselves.

We wish our heartfelt thanks to our respected Founder of K S R group of

Institutions Theivathiru Lion Dr. K. S. RANGASAMY, MJF and the
Chairman of K S R Group of Institutions Mr. R. SRINIVASAN, B.B.M.,
M.I.S.T.E., for giving as an opportunity and facilities for the completion of this
project in this institution.

We are expressing our extreme gratitude and heartfelt thanks to the

respected Principal Dr. M. VENKATESAN, M.E., Ph.D., for allowing us to
have the extensive use of the college facilities to do this project effectively.

We would like to express our profound interest and sincere gratitude to our
respected Director - Academics Dr. P. MEENAKSHI DEVI, M.E., Ph.D., for
her valuable guidance and constant support in successful completion of the
project.

We would like to express our profound interest and sincere gratitude to our
respected Head of the Department Dr. L. SELVAM, M.E., Ph.D., for his
valuable guidance and constant support in successful completion of the project.

We feel proud and honor in extending our sincere thanks to our project guide
Mr. N. KANNAN, M.E., for being more informative and providing guidelines
regarding this project all the time.

We would like to extend thanks to our Project Coordinator Mr. P. RAVI,

M.E., for being more informative and providing guidelines regarding this
project all the time. Finally, we sincerely thank our staff members, parents,
friends and well-wishers for their moral support.
 ABSTRACT

Online shopping has flourished as a result of the quick growth of Internet

technology as a method of placing orders for items and using other services.
Finding a product that suits a consumer's wants and preferences can be difficult
as a result. Example: Flipkart, Amazon People are using online shopping more
frequently to purchase and consume items. Deep learning methods, notably
transformer networks, have demonstrated encouraging results in recent years for
problems involving natural language processing, such as sentiment analysis. In
recent years, sentiment analysis has become increasingly popular with
businesses, governments, and organizations in addition to scholars. It involves
acquiring and examining consumer opinions, ideas, and first impressions of a
range of goods, services, and topics. By the corporations, governments, and
people can all benefit from public opinion when gathering data and making
judgment calls. Textual data are categorized into categories of positive, negative,
or neutral sentiment as part of sentiment analysis. E-commerce businesses can
learn a great deal about client preferences by examining customer evaluations
and feedback, and they can then adjust their goods and services accordingly.
Customers will receive personalized product recommendations from the system
once it has analyzed customer reviews and feedback to categorize them according
to sentiment. To capture the intricate relationships between words and phrases in
customer evaluations, the system will use transformer networks, which will
increase the precision of sentiment analysis and suggestion predictions.

iv
 TABLE OF CONTENTS
CHAPTER PAGE
TITLE
NO. NO.

ABSTRACT IV
LIST OF FIGURES V
LIST OF ABBREVIATIONS VI
1 INTRODUCTION 1
1.1 INTRODUCTION 1
1.2 E-COMMERCE REVIEW SYSTEM 2
1.3 SENTIMENT ANALYSIS 3
2 LITERATURE REVIEW 5
2.1 LITERATURE REVIEW 5
3 SYSTEM ANALYSIS 10
3.1 EXISTING SYSTEM 10
3.1.1 Limitations of the Existing System 11
3.2 PROPOSED SYSTEM 11
3.2.1 Advantages of the Proposed System 12
4 SYSTEM REQUIREMENT 13
4.1 SOFTWARE REQUIREMENT 13
5 SOFTWARE DESCRIPTION 14
5.1 PYTHON 14
5.2 LIBRARIES 15
5.2.1 NumPy 15
5.2.2 Pandas 15
5.2.3 Matplotlib 15
5.2.4 Tensorflow 16
5.2.5 Scikit - Learn 16
 5.2.6 TQDM 16
5.2.7 PyTorch 17
5.2.8 Transformers 17
5.3 GOOGLE COLAB 17
6 SYSTEM DESIGN AND METHODOLOGY 20
6.1 PROJECT DESCRIPTION 20
6.2 BLOCK DIAGRAM 21
6.3 DATASET DESCRIPTION 23
7 PERFORMANCE ANALYSIS 25
8 CONCLUSION AND DISCUSSIONS 29
9 FUTURE ENHANCEMENT 30
APPENDIX - Source Code 31
REFERENCES 38
 LIST OF FIGURES

FIGURE NO FIGURE NAME PAGE NO

6.1 Block Diagram of Transformer 21

6.2 Review Dataset 24
7.l (a) F1-score calculation of CNN-LSTM 26
7.1 (b) Graphical representation of CNN-LSTM 26
model
7.2(a) F1-score calculation of BERT model 27
7.2(b) Graphical representation of BERT model 27
7.3 Bar Chart Represents the Comparison of the 27
CNN-LSTM and Bert model
7.4 Different Results on Each Iteration Based 28
on Training Data

vii
 LIST OF ABBREVIATION

BERT Bidirectional Encoder Representation from Transformers

CNN Convolution Neural Network

GIMP GNU Image Manipulation Program

LSTM Long Short-Term Memory

MLM Masked Language Model

NLP Natural Language Process

NSP Next Sentence Prediction

RNN Recurrent Neural Network

viii
 CHAPTER 1

INTRODUCTION

1.1 INTRODUCTION

Online shopping has become a very common way to make purchases and use
other services as a result of the quick growth of Internet technology. Finding the
ideal product that suits a customer's wants and tastes can sometimes be difficult.
For instance, Flipkart, Amazon As a method of purchasing and consuming things,
online shopping is gaining popularity. Deep learning methods, notably
transformer networks, have recently demonstrated promising results in natural
language processing tasks like sentiment analysis.

In recent years, sentiment analysis has become increasingly popular, not just
among researchers but also among businesses, governments, and organizations.
It is the process of compiling and evaluating the views, ideas, and perceptions of
the public on a wide range of issues, goods, and services. Corporations,
governments, and individuals can all benefit from the public's perspectives while
gathering data and making judgment calls [2].

A system that uses cutting-edge machine learning methods to offer tailored

product recommendations to customers in the e-commerce sector is known as an
Intelligent Recommendation System utilizing Transformer Network for
Sentiments. Built-in to the system is a Transformer Network, a potent deep
learning model that excels at tasks requiring natural language processing, such
sentiment analysis. Textual data are categorized into categories of positive,
negative, or neutral sentiment as part of sentiment analysis.

E-commerce businesses can learn a great deal about client preferences by

examining customer evaluations and feedback, and they can then adjust their
goods and services accordingly. This research suggests using an intelligent
recommendation system for transformer network-based e-commerce attitudes.
Bert, a deep learning model built on transformers, has demonstrated promising
performance in sentiment analysis and other natural language processing tasks.
Bert's ability to identify intricate word and phrase associations in textual material
improves the precision of sentiment analysis and suggestion forecasting.
Customers will receive personalized product recommendations from the system
once it has analyzed customer reviews and feedback to categorize them according
to sentiment.

To capture the intricate relationships between words and phrases in customer

evaluations, the system will use transformer networks, which will increase the
precision of sentiment analysis and suggestion predictions. The paper will give a
summary of the state-of-the-art in sentiment analysis and intelligent
recommendation systems, discuss the architecture and implementation specifics
of the proposed system, and present experimental findings proving the system's
efficacy.

1.2 E-COMMERCE REVIEW SYSTEM

Review systems were introduced to provide customers with a platform to

share their experiences and opinions about products, services, and businesses.
Reviews allow customers to express their satisfaction or dissatisfaction with a
product or service and share their experiences with others. Review systems also
enable businesses to receive feedback from their customers and make
improvements to their products or services based on customer input.

Review systems have become increasingly popular with the rise of e-

commerce and online shopping. Customers often rely on reviews to make
purchasing decisions, as they provide insight into the quality, reliability, and
value of products and services. Review systems also play a significant role in
building trust and credibility between businesses and customers.

2
 The introduction of review systems has revolutionized the way customers
shop and the way businesses operate. Reviews can greatly impact a business's
reputation and success, making it crucial for businesses to actively manage and
respond to customer feedback [17] . Review systems provide a way for businesses
to engage with their customers, gain insights into customer preferences and needs,
and improve their products and services accordingly.

1.3 SENTIMENT ANALYSIS

A subfield of natural language processing known as sentiment analysis or

opinion mining aims to identify and extract the sentiment or emotional tone
expressed in a given piece of text. Sentiment analysis has grown in importance in
a variety of fields, including marketing, finance, politics, healthcare, and the
exponential growth of user-generated content on social media, websites, and
blogs [18].

Classifying text into one of three categories is the primary goal of sentiment
analysis: neutral, negative, or positive This is finished by utilizing AI
calculations, measurable models, and dictionary-based approaches. Customers'
reviews, social media posts, news articles, and customer feedback forms are all
examples of large volumes of text data that can be subjected to sentiment analysis
to learn more about how people feel about a variety of subjects. Feeling
investigation has a large number of utilizations, including checking brand
notoriety, distinguishing client opinion towards items and administrations,
foreseeing buyer conduct, and breaking down the effect of promoting and
showcasing efforts. It can also be used in healthcare to look at how patients feel
about providers, treatments, and facilities for healthcare.

Sentiment analysis is useful, but it has some drawbacks. It battles to identify

mockery, incongruity, and different types of non-literal language, and may not
precisely catch complex feelings and social subtleties. The quality of the data, the

3
language used, and the context in which the text is written all affect sentiment
analysis's accuracy.

Overall, sentiment analysis is an important part of the operations of many

businesses and organizations because it is a powerful tool for understanding
human emotions and opinions on a large scale.

4
 CHAPTER 2

LITERATURE REVIEW

2.1 LITERATURE REVIEW

In the year 2021, "ABCDM: An attention-based bidirectional CNN-RNN

deep model for sentiment analysis" was published by Basiri, Nemati, Abdar,
Cambria, and Acharya presents a novel deep learning model for sentiment
analysis that uses bidirectional RNNs and attention mechanisms. The authors
provide a comprehensive overview of sentiment analysis and its limitations,
highlighting the need for more sophisticated models. The proposed model is
evaluated on several benchmark datasets, and the results show that it outperforms
several state-of-the-art models in terms of accuracy and F1 score. Overall, this
paper makes a significant contribution to the field of sentiment analysis and
presents a promising direction for future research [5].

As a chapter in the book "A Practical Guide to Sentiment Analysis," the paper
"Affective computing and sentiment analysis" was published by Cambria, Erik,
Dipankar Das, Sivaji Bandyopadhyay, and Antonio Feraco in 2017. An outline of
emotional registering and opinion examination, as well as their different fields of
utilization, is given in this paper. The concept of affective computing, which is
the study of how computers can recognize, interpret, and respond to human
emotions, is first presented by the authors. They then concentrate on sentiment
analysis, a subfield of affective computing in which opinions and feelings
expressed in text data are automatically identified. The paper goes over lexicon-
based, machine learning-based, and deep learning-based sentiment analysis
methods and their applications. Additionally, it demonstrates the difficulties of
sentiment analysis, such as dealing with sarcasm, irony, and other figurative
language [6].

5
 The paper "Comparative Analysis of Convolutional Neural Network and
LSTM in Text-Based Sentiment Classification" was published in 2021 in the
proceedings of the Fifth International Conference on I-SMAC. The paper presents
a comparative analysis of two popular deep learning models, Convolutional
Neural Network (CNN) and Long Short-Term Memory (LSTM), for text-based
sentiment classification. They propose to evaluate the performance of CNN and
LSTM models on a dataset of movie reviews, which consists of positive and
negative reviews. The paper presents a detailed description of both models and
their architectures. The authors then compare the performance of the two models
in terms of accuracy, precision, recall, and F1-score. The experimental results
show that the LSTM model outperforms the CNN model in terms of accuracy and
F1-score. The paper highlights the importance of selecting an appropriate model
for sentiment analysis, depending on the nature of the text data and the specific
task at hand [10].

The paper "Data analytics for the identification of fake reviews using
supervised learning" was published in 2022 by Alsubari, S. Nagi, Sachin N.
Deshmukh, A. Abdullah Alqarni, Nizar Alsharif, T. H. Aldhyani, Fawaz
Waselallah Alsaade, and Osamah I. Khalaf in the journal Computers, Materials
& Continua. The paper evaluates the proposed approach on two publicly available
datasets of reviews, and the experimental results show that the approach achieves
high accuracy and outperforms several baseline methods. Overall, the paper
provides a promising approach to identify fake reviews using data analytics and
supervised learning techniques. The results suggest that this approach can be
useful for maintaining the credibility of online review systems and improving the
shopping experience for consumers [3].

The paper "Developing an intelligent system with deep learning algorithms

for sentiment analysis of e-commerce product reviews" by Alzahrani et al. (2022)
presents a novel approach for sentiment analysis of e-commerce product reviews

6
using deep learning algorithms. The authors collected a dataset of e-commerce
product reviews and preprocessed the data using NLP techniques. They then
trained a Convolutional Neural Network (CNN) model on the preprocessed data
and achieved high accuracy in classifying the sentiment of product reviews. The
paper provides valuable insights into the field of sentiment analysis in e-
commerce and presents a promising approach for analyzing large amounts of
unstructured data [4].

In the year 2020, Petr Hajek, Aliaksandr Barushka, and Michal Munk
published an article in the journal Neural Computing and Applications titled
"Fake consumer review identification using deep neural networks incorporating
word embeddings and emotion mining." The research suggests a different
approach for dealing with the problem of identifying fake surveys by combining
word embeddings and sentiment mining techniques with in-depth brain
architecture. The authors contend that because fraudulent reviews have a
substantial impact on consumers' purchase decisions, their detection is crucial for
preserving trust in online review systems. The suggested method begins by
extracting textual features from the review text using word embeddings and
sentiment analysis techniques [8].

Li, Lin, Tiong-Thye Goh, and Dawei Jin published the article "How textual
quality of online reviews affect classification performance: a case of deep
learning sentiment analysis" in the Neural Computation and Applications journal
in 2020. The experimental findings demonstrate that the textual quality of the
reviews has a considerable impact on the model's performance. The model
performs best, in particular, on evaluations that have high linguistic quality and
high informative quality. The most important features that affect classification
performance are determined by the authors through a feature significance analysis
[13].

7
 Dong, Jian, Yu Chen, Aihua Gu, Jingwei Chen, Lili Li, Qinling Chen, Shujun
Li, and Qifeng Xun published the article "Potential Trend for Online Shopping
Data Based on the Linear Regression and Sentiment Analysis" in the journal
Mathematical Problems in Engineering in 2020. The article suggests a novel
method for trend prediction based on sentiment analysis and linear regression of
online purchasing data. Using a real-world dataset of online shopping data from
a Chinese e-commerce site, the study assesses the proposed technique. The
experimental results demonstrate that the suggested methodology outperforms
many baseline methods and is capable of accurately predicting the probable trend
of online purchasing data. Overall, the research offers a helpful method for
forecasting a prospective trend in online buying data by utilizing techniques for
linear regression and sentiment analysis [7].
Jagdale, Rajkumar S., Vishal S. Shirsat, and Sachin N. Deshmukh published
the paper "Sentiment analysis on product evaluations using machine learning
approaches" in the book "Cognitive Informatics and Soft Computing: Proceeding
of CISC 2017" in 2019. The research suggests a sentiment analysis method based
on machine learning for customer feedback. The paper offers a practical method
for conducting machine learning-based sentiment analysis on customer
evaluations. The findings imply that this strategy can help companies understand
the thoughts and preferences of their clients so that they can modify their goods
and services accordingly. For scholars and practitioners working on sentiment
analysis in the context of online reviews, the study offers helpful insights [9].

2019 saw the release of Alharbi and de Doncker's article, "Twitter sentiment
analysis using a deep neural network: An upgraded technique leveraging user
behavior information," in the journal Cognitive Systems Research. By combining
user behavioral data into a deep neural network model, the article suggests an
improved method for sentiment analysis on Twitter. The suggested model is
thoroughly described in the study, which also assesses how well it performs using

8
a dataset of tweets. The performance of their model is compared to a number of
baseline models, including a deep learning-based method and a typical machine
learning-based approach without user behavioral data. Researchers and
professionals working on sentiment analysis in the context of Twitter data may
find the suggested approach interesting [2].

9
 CHAPTER 3

SYSTEM ANALYSIS

3.1 EXISTING SYSTEM

The "existing problem" refers to a problem that already exists and affects
the performance or implementation of a system or project. In the context of the
project "Intelligent Recommendation System Using CNN-LSTM for Sentiments
in E-Commerce" the existing problems are the challenges and limitations that
can affect the accuracy, scalability, and interpretability of the recommendation
system. These problems include data scarcity, the cold-start problem, accuracy
of sentiment analysis, limited interpretability, and scalability. Addressing these
existing problems is important to improve the performance and effectiveness of
the recommendation system in the e-commerce domain.

Previously, the model used CNN – LSTM algorithm. Long short-term

memory (LSTM) networks and convolutional neural networks (CNNs), two
varieties of deep learning networks, are coupled to form CNN-LSTM. This
model is commonly used for tasks such as image captioning, video
classification, and natural language processing (NLP).

CNNs are commonly used in computer vision tasks, such as image

classification, object detection, and segmentation. LSTM networks are a type of
recurrent neural network (RNN) that are commonly used for sequence prediction
tasks, such as speech recognition, machine translation, and sentiment analysis.

CNN-LSTMs are the training model, we need to feed more data for this
model. We must therefore have a vast amount of data. These models had some
limitations.

10
3.1.1 Limitations of the Existing System
● Limited Contextual Understanding
● Requires More Data
● Limited Multilingual Capabilities
● Lack of Dynamic Context

3.2 PROPOSED SYSTEM

A proposed system is a new or improved system that is being suggested as

a solution to a particular problem or challenge. In the context of an intelligent
recommendation system for e-commerce, a proposed system may involve using
a different model architecture, incorporating new features or data sources, or
improving the existing system in some other way.

The proposed system aims to address the limitations of the existing system
and improve its performance or accuracy in making recommendations to users.
The proposed system is typically evaluated through experimentation and
compared against the performance of the existing system to determine its
effectiveness.

To overcome the following limitations of the existing CNN-LSTM model,

we use the BERT model which was derived from the Transformer networks. In
2018, Google debuted the Transformers for the first time. When language
models were first developed, RNN and CNN were the primary NLP tools
employed.

Transformers-inspired BERT stands for Bidirectional Encoder

Representations. It is bidirectional, meaning it takes into account both the left
and right context of a word in a sentence when generating its embedding. It is
also pre-trained on two tasks: Masked Language Modeling (MLM) Next
Sentence Prediction (NSP).

11
3.2.1 Advantages of the Proposed System

● Contextual Understanding
● Pre-Trained
● Attention Mechanism
● Long-Term Dependencies
● Higher Accuracy

12
 CHAPTER 4

SYSTEM REQUIREMENT

4.1 SOFTWARE REQUIREMENT

Operating System : Windows 10

IDE : Google Colab

Language : PYTHON

Libraries Used : Pandas, Numpy, PyTorch, tqdm, Scikit-learn,

Transformers, Tensorflow, Matplotlib

The software used for this project is based on the Windows 10 operating
system, with Google Colab as the integrated development environment and
Python as the primary programming language. Several popular libraries are
employed, including Pandas and Numpy for data manipulation and numerical
computing, PyTorch and Tensorflow for deep learning models, Scikit-learn for
machine learning models, and Transformers for natural language processing
tasks. The tqdm library is utilized to monitor progress, and Matplotlib is used to
generate visualizations. This combination of tools provides a powerful and
flexible platform for conducting a variety of machine learning and data analysis
tasks.

13
 CHAPTER 5

SOFTWARE DESCRIPTION

5.1 PYTHON

Python is an object-oriented, high-level translator language with dynamic

semantics. Because of its undeniable level of implicit information structures,
dynamic composing, and dynamic restricting, it is extremely engaging for Fast
Application Improvement. Additionally, it can be utilized as a scripting or glue
language to connect already-existing components. Python's easy-to-learn
etymological design emphasizes importance, which lowers program
maintenance costs. Python keeps up with modules and gatherings, which draws
in program isolation and code reuse. For the critical stages in general, the Python
interpreter and the expansive standard library are permitted to scatter in equal or
source structure.

Due to its increased efficiency, developers frequently have passionate

feelings for Python. There is no aggregation step in the change test-investigate
cycle, which results in an insanely quick cycle. Python programming is simple:
A segmentation fault will never be caused by a bug or bad input. Instead, an
exception is raised whenever the interpreter discovers an error.

The capacity to review neighborhood and worldwide factors, assess

inconsistent articulations, set breakpoints, and stroll through the code line by line
are highlights of a source-level debugger. Python's thoughtful force can be seen
in the way the debugger was written. However, frequently including a few print
statements in the source code is the quickest way to debug a program: Due to the
quick change, test, and troubleshoot cycle, this straightforward method is
extremely viable. Python is an easy-to-understand and effective high-level
programming language.

14
 It is open source, so it can be used for business purposes in any case. Like
Ruby and Perl, Python is a scripting language that is frequently used to create
dynamic Web content and applications. Additionally, it is compatible with a
number of imaging programs for two dimensions and three dimensions, allowing
Python users to develop custom plug-ins and extensions. GIMP, Inkscape,
Blender, and Autodesk Maya are applications that help a Python Programming
point of interaction. Python scripts (.PY documents) can be parsed and run
quickly. They can also be saved as accumulated tasks (.PYC records), which are
frequently used as programming modules that other Python programs can
reference.

5.2 LIBRARIES

5.2.1 NumPy

NumPy stands for Numerical Python. It is a popular Python library that

provides support for large, multi-dimensional arrays and matrices, along with a
wide range of mathematical functions to operate on them. NumPy is an essential
tool for scientific computing with Python and is widely used in data analysis,
machine learning, and other scientific applications.

5.2.2 Pandas

Pandas is a Python framework for managing and analyzing data that is free
and open-source. It provides straightforward data formats and data analytic tools
for managing structured data, including tabular, time-series, and matrix data.
Pandas is a key component of the data science ecosystem and is built on top of
NumPy.

5.2.3 Matplotlib

An open-source library for data visualization in Python is called

15
Matplotlib. It offers a variety of options for producing interactive, animated, or
static Python visual representations. Matplotlib, which is based on NumPy,
works nicely with Pandas and SciPy as well as other scientific libraries.

Matplotlib provides a variety of plots, including line plots, scatter plots,

bar plots, histograms, 3D plots, and more. It also supports customization of plot
properties, such as color, marker styles, labels, titles, and axes.

5.2.4 Tensorflow

TensorFlow is an open-source program developed by Google for advanced

machine learning and numerical computing. From straightforward linear models
to intricate deep learning neural networks, TensorFlow is a versatile framework
for building and training machine learning models.

5.2.5 Scikit-Learn

Python's open-source scikit-learn (often referred to as sklearn) library for

machine learning. It offers a variety of tools for supervised and unsupervised
machine learning, data preparation, model selection, and assessment and is built
on top of NumPy, SciPy, and matplotlib.

5.2.6 TQDM

TQDM stands for "progress" in Arabic ("taqadum"). It is an open-source

Python library used for adding progress bars to iterations in loops, enabling users
to monitor the progress of their code in real-time.

TQDM is simple to use and provides a customizable progress bar that

displays the estimated time remaining for the completion of a loop. It also
supports nested loops, parallel processing, and integrates with other libraries
such as Pandas, NumPy, and Scikit-learn.

16
5.2.7 PyTorch

Facebook created the open-source Python machine learning framework

PyTorch. It is designed to provide an easy-to-use interface for building and
training machine learning models, with a focus on deep learning neural
networks.

PyTorch provides a dynamic computational graph, which allows users to

modify and rerun the graph during runtime, making it flexible and efficient for
experimentation and debugging. It also supports automatic differentiation, which
enables users to compute gradients of functions automatically, simplifying the
process of optimizing machine learning models.

5.2.8 Transformers

The Transformer Library is an open-source Python library for natural

language processing (NLP) developed by Hugging Face. It provides a simple
and easy-to-use interface for accessing and using state-of-the-art pre-trained
NLP models.

The Transformer Library offers a wide range of pre-trained models for a

variety of NLP tasks, such as text classification, named entity recognition,
question-answering, and language translation.

5.3 GOOGLE COLAB

Google Colab (short for Collaboratory) is a cloud-based Jupyter journal

climate given by Google. It eliminates the need for users to install software or
set up a development environment on their local machine and enables them to
write and execute Python code in their browser. Google Colab gives admittance
to different registering assets, including computer processor, GPU and TPU,
which can be utilized to run AI models and other computationally escalated

17
errands. Additionally, it lets you access a number of pre-installed libraries, such
as well-known machine learning libraries like TensorFlow, PyTorch, and Scikit-
learn, among others.

The ability to store and share Jupyter notebooks in Google Drive, which
enables users to collaborate on projects in real time, is one of Google Colab' s
main features. It also integrates with popular code hosting platforms like GitHub
and supports Git version control. For data analysis, machine learning, and deep
learning, among other applications, Google Colab is widely used in academia
and industry. Students, researchers, and data scientists who require a scalable
and adaptable work environment are among the most common users.

Additionally, it provides free access to GPU and TPU resources, making it

appealing to users who are unable to afford costly hardware. Python code is
executed in a browser-based Jupyter notebook environment in Google Colab. An
interactive computing environment called Jupyter Notebook lets users write and
execute code, visualize data, and share results with others. At the point when a
client composes Python code in a Colab journal and runs a cell, the code is
executed on a distant server facilitated by Google utilizing the registering assets
designated to the client's meeting.

This incorporates admittance to computer chip, GPU or TPU relying upon

the client's decision and accessibility. The print area of the laptop displays the
outcomes of code execution, such as printouts or visualizations. Clients can
likewise collaborate with the PC utilizing different info fastens, for example, text
boxes, sliders and buttons to change the way of behaving of the code. Using
cloud storage services like Google Drive, GitHub, or others, users can save and
share notebooks with others.

By sharing their notebooks and working on the same code together, they can
also collaborate with other people in real time. By and large, Google Colab gives

18
a helpful and productive climate to run Python code in the cloud with next to no
nearby establishment or programming. Jupyter Journal is an intuitive figuring
climate that permits clients to compose and run code, envision information, and
offer outcomes with others.

At the point when a client composes Python code in a Colab journal and runs
a cell, the code is executed on a far off server facilitated by Google utilizing the
figuring assets designated to the client's meeting. The aftereffects of code
execution, like printouts or perceptions, are shown in the print region of the PC.
Clients can likewise connect with the PC utilizing different information fastens,
for example, text boxes, sliders and buttons to change the way of behaving of
the code.

19
 CHAPTER 6

SYSTEM DESIGN AND METHODOLOGY

6.1 PROJECT DESCRIPTION

The project "Intelligent Recommendation System Using Transformer

Network for Sentiments in E-Commerce" aims to develop a machine learning-
based system that can analyze customer reviews of e-commerce products and
provide personalized product recommendations based on sentiment analysis.
The system will use a Transformer network, a type of neural network
architecture that has shown significant improvements in natural language
processing (NLP) tasks compared to traditional models.
The proposed system will be trained on a large dataset of e-commerce
product reviews and will use a combination of supervised and unsupervised
learning techniques to generate accurate recommendations. The system will
analyze the sentiment of the reviews, as well as other factors such as product
attributes, user demographics, and previous purchase history, to provide
personalized recommendations.
The Transformer network's attention mechanism will allow the system to
capture the context of the words in a sentence and learn the relationships between
them, enabling it to better understand the sentiment of the reviews. This will
allow the system to provide more accurate and personalized recommendations
for each customer.
The system will be designed to be user-friendly and easy to use, with a
simple interface that allows customers to input their preferences and receive
personalized recommendations. The system will also provide customers with an
explanation of how the recommendations were generated to build trust and
increase customer satisfaction.

20
 Overall, the project aims to develop an intelligent recommendation system
that leverages the power of Transformer networks to provide personalized
product recommendations based on sentiment analysis in e-commerce.

6.2 BLOCK DIAGRAM

Block diagram of the model represented in Fig. 6.1, each component of the
system is represented by a rectangle, a square, a circle, or another shape, and the
components are connected by lines or arrows to show the flow of information or
materials between them. The block diagram can be used to describe the overall
architecture of a system, to illustrate the relationships between different
components, or to represent the steps in a process.

Fig. 6.1 BLOCK DIAGRAM OF TRANSFORMER

i. Input Data

Data collection which includes customer reviews, customer IDs and customer
sentiments.

21
ii. Transformer

The encoder and decoder in the Transformer network are each made up of
multiple layers of feed-forward and self-attentional neural networks. The decoder
creates the output sequence, such as a translation of the input text, while the
encoder processes the input sequence.

iii. Tokenization

Tokenization is the process of dividing a text document into smaller, more

concise tokens. These tokens often correlate to words or sub-words in natural
language processing.

iv. Bert Tokenizer

The BERT Tokenizer employs a more sophisticated method that takes into
account the context and structure of the input text, in contrast to conventional
tokenization techniques that break text into individual words. As a result, the
BERT model is better able to discern the subtleties and meaning of natural
language text.

v. Fine Tuning and Pre- Training

A pre-trained model is further trained on a fresh, smaller dataset that is unique

to a certain job as part of the transfer learning process known as fine tuning. A
deep learning technique called pre-training involves training a model on a sizable,
unlabeled dataset in order to obtain a set of initial weights that may be used to
initialize the model for a particular job.

vi. MLM

MLM abbreviated as Masked Language Modeling. This pre-training objective

entails masking a particular percentage of the input tokens in a sentence and

22
training the BERT model to predict the masked tokens based on the context of
the other tokens in the sentence.

vii. NSP

Next Sentence Prediction is referred to as NSP. The BERT model will be

trained to determine whether two sentences are sequential or not as part of this
pre-training objective. Pre-training involves feeding the model pairs of sentences
so that it can learn to predict whether the second sentence continues the first or
not.

[Link]

Finding a collection of model parameters that minimizes the loss function on

a particular dataset is referred to as optimization. Finding the ideal model
parameter values that allow the model to make precise predictions on fresh,
unforeseen data is the aim of optimization.

ix. Output Data

A machine learning model's performance at a given job is significantly
influenced by the accuracy of the output data it generates. Performance indicators,
including accuracy, precision, recall, F1-score, and mean squared error, are
frequently used to assess the output data.

6.3 DATASET DESCRIPTION

In the context of artificial intelligence and machine learning, a dataset is a

collection of data that has been carefully chosen and prepared for training,
validating, or testing a machine learning model. These datasets frequently consist
of labeled examples, where each example has a set of input qualities and an
output label or value that the model is attempting to predict. The input properties
are commonly referred to as the independent variables or predictors, whilst the
output value or label is the dependent variable or target.
23
 For it to be useful for training a machine learning model, it must be big and
varied enough to encompass a wide range of scenarios and instances. It must also
be correct with regard to the setting in which the model will be used in reality.
To make sure that the dataset is free of errors, missing values, and other
irregularities that might impair the model's performance, data pretreatment and
cleaning are typically necessary. The effectiveness of the final model depends
critically on the dataset's quality.

Figure 6.2 shows that the sizable public collection of consumer evaluations
obtained from the Amazon e-commerce platform is known as the Amazon review
dataset. It includes product reviews for items in many different categories,
including electronics, books, movies, and household appliances. The dataset is
frequently employed in sentiment analysis studies. The Amazon review dataset
is freely accessible for study and may be obtained from Kaggle or other websites,
as well as the Amazon website. The dataset contains a number of characteristics,
such as the product ID, reviewer ID, review text, title, rating, and date. Each
review is categorized as either positive or negative depending on the rating the
reviewer offers.

Fig. 6.2 Review Dataset

24
 CHAPTER 7
PERFORMANCE ANALYSIS

The Intelligent Recommendation System Using Transformer Network for

Sentiments in E-commerce project aimed to develop a more precise and efficient
recommendation system for e-commerce websites by combining sentiment
analysis with a transformer network. The results of our experiment demonstrate
that incorporating sentiment analysis into recommendation systems has the
potential to enhance both their accuracy and efficacy. To give clients more
appropriate proposals, our calculation had the option to recognize the feeling of
client assessments definitively. Another important result of our work is the
possibility that e-commerce businesses can use sentiment analysis to get a better
understanding of their customers and improve the entire customer experience. By
looking at customer feedback, e-commerce businesses can learn about common
problems and areas for improvement, which can help them improve their product
development and marketing strategies.
Compared to conventional recommendation systems, our technique
significantly increased recommendation accuracy. On a dataset comprising a few
customer reviews from an online store, our system's accuracy was assessed.
Precision and Recall were the two metrics used in the evaluation. The precision
ratio is calculated by dividing the number of true positives by the sum of true
positives and false positives. Recall is calculated by dividing the total number of
true positives by the sum of all true positives and false negatives. It gauges how
well the model can locate each successful case. The harmonic mean of recall and
precision, as determined by the following formula, is the F1 score.
F1 = 2 * (precision * recall) / (precision + recall)
Higher numbers correspond to better performance, and the scale runs from 0
to 1. When the dataset is unbalanced, that is, when there are not an equal number
of positive and negative cases, the F1 score is helpful.

25
 Due to its use of pre-trained language models, the BERT model is a very
successful method for sentiment analysis applications. The model is better than
other methods at capturing the subtleties of language and context since it was
trained on a lot of text data. This makes it ideal for sentiment analysis, where it's
crucial to faithfully capture a text's underlying sentiment. In comparison to the
F1 score of 53% shown in Fig. 7.1 attained by the prior CNN-LSTM model, our
BERT-based sentiment analysis review system achieved an F1 score of 73%
shown in Fig. 7.2.

(a) (b)

Fig. 7.1 (a) F1-Score calculation and (b) graphical representation of CNN-LSTM model
The design of the models and how they handle text data are some of the
reasons why the CNN-LSTM model performs better than the BERT model in
terms of performance. Nonetheless, the usage of pre-trained language models is
probably a major contributor to the BERT model's higher performance, that was
represented in Fig. 7.3.
The BERT model is able to better capture the mood of a text and deliver more
accurate classifications by making use of pre-existing information about language
and context.

26
 (a) (b)

Fig. 7.2 (a) F1-Score calculation and (b) graphical representation of BERT model

Overall, our findings show that the BERT model is superior for sentiment
analysis tasks and that it may have significant ramifications for related domains.

Fig. 7.3 Bar-chart represents the comparison of the CNN-LSTM and BERT model

27
 Fig. 7.4 Different results on each iteration based on training data

Fig 7.4 shows that the data showed that the categories in the dataset could be
accurately counted and that it was feasible to ascertain how many customers
loved the products. By doing this, the algorithm may quickly estimate the
sentimental review from the text reviews of the customers.

28
 CHAPTER 8

CONCLUSION AND DISCUSSION

In conclusion, our analysis shows that the use of the BERT model for
sentiment analysis can significantly improve accuracy compared to other
approaches, such as the CNN-LSTM model. Our sentiment analysis review
system achieved an f1 score of 73%, which is significantly higher than the 53%
f1 score achieved by the previous CNN-LSTM model. The use of pre-trained
language models in the BERT model allows it to capture the nuances of language
and context more effectively, which is especially important in sentiment analysis
tasks where subtle nuances can significantly impact the sentiment classification.
On the other hand, e-commerce businesses can use the system to boost sales
and customer loyalty by providing personalized recommendations to customers.
However, there are some drawbacks to this strategy that must be addressed. For
instance, the majority of the system is based on user reviews, which may or may
not always be reliable.
Also, the framework could experience difficulty handling immense measures
of information, which could influence execution. The intelligent recommendation
system for e-commerce that makes use of the transformer network is, in
conclusion, a promising option that has the potential to alter the e-commerce
industry.
Additional research is required to address the system's shortcomings and
further enhance its effectiveness. Overall, our findings suggest that the BERT
model is a highly effective approach for sentiment analysis and could have
important implications for related fields, such as natural language processing and
machine learning.

29
 CHAPTER 9

FUTURE ENHANCEMENT

Social media may give us useful information on the interests and attitudes of
users. The accuracy and relevancy of the suggestions may be improved by
including social media data into the algorithm. Product reviews and professional
judgements are examples of external knowledge sources that can offer insightful
data that can improve the precision and applicability of suggestions. The
recommendation system can provide consumers more accurate and pertinent
recommendations by using this data. Your recommendation system's present
configuration makes use of textual data. However, the system may be improved
by including more data modalities like pictures, audio, and video. This may lead
to a deeper comprehension of user preferences and enable more precise
suggestion making.

The recommendation system can provide consumers more accurate and

pertinent recommendations by using this data. Your recommendation system's
current configuration makes use of textual data. However, the system can be
improved by adding data such as images, audio and video. This may lead to a
more thorough understanding of user preferences and enable more precise
suggestion making.

30
 APPENDIX

SOURCE CODE :

from [Link] import drive

[Link]('/content/drive')

import torch

import pandas as pd

from [Link] import tqdm

df =
pd.read_csv('/content/drive/MyDrive/MAIN_PROJECT/smileannotationsfinal.c
sv', names=['id', 'text', 'category'])

df.set_index('id', inplace=True)

#converting label into numeric values

label_dict = {}

for index, possible_label in enumerate(possible_labels):

label_dict[possible_label] = index

df['label'] = [Link](label_dict)

from sklearn.model_selection import train_test_split

X_train, X_val, y_train, y_val = train_test_split([Link],

[Link],

test_size=0.15,

random_state=17,

stratify=[Link])
31
#fill in the datatype

[Link][X_train, 'data_type'] = 'train'

[Link][X_val, 'data_type'] = 'val'

from transformers import BertTokenizer

from [Link] import TensorDataset

encoded_data_train = tokenizer.batch_encode_plus(

df[df.data_type=='train'].[Link],

add_special_tokens=True,

return_attention_mask=True,

pad_to_max_length=True,

max_length=256,

return_tensors='pt'

encoded_data_val = tokenizer.batch_encode_plus(

df[df.data_type=='val'].[Link],

add_special_tokens=True,

return_attention_mask=True,

pad_to_max_length=True,

max_length=256,

return_tensors='pt'

32
#encode train set

input_ids_train = encoded_data_train['input_ids']

attention_masks_train = encoded_data_train['attention_mask']

labels_train = [Link](df[df.data_type=='train'].[Link])

#encode val set

input_ids_val = encoded_data_val['input_ids']

attention_masks_val = encoded_data_val['attention_mask']

#convert data type to [Link]

labels_val = [Link](df[df.data_type=='val'].[Link])

#create dataloader

dataset_train = TensorDataset(input_ids_train, attention_masks_train,

labels_train)

dataset_val = TensorDataset(input_ids_val, attention_masks_val, labels_val)

#load pre-trained BERT

model = BertForSequenceClassification.from_pretrained("bert-base-uncased",

num_labels=len(label_dict),

output_attentions=False,

output_hidden_states=False)

batch_size = 32

dataloader_train = DataLoader(dataset_train,

sampler=RandomSampler(dataset_train),
33
 batch_size=batch_size)

dataloader_validation = DataLoader(dataset_val,

sampler=SequentialSampler(dataset_val),

batch_size=batch_size)

optimizer = AdamW([Link](),

lr=1e-5,

eps=1e-8)

epochs = 25

scheduler = get_linear_schedule_with_warmup(optimizer,

num_warmup_steps=0,

num_training_steps=len(dataloader_train)*epochs)

#Calculating F1_Score

def f1_score_func(preds, labels):

preds_flat = [Link](preds, axis=1).flatten()

labels_flat = [Link]()

return f1_score(labels_flat, preds_flat, average='weighted')

#Accuracy calculation

def accuracy_per_class(preds, labels):

label_dict_inverse = {v: k for k, v in label_dict.items()}

34
 preds_flat = [Link](preds, axis=1).flatten()

labels_flat = [Link]()

for label in [Link](labels_flat):

y_preds = preds_flat[labels_flat==label]

y_true = labels_flat[labels_flat==label]

print(f'Class: {label_dict_inverse[label]}')

print(f'Accuracy: {len(y_preds[y_preds==label])}/{len(y_true)}\n')

def evaluate(dataloader_val):

[Link]()

loss_val_total = 0

predictions, true_vals = [], []

for batch in dataloader_val:

batch = tuple([Link](device) for b in batch)

inputs = {'input_ids': batch[0],

'attention_mask': batch[1],

'labels': batch[2],

with torch.no_grad():

outputs = model(**inputs)

loss = outputs[0]

35
 logits = outputs[1]

loss_val_total += [Link]()

logits = [Link]().cpu().numpy()

label_ids = inputs['labels'].cpu().numpy()

[Link](logits)

true_vals.append(label_ids)

loss_val_avg = loss_val_total/len(dataloader_val)

predictions = [Link](predictions, axis=0)

true_vals = [Link](true_vals, axis=0)

return loss_val_avg, predictions, true_vals

# Import library file for plot class distribution

import [Link] as plt

import seaborn as sns

#plot class distribution

[Link](figsize=(10, 10))

[Link]([Link])

[Link]('Classes')

[Link]('Count')

[Link]('Class Distribution')

36
REFERENCES

1. Abid, Fazeel, Muhammad Alam, Muhammad Yasir, and Chen Li. "Sentiment
analysis through recurrent variants latterly on convolutional neural network
of Twitter." Future Generation Computer Systems 95 (2019): 292-308.
2. Alharbi, Ahmed Sulaiman M., and Elise de Doncker. "Twitter sentiment
analysis with a deep neural network: An enhanced approach using user
behavioral information." Cognitive Systems Research 54 (2019): 50-61.
3. Alsubari, S. Nagi, Sachin N. Deshmukh, A. Abdullah Alqarni, Nizar Alsharif,
T. H. Aldhyani, Fawaz Waselallah Alsaade, and Osamah I. Khalaf. "Data
analytics for the identification of fake reviews using supervised learning."
Computers, Materials & Continua 70, no. 2 (2022): 3189-3204.
4. Alzahrani, Mohammad Eid, Theyazn HH Aldhyani, Saleh Nagi Alsubari,
Maha M. Althobaiti, and Adil Fahad. "Developing an intelligent system with
deep learning algorithms for sentiment analysis of e-commerce product
reviews." Computational Intelligence and Neuroscience 2022 (2022).
5. Basiri, Mohammad Ehsan, Shahla Nemati, Moloud Abdar, Erik Cambria, and
U. Rajendra Acharya. "ABCDM: An attention-based bidirectional CNN-
RNN deep model for sentiment analysis." Future Generation Computer
Systems 115 (2021): 279-294.
6. Cambria, Erik, Dipankar Das, Sivaji Bandyopadhyay, and Antonio Feraco.
"Affective computing and sentiment analysis." A practical guide to sentiment
analysis (2017): 1-10.
7. Dong, Jian, Yu Chen, Aihua Gu, Jingwei Chen, Lili Li, Qinling Chen, Shujun
Li, and Qifeng Xun. "Potential Trend for Online Shopping Data Based on the

37
 Linear Regression and Sentiment Analysis." Mathematical Problems in
Engineering 2020 (2020): 1-11.
8. Hajek, Petr, Aliaksandr Barushka, and Michal Munk. "Fake consumer review
detection using deep neural networks integrating word embeddings and
emotion mining." Neural Computing and Applications 32 (2020): 17259-
17274.
9. Jagdale, Rajkumar S., Vishal S. Shirsat, and Sachin N. Deshmukh.
"Sentiment analysis on product reviews using machine learning techniques."
In Cognitive Informatics and Soft Computing: Proceeding of CISC 2017, pp.
639-647. Springer Singapore, 2019.
[Link], M. S., and S. Jayalakshmi. "Comparative Analysis of
Convolutional Neural Network and LSTM in Text-Based Sentiment
Classification." In 2021 Fifth International Conference on I-SMAC (IoT in
Social, Mobile, Analytics and Cloud) (I-SMAC), pp. 1205-1211. IEEE, 2021.
[Link], Mark JS. Advanced positioning, flow, and sentiment analysis in
commodity markets: bridging fundamental and technical analysis. John
Wiley & Sons, 2020.
[Link], Mathias, and Stefan Feuerriegel. "Sentiment analysis based on
rhetorical structure theory: Learning deep neural networks from discourse
trees." Expert Systems with Applications 118 (2019): 65-79.
[Link], Lin, Tiong-Thye Goh, and Dawei Jin. "How textual quality of online
reviews affect classification performance: a case of deep learning sentiment
analysis." Neural Computing and Applications 32 (2020): 4387-4415.
[Link], Sanjiban Sekhar, Marenglen Biba, Rohan Kumar, Rahul Kumar, and
Pijush Samui. "A new SVM method for recognizing polarity of sentiments in
twitter." In Handbook of Research on Soft Computing and Nature-Inspired
Algorithms, pp. 281-291. IGI Global, 2017.
[Link], Fei, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng
Jiang. "BERT4Rec: Sequential recommendation with bidirectional encoder

38
 representations from transformer." In Proceedings of the 28th ACM
international conference on information and knowledge management, pp.
1441-1450. 2019.
[Link], S., Rajdev Tiwari, and R. Prasad. "Opinion mining and sentiment
analysis on customer review documents—a survey." IJARCCE 6, no. 2
(2017): 156-159.
[Link], Surabhi, Surendrabikram Thapa, Usman Naseem, Hai Ya Lu,
Gnana Bharathy, and Mukesh Prasad. 2023. “Explainable Hybrid Word
Representations for Sentiment Analysis of Financial News.” Neural
Networks: The Official Journal of the International Neural Network Society
164 (April): 115–23.
[Link], Nie. 2022. “E-Commerce Brand Ranking Algorithm Based on User
Evaluation and Sentiment Analysis.” Frontiers in Psychology 13 (June):
907818.
[Link], George. 2019. Online Shopping - Everything You Need to Know.
Blurb.

39
PUBLICATIONS

1. Dilipkumar. S, Krithick Sanjay. K, Manojkumar. E, Surya. S. K, “Intelligent

Recommendation System Using Transformer Network for Sentiments in E-
commerce”, proceedings of 4th International Conference on Recent
Innovations in Engineering and Technology at Society for Aerospace and
Mechanical Professionals and Society for Information, Communication and
Electrical Engineers, Chennai, Tamilnadu on 6th April 2023.
2. Dilipkumar. S, Krithick Sanjay. K, Manojkumar. E, Surya. S. K, “Intelligent
Recommendation System Using Transformer Network for Sentiments in E-
commerce”, proceedings of National Conference on Emerging Trends in
Engineering and Technology at Nandha Engineering College, Perundurai,
Tamilnadu on 19th April 2023.

40
41
42
43
44