Advanced AI and ML 21AI71

MODULE 4
Recommender System

4.1 OVERVIEW
• Recommendation systems are a set of algorithms which recommend most relevant
items to users based on their preferences predicted using the algorithms.
• It acts on behavioral data, such as customer’s previous purchase, ratings or reviews to
predict their likelihood of buying a new product or service.
• Amazon’s “Customers who buy this item also bought”, Netflix’s “shows and movies
you may want to watch” are examples of recommendation systems.
• Recommender systems are very popular for recommending products such as movies,
music, news, books, articles, groceries and act as a backbone for cross-selling across
industries.

4.1.1 Datasets
For exploring the algorithms, we will be using the following two publicly available datasets
and build recommendations.
1. [Link]: This dataset contains transactions of a grocery store and can be
downloaded from
[Link]

2. Movie Lens: This dataset contains 20000263 ratings and 465564 tag applications
across 27278 movies. As per the source of data, these data were created by 138493
users between January 09, 1995 and March 31, 2015. This dataset was generated on
October 17, 2016. Users were selected and included randomly. All selected users had
rated at least 20 movies. The dataset can be downloaded from the link
[Link]

1 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

4.2 ASSOCIATION RULES (ASSOCIATION RULE MINING)

• Association rule finds combinations of items that frequently occur together in orders
or baskets (in a retail context).
• The items that frequently occur together are called itemsets. Itemsets help to discover
relationships between items that people buy together and use that as a basis for
creating strategies like combining products as combo offer or place products next to
each other in retail shelves to attract customer attention.
• An application of association rule mining is in Market Basket Analysis (MBA).
MBA is a technique used mostly by retailers to find associations between items
purchased by customers.

To illustrate the association rule mining concept, let us consider a set of baskets and the items
in those baskets purchased by customers as depicted in Figure.

Items purchased in different baskets are:

1. Basket 1: egg, beer, sugar, bread, diaper
2. Basket 2: egg, beer, cereal, bread, diaper
3. Basket 3: milk, beer, bread
4. Basket 4: cereal, diaper, bread

• The primary objective of a recommender system is to predict items that a customer

may purchase in the future based on his/her purchases so far.
• In future, if a customer buys beer, can we predict what he/she is most likely to buy
along with beer? To predict this, we need to find out which items have shown a strong
association with beer in previously purchased baskets. We can use association rule
mining technique to find this out.

2 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

• Association rule considers all possible combination of items in the previous baskets
and computes various measures such as support, confidence, and lift to identify rules
with stronger associations.
• One of the challenges in association rule mining is the number of combination of items
that need to be considered; as the number of unique items sold by the seller increases,
the number of associations can increase exponentially.
• One solution to this problem is to eliminate items that possibly cannot be part of any
itemsets. One such algorithm the association rules use Apriori algorithm.
• The Apriori algorithm was proposed by Agrawal and Srikant (1994).
The rules generated are represented as

which means that customers who purchased diapers also purchased beer in the same
basket. {diaper, beer} together is called itemset. {diaper} is called the antecedent and
the {beer} is called the consequent.

Both antecedents and consequents can have multiple items. The below example is also
a valid rule

4.2.1 Metrics
Concepts such as support, confidence, and lift are used to generate association rules.
1. Support
• Support indicates the frequencies of items appearing together in baskets with respect
to all possible baskets being considered (or in a sample).
• For example, the support for (beer, diaper) will be 2/4 (based on the data shown in
Figure 9.1), that is, 50% as it appears together in 2 baskets out of 4 baskets.

3 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

2. Confidence
• Confidence measures the proportion of the transactions that contain X, which also
contain Y. X is called antecedent and Y is called consequent.
• Confidence can be calculated using the following formula:

where P(Y|X) is the conditional probability of Y given X.

3. Lift
Lift is calculated using the following formula:

• Lift can be interpreted as the degree of association between two items.

• Lift value 1 indicates that the items are independent (no association), lift value of less
than 1 implies that the products are substitution (purchase one product will decrease
the probability of purchase of the other product) and lift value of greater than 1
indicates purchase of Product X will increase the probability of purchase of Product Y.
• Lift value of greater than 1 is a necessary condition of generating association rules.

4.2.2 Applying Association Rules

To understand and apply association rules using transaction data in [Link]. This will
involve loading, encoding, and analysing transaction data to uncover patterns and
associations in customer purchasing behaviors.

4 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

all_txns = []
with open('[Link]') as f:
content = [Link]()
txns = [[Link]() for x in content] # Remove whitespace
for each_txn in txns:
all_txns.append(each_txn.split(','))

2. Encoding the Transactions

Convert the list of transactions into a one-hot-encoded matrix for easier rule generation.
Library: mlxtend provides OnehotTransactions for this purpose.

import pandas as pd
from [Link] import OnehotTransactions

one_hot_encoding = OnehotTransactions()
one_hot_txns = one_hot_encoding.fit(all_txns).transform(all_txns)
one_hot_txns_df =
[Link](one_hot_txns, columns=one_hot_encoding.columns_)

Matrix Structure: Rows represent transactions; columns represent items, with 1 for purchased
items and 0 otherwise.

3. Generating Association Rules

Use the Apriori algorithm to find frequent itemsets with a specified minimum support
threshold.

5 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

Apriori algorithm takes the following parameters:

1. df: pandas − DataFrame in a one-hot-encoded format.
2. min_support: float − A float between 0 and 1 for minimum support of the itemsets
returned. Default is 0.5.
3. use_colnames: boolean − If true, uses the DataFrames’ column names in the returned
DataFrame instead of column indices.
We will be using a minimum support of 0.02, that is, the itemset is available in at least 2% of
all transactions.

from mlxtend.frequent_patterns import apriori

frequent_itemsets = apriori(one_hot_txns_df, min_support=0.02,

use_colnames=True)

frequent_itemsets.sample(10, random_state=90)

4. Creating Association Rules

Use association_rules to generate rules from frequent itemsets, with lift as the evaluation
metric.

6 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

The corresponding association rules are

Let us look at the top 10 association rules sorted by confidence. The rules stored in the
variable rules are sorted by confidence in descending order.

7 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

From Table 9.4, we can infer that the probability that a customer buys (whole milk), given
he/she has bought (yogurt, other vegetables), is 0.51.

4.3 COLLABORATIVE FILTERING

• Collaborative filtering is based on the notion of similarity (or distance).
• For example, if two users A and B have purchased the same products and have rated
them similarly on a common rating scale, then A and B can be considered similar in
their buying and preference behavior.
• Hence, if A buys a new product and rates high, then that product can be recommended
to B. Alternatively, the products that A has already bought and rated high can be
recommended to B, if not already bought by B.

4.3.1 How to Find Similarity between Users

• Similarity or the distance between users can be computed using the rating the users
have given to the common items purchased.
• If the users are similar, then the similarity measures such as Jaccard coefficient and
cosine similarity will have a value closer to 1 and distance measures such as Euclidian
distance will have low value.
• Example: The picture in Figure 9.2 depicts three users Rahul, Purvi, and Gaurav and
the books they have bought and rated.

8 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

The users are represented using their rating on the Euclidean space in Figure 9.3. Here the
dimensions are represented by the two books Into Thin Air and Missoula, which are the two
books commonly bought by Rahul, Purvi, and Gaurav.

9 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

Figure 9.3 shows that Rahul’s preferences are similar to Purvi’s rather than to Gaurav’s. So,
the other book, Into the Wild, which Rahul has bought and rated high, can now be
recommended to Purvi.

4.3.2 User-Based Similarity

This approach recommends items to a user based on the preferences of similar users. If two
users have rated the same items similarly, they’re considered similar. Therefore, items liked
by one user can be recommended to the other. This similarity is often computed using metrics
like cosine similarity, Pearson correlation, or Jaccard coefficient.

• We will use MovieLens dataset for finding similar users based on common movies the
users have watched and how they have rated those movies.
• The file [Link] in the dataset contains ratings given by users. Each line in this file
represents a rating given by a user to a movie.
• The ratings are on the scale of 1 to 5. The dataset has the following features:
1. userId
2. movieId
3. rating
4. timestamp

Example Using the MovieLens Dataset

In this example, we use the MovieLens dataset, which provides movie ratings by users, with
each rating recorded in a CSV file. The following steps outline how to perform collaborative
filtering using user-based similarity:

1. Data Preparation:
• Load the dataset and drop unnecessary columns, such as the timestamp.
• Create a pivot table where rows represent users, columns represent movies, and values
are the ratings. This pivot table, which is sparse, has NaNs where users haven’t rated
specific movies. These NaNs are then filled with 0s to facilitate similarity calculations.

10 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

Create a pivot table or matrix and represent users as rows and movies as columns. The values
of the matrix will be the ratings the users have given to those movies
Those movies that the users have not watched and rated yet, will be represented as NaN.

2. Calculating Cosine Similarity between Users

The formula for cosine similarity between two users u and v is:

11 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

• [Link].pairwise_distances can be used to compute distance between all pairs

of users.
• pairwise_distances() takes a metric parameter for what distance measure to use. We
will be using cosine similarity for finding similarity. Cosine similarity closer to 1
means user are very similar and closer to 0 means users are very dissimilar.

3. Finding Similar Users:

• For each user, the user with the highest similarity score is identified.
• For instance, if user 338 is most similar to user 2 based on a cosine similarity score of
0.58, this means user 338’s ratings are closely aligned with those of user 2.

Challenges with User-Based Similarity

12 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

1. Cold Start Problem: New users lack sufficient data (ratings or purchases) to find
similar users. Recommendations can't be made effectively until the new user interacts
with the system by purchasing and rating a few items.
2. Sparse Data: User preferences might be sparse (i.e., not all users rate all items),
which reduces the effectiveness of identifying user similarities, especially in niche
categories.
This can be overcome by using item-based similarity. Item-based similarity is based on the
notion that if two items have been bought by many users and rated similarly, then there must
be some inherent relationship between these two items. In other terms, in future, if a user
buys one of those two items, he or she will most likely buy the other one.

4.3.3 Item-Based Similarity

• Item-based similarity is based on the notion that if two items have been bought by
many users and rated similarly, then there must be some inherent relationship between
these two items.
• Eg. If two movies, movie A and movie B, have been watched by several users and
rated very similarly, then movie A and movie B can be similar in taste. In other words,
if a user watches movie A, then he or she is very likely to watch B and vice versa.

1. Calculating Cosine Similarity between Movies

The Cosine Similarity between Movie A and Movie B is calculated as:

13 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

In this approach, create a pivot table, where the rows represent movies, columns represent
users, and the cells in the matrix represent ratings the users have given to the movies. So, the
pivot() method will be called with movieId as index and userId as columns as described
below:

Print similarity between the first 5 movies.

2. Finding Most Similar Movies

The provided method get_similar_movies() is used to return the top N movies similar to a
given movie based on cosine similarity.

14 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

• movieidx: We first find the index of the movie in movies_df based on the movieid.
• movie_similarities: We extract the similarity scores for the given movie using
movie_sim_df.iloc[movieidx]. This gives us a row of similarity scores between the
movie and all other movies.
• Add Similarity: We assign these similarity scores to a new column similarity in
movies_df.
• Sort and Filter: We sort the movies_df by the similarity column in descending order,
excluding the original movie, and return the top N similar movies.

The users who watched ‘Godfather, The’, also watched ‘Godfather: Part II’ the most. So, in
future, if any user watches ‘Godfather, The’, the other movies can be recommended to them.

15 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

4.4 USING SURPRISE LIBRARY

The Surprise library is a powerful tool for building and evaluating recommender systems in
Python.
It provides the following features:
• Various ready-to-use prediction algorithms like neighborhood methods (user similarity
and item similarity), and matrix factorization-based.
• It also has built-in similarity measures such as cosine, mean square distance (MSD),
Pearson correlation coefficient, etc.
• Tools to evaluate, analyze, and compare the performance of the algorithms. It also
provides methods to recommend.

We import the required modules or classes from surprise library

The [Link] is used to load the datasets and has a method load_from_df to convert
DataFrames to Dataset.
Reader class can be used to provide the range of rating scales that is being used.

4.4.1. User-Based Similarity Algorithm

The following code implements movies recommendation based on Pearson correlation and 20
nearest similar users.

16 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

We can do 5-fold cross-validation to measure RMSE score to find out how the algorithm
performs on the dataset.

The average accuracy across all the folds.

4.4.2. Finding the Best Model

Surprise provides GridSearchCV to search through various models and similarity indexes to
find the model that gives the highest accuracy.
The surprise.model_selection.[Link] takes the following parameters:

17 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

4.4.3 Making Predictions

18 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

4.5 MATRIX FACTORIZATION

• Matrix factorization is a matrix decomposition technique.
• Matrix decomposition is an approach for reducing a matrix into its constituent parts.
• Matrix factorization algorithms decompose the user-item matrix into the product of
two lower dimensional rectangular matrices.
• In Figure 9.4, the original matrix contains users as rows, movies as columns, and
rating as values. The matrix can be decomposed into two lower dimensional
rectangular matrices.

• The Users–Movies matrix contains the ratings of 3 users (U1, U2, U3) for 5 movies
(M1 through M5). This Users–Movies matrix is factorized into a (3, 3) Users–Factors
matrix and (3, 5) Factors–Movies matrix. Multiplying the Users–Factors and Factors–
Movies matrix will result in the original Users– Movies matrix.
• The idea behind matrix factorization is that there are latent factors that determine why
a user rates a movie, and the way he/she rates. The factors could be the story or actors
or any other specific attributes of the movies.
• A matrix with size (n, m), where n is the number of users and m is the number of
movies, can be factorized into (n, k) and (k, m) matrices, where k is the number of
factors.
• The Users–Factors matrix represents that there are three factors and how each user has
preferences towards these factors. Factors–Movies matrix represents the attributes the
movies possess.
• In the above example, U1 has the highest preference for factor F2, whereas U2 has the
highest preference for factor F3. Similarly, the F2 factor is high in movies M2 and M4.
Probably this is the reason why U1 has given high ratings to movies M2 (4) and M4
(5).

19 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

MODULE 4
Text Analytics

10.1 OVERVIEW
In today's data-driven world, text data serves as one of the largest sources of information.
Unlike structured data (e.g., rows and columns in a table), text data is inherently unstructured.
Examples of text analytics include:
• Identifying customer sentiments from product reviews or feedback.
• Extracting opinions from social media posts.
Challenges in Text Analytics
• Complexity of Unstructured Data: Unlike structured data, deriving insights from text
requires extensive data preprocessing.
• Need for Structured Representation: Most machine learning algorithms (e.g.,
regression, classification, clustering) operate on structured data in a matrix format.
Thus, text must be transformed into such a format before applying these techniques.
Applications of Text Analytics
• Regression: Predicting stock price movements using positive and negative sentiments
extracted from news articles.
• Classification: Categorizing a review as positive or negative sentiment using
comments provided by customers.

10.2 SENTIMENT CLASSIFICATION

The dataset used here is sentiment_train ( [Link] ) dataset
contains review comments on several movies. Comments in the dataset are already labeled as
either positive or negative. The dataset contains the following two fields separated by a tab
character:
1. text: Actual review comment on the movie.
2. sentiment: Positive sentiments are labelled as 1 and negative sentiments are labelled as
0.

20 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

10.2.1. Load the Dataset

Loading the data using pandas’ read_csv() method

Each record or example in the column text is called a document.

First five positive sentiments

List of negative comments

10.2.2. Exploring the Dataset

Exploratory data analysis can be carried out by counting the number of comments, positive
comments, negative comments, etc.
For example, print metadata of the DataFrame using info() method to check how many
reviews are available in the dataset? Are the positive and negative sentiment reviews well
represented in the dataset?

21 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

From the output we can infer that there are 6918 records available in the dataset. We create a
count plot to compare the number of positive and negative sentiments.

There is total 6918 records (feedback on movies) in the dataset. Out of 6918 records, 2975
records belong to negative sentiments, while 3943 records belong to positive sentiments.
Thus, positive and negative sentiment documents have fairly equal representation in the
dataset.

22 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

10.2.3. Text Pre-processing

• Before building the model, text data needs pre-processing for feature extraction.
• Consider each word as a feature and find a measure to capture whether a word exists
or does not exist in a sentence. This is called the bag-of-words (BoW) model.
• Each sentence (comment on a movie or a product) is treated as a bag of words. Each
sentence (record) is called a document and collection of all documents is called
corpus.

[Link] Bag-of-Words (BoW) Model:

• To create a BoW model, first create a dictionary of all the words used in the corpus.
• Then convert each document to a vector that represents words available in the
document.
There are three ways to identify the importance of words in a BoW model:
1. Count Vector Model
2. Term Frequency Vector Model
3. Term Frequency-Inverse Document Frequency (TF-IDF) Model

1. Count Vector Model

Consider the following two documents:
1. Document 1 (positive sentiment): I really really like IPL.
2. Document 2 (negative sentiment): I never like IPL.
Note: IPL stands for Indian Premier League.
• The complete vocabulary set for the above two documents will have words such as I,
really, never, like, IPL. These words can be considered as features (x1 through x5).
• For creating count vectors, we count the occurrence of each word in the document as
shown below. The y-column in below table indicates the sentiment of the statement: 1
for positive and 0 for negative sentiment.

23 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

2. Term Frequency Vector Model

• Term frequency (TF) vector is calculated for each document in the corpus and is the
frequency of each term in the document.
• It is given by,

where TFi is the term frequency for word (aka token). TF representation for the two
documents is shown in Table

3. Term Frequency-Inverse Document Frequency (TF-IDF)

• TF-IDF measures how important a word is to a document in the corpus. The
importance of a word (or token) increases proportionally to the number of times a
word appears in the document but is reduced by the frequency of the word present in
the corpus.
• TF-IDF for a word i in the document is given by

The IDF value for each word for the above two documents is given in Table

24 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

The TF-IDF values for the two documents are shown in Table

[Link] Creating Count Vectors for sentiment_train Dataset

• Each document in the dataset needs to be transformed into TF or TF-IDF vectors
• Use CountVectorizer to create count vectors. In CountVectorizer, the documents will
be represented by the number of times each word appears in the document.
• Here a dictionary is created of all words present across all the documents. The
dictionary will contain all unique words across the corpus. And each word in the
dictionary will be treated as feature.
Total number of features or unique words in the corpus are 2132.
• Using the above dictionary, convert all the documents in the dataset to count vectors
using transform() method of count vectorizer.
• After converting the document into a vector, we will have a sparse matrix with 2132
features or dimensions.
• Each document is represented by a count vector of 2132 dimensions and if a specific
word exists in a document, the corresponding dimension of the vector will be set to the
count of that word in the document. But most of the documents have only few words
in them, hence most of the dimensions in the vectors will have value set to 0. That is a
lot of 0’s in the matrix! So, the matrix is stored as a sparse matrix.

[Link] Displaying Document Vectors

• To visualize the count vectors, convert this matrix into a DataFrame and set the
column names to the actual feature names.

Print the dimensions (words) from index 150 to 157. This index range contains the word
awesome, which actually encoded into 1.

25 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

[Link] Removing Low-frequency Words

• One of the challenges of dealing with text is the number of words or features available
in the corpus is too large.
• The number of features could easily go over tens of thousands. Some words would be
common words and be present across most of the documents, while some words would
be rare and present only in very few documents.
• To calculate the total occurrence of each feature or word, we will use [Link]()
method.

[Link] Removing Stop Words

sklearn.feature_extraction.text provides a list of pre-defined stop words in English, which can
be used as a reference to remove the stop words from the dictionary, that is, feature set.

[Link] Creating Count Vectors

All vectorizer classes take a list of stop words as a parameter and remove the stop words
while building the dictionary or feature set. And these words will not appear in the count
vectors representing the documents.

26 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

• Many words appear in multiple forms. For example, love and loved.
• The vectorizer treats the two words as two separate words and hence creates two
separate features. But, if a word has similar meaning in all its form, we can use only
the root word as a feature.
• Stemming and Lemmatization are two popular techniques that are used to convert the
words into root words.
Stemming:
• This removes the differences between inflected forms of a word to reduce each word
to its root form. This is done by mostly chopping off the end of words (suffix).
• For instance, love or loved will be reduced to the root word love. The root form of a
word may not even be a real word. For example, awesome and awesomeness will be
stemmed to awesom. One problem with stemming is that chopping of words may
result in words that are not part of vocabulary
Lemmatization:
• This takes the morphological analysis of the words into consideration.
• It uses a language dictionary (i.e., English dictionary) to convert the words to the root
word. For example, stemming would fail to differentiate between man and men, while
lemmatization can bring these words to its original form man.

27 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

It can be noted that words love, loved, awesome have all been stemmed to the root words.

[Link] Distribution of Words Across Different Sentiment

• The words which have positive or negative meaning occur across documents of
different sentiments. This could give an initial idea of how these words can be good
features for predicting the sentiment of documents.
• For example, let us consider the word awesome.

28 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

This gives us an initial idea that the words awesom and hate could be good features in
determining sentiments of the document.

10.3 NAÏVE–BAYES MODEL FOR SENTIMENT CLASSIFICATION

Assume that we would like to predict whether the probability of a document is positive or
negative given that the document contains a word awesome.
This can be computed if the probability of the word awesome appearing in a document given
that it is a positive (or negative) sentiment multiplied by the probability of the document
being positive (or negative).

The posterior probability of the sentiment is computed from the prior probabilities of all the
words it contains. The assumption is that the occurrences of the words in a document are
considered independent and they do not influence each other. So, if the document contains N
words and words are represented as W1, W2, …., WN, then

29 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

The steps involved in using Naïve–Bayes Model for sentiment classification are as follows:
1. Split dataset into train and validation sets.
2. Build the Naïve–Bayes model.
3. Find model accuracy.
1. Split the Dataset
Split the dataset into 70:30 ratio for creating training and test datasets using the following
code.

2. Build Naïve–Bayes Model

Build Naïve–Bayes model using the training set.

3 Make Prediction on Test Case

Predicted class will be the one which has the higher probability based on the Naïve–Bayes’
probability calculation. Predict the sentiments of the test dataset using predict() method.

4 Finding Model Accuracy

Let us print the classification report.

30 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

The model is classifying with very high accuracy. Both average precision and recall is about
98% for identifying positive and negative sentiment documents.
The confusion matrix:

As per the model prediction, that there are only 14 positive sentiment documents classified
wrongly as negative sentiment documents (False Negatives) and there are only 28 negative
sentiment documents classified wrongly as positive sentiment documents (False Positives).
Rest all have been classified correctly.

10.4 USING TF-IDF VECTORIZER

TfidfVectorizer is used to create both TF Vectorizer and TF-IDF Vectorizer. It takes a
parameter use_idf (default True) to create TF-IDF vectors. If use_idf set to False, it will
create only TF vectors and if it is set to True, it will create TF-IDF vectors.

TF-IDF are continuous values and these continuous values associated with each class can be
assumed to be distributed according to Gaussian distribution. So, Gaussian Naïve–Bayes can
be used to classify these documents. We will use GaussianNB, which implements the
Gaussian Naïve–Bayes algorithm for classification.

31 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

The accuracy is very high in this example as the dataset is clean and carefully curated.

10.5 CHALLENGES OF TEXT ANALYTICS

Key Challenges:
1. Context-Specific Language:
• Text data is highly context-sensitive. The language used to describe one domain
(e.g., movies) may differ significantly from another domain (e.g., apparel).
• Solution: Training data should come from the same domain as the target
application.
2. Informal Language:
• Social media data may include mixed languages, slang, and emoticons, making
analysis challenging.
• Solution: Include similar informal examples in the training data.
3. Bag-of-Words Limitation:
• Bag-of-Words (BoW) ignores sentence structure and word order, losing
contextual meaning.
• Solution: Use n-grams to retain some contextual information.

10.5.1 Using n-Grams

The features are created out of each token or word. But the meaning of some of the words
might be dependent on the words it precedes or succeeds, for example not happy. It should be
considered as one feature and not as two different features.
• n-Gram: A sequence of n contiguous words in a sentence.
• Bigram: Two consecutive words treated as one feature (e.g., not happy).
• Trigram: Three consecutive words as one feature.

32 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

Example:
• A unigram model represents individual words: [not, happy].
• A bigram model captures phrases: [not happy].

Custom Tokenizer with Stemming:

• The custom function get_stemmed_tokens() tokenizes text, removes non-alphabetic
characters, and applies stemming.
Using TfidfVectorizer:
• The TfidfVectorizer can use the custom tokenizer and n-grams for feature extraction.

10.5.2 Build the Model Using n-Grams

Split the dataset to 70:30 ratio for creating training and test datasets and then apply
BernoulliNB for classification. Apply the model to predict the test set and then print the
classification report.

33 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru

 Advanced AI and ML 21AI71

Review Questions

1. Discuss the applications in Text Analytics.

2. Define corpus. Explain the ways to identify the importance of words in Bag-of-Words
(BoW) Model.
3. How does the Naïve-Bayes model classify sentiments in text data?
4. List and explain key challenges in text analytics. Explain how the use of n-grams helps
address these challenges.

34 Deepak D, Asst. Prof., Dept. of AIML, Canara Engineering College, Mangaluru