lOMoARcPSD|69216461

MMC102 Module 3 Notes

MCA 2nd sem (Visvesvaraya Technological University)

messages.pdf_cover_qr_code_label

messages.studocu_not_sponsored_or_endorsed_by_college
messages.downloaded_by
 lOMoARcPSD|69216461

MACHINE LEARNING
&
DATA ANALYTICS USING PYTHON
[MMC201]
(2024-26)

Dr. Srinivasa Rao K

Prof. & Head, Department of MCA
Sri Venkateshwara College of Engineering
Bengaluru.

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Machine learning and Data analytics using Python

Course Code MMC201 CIE Marks 50
Teaching Hours/Week (L:P:
2:2:0 SEE Marks 50
SDA/T/T)
Total Hours of Pedagogy 50 Total Marks 100
Credits 04 Exam Hours 03
Course Learning Objectives:
1. Understand foundational concepts in machine learning and data analytics.
2. Gain proficiency in Python for data analysis and machine learning tasks.
3. Learn and apply various machine learning algorithms and techniques.
4. Develop skills in data preprocessing, visualization, and model evaluation.
5. Prepare students for industry roles involving data-driven decision making and
predictive modeling.
Module-3 08 Hours
Unsupervised Learning:
Clustering: K-Means clustering, Hierarchical clustering, Evaluation of clustering results.
Dimensionality Reduction: Principal Component Analysis (PCA), Linear Discriminant
Analysis (LDA) t-Distributed Stochastic Neighbor Embedding (t-SNE).
Association Rule Learning: Apriori algorithm, Market Basket Analysis, Evaluation
metrics for association rules
Teaching Learning Process:
Lab exercises on clustering and dimensionality reduction, Practical coding sessions with
unsupervised learning techniques, Group projects on applying unsupervised learning to
real- world data, Continuous assessment through quizzes and practical tests.

Sl.
Experiments
NO
Implement and demonstrate the FIND-S algorithm for finding the most specific hypothesis
1
based on a given set of training data samples. Read the training data from a .CSV file.
For a given set of training data examples stored in a .CSV file, implement and demonstrate
2 the Candidate- Elimination algorithm to output a description of the set of all hypotheses
consistent with the training examples.
Write a program to demonstrate the working of the decision tree based ID3 algorithm. Use
3 an appropriate da set for building the decision tree and apply this knowledge to classify a
new sample.
Write a program to implement the naïve Bayesian classifier for a sample training data set
4
stored as a .CSV fil Compute the accuracy of the classifier, considering few test data sets.
Write a program to implement k-Nearest Neighbour algorithm to classify the iris data set.
5
Print both correct a wrong predictions.
Build an Artificial Neural Network by implementing the Backpropagation algorithm and
6
test the same using appropriate data sets.
7 Write a program to demonstrate Regression analysis with residual plots on a given data set.
Write a program to compute summary statistics such as mean, median, mode, standard
8
deviation and variance the given different types of data.
Write a program to implement k-Means clustering algorithm to cluster the set of data stored
9
in .CSV file.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

MODULE 3
UNSUPERVISED LEARNING

INTRODUCTION
Unsupervised learning is a branch of machine learning that deals with unlabeled data.
• Unlike supervised learning, where the data is labeled with a specific category or outcome,
• unsupervised learning algorithms are tasked with finding patterns and relationships within the data
without any prior knowledge of the data's meaning.
• Unsupervised machine learning algorithms find hidden patterns and data without any human
intervention, i.e., we don't give output to our model.
• The training model has only input parameter values and discovers the groups or patterns on its own.

Key Characteristics of Unsupervised Learning:

• Unlabeled Data: The primary distinguishing feature is the absence of pre-existing labels or target
variables in the training data.
• Pattern Discovery: The goal is to uncover inherent structures, groupings, or associations within
the data.
• Exploratory: It's often used for exploratory data analysis to gain insights into the underlying
distribution of the data.
• No Explicit Feedback: The algorithms learn without explicit feedback on the correctness of their
outputs.

Applications of Unsupervised Learning:

Unsupervised learning has a wide range of applications across various industries, including:
• Business Intelligence: Customer segmentation, market basket analysis, anomaly detection (fraud,
unusual website activity).
• Healthcare: Grouping patients with similar symptoms, analysing gene expression data, medical
image analysis.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

• Natural Language Processing (NLP): Topic modeling (categorizing news articles), text
summarization, word embeddings.
• Image and Video Analysis: Image compression, object recognition (learning features without
labeled examples), video surveillance.
• Cybersecurity: Detecting unusual network traffic patterns indicating a cyberattack.
• Recommendation Systems: Personalizing recommendations for online shoppers or content
consumers.
• Scientific Research: Classifying galaxies in astronomy, grouping weather patterns in climate
science.
Advantages of Unsupervised Learning:
• Handles Unlabeled Data: Its biggest strength is its ability to extract insights from data where
labeling is expensive, time-consuming, or simply not feasible.
• Discovers Hidden Patterns: Can uncover patterns and relationships that human analysts might
miss.
• Exploratory Analysis: Excellent for initial data exploration and understanding the underlying
structure of a dataset.
• Scalability: Can be applied to large and diverse datasets.
Limitations of Unsupervised Learning:
• Lack of Ground Truth: Without labels, it can be challenging to objectively evaluate the
performance of unsupervised models or interpret the meaning of the discovered patterns.
• Subjectivity in Interpretation: The interpretation of clusters or associations often requires domain
expertise and can be subjective.
• Computational Complexity: Some unsupervised learning algorithms can be computationally
intensive, especially for very large datasets.
• Irrelevant Patterns: The algorithm might discover patterns that are not meaningful or relevant to
the problem at hand.

Distinct from Supervised Learning:

Supervised Learning: Works with labeled data, where each input has a corresponding known output (e.g.,
predicting house prices given features, classifying emails as spam/not-spam). The algorithm learns a
mapping from inputs to outputs based on these labels.
Unsupervised Learning: Works with unlabeled data. The algorithm identifies inherent structures or
patterns without explicit guidance. It's about finding relationships, not predicting a known outcome.

Supervised vs Unsupervised Learning

Feature Supervised Learning Unsupervised Learning

Labeled Data Yes No

Goal Predict output Find patterns

Examples Classification, Regression Clustering, PCA

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Unsupervised Learning Algorithms

There are mainly 3 types of Algorithms which are used for Unsupervised dataset.
• Clustering
• Association Rule Learning
• Dimensionality Reduction

CLUSTERING

Clustering in unsupervised machine learning is the process of grouping unlabeled data into clusters based
on their similarities. The goal of clustering is to identify patterns and relationships in the data without any
prior knowledge of the data's meaning.
Broadly this technique is applied to group data based on different patterns, such as similarities or
differences, our machine model finds. These algorithms are used to process raw, unclassified data objects
into groups.
Think of it as you have a dataset of customers shopping habits. Clustering can help you group customers
with similar purchasing behaviours, which can then be used for targeted marketing, product
recommendations, or customer segmentation.
For Example, In the graph given below, we can clearly see that there are 3 circular clusters forming on the
basis of distance.

Now it is not necessary that the clusters formed must be circular in shape. The shape of clusters can be
arbitrary. There are many algorithms that work well with detecting arbitrary shaped clusters.
For example, In the below given graph we can see that the clusters formed are not circular in shape.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Types of Clustering Methods

At the surface level, clustering helps in the analysis of unstructured data. Graphing, the shortest distance,
and the density of the data points are a few of the elements that influence cluster formation. Clustering is
the process of determining how related the objects are based on a metric called the similarity measure.
Similarity metrics are easier to locate in smaller sets of features and harder as the number of features
increases. Depending on the type of clustering algorithm being utilized, several techniques are employed
to group the data from the datasets. In this part, the clustering techniques are described. Various types of
clustering algorithms are:
1. Centroid-based Clustering (Partitioning methods)
2. Connectivity-based Clustering (Hierarchical clustering)

Centroid-based Clustering (Partitioning methods)

Centroid-based clustering organizes data points around central vectors (centroids) that represent clusters.
Each data point belongs to the cluster with the nearest centroid.
For example, if the input data is (3,3), (2,6) 𝑎𝑛𝑑 (7,9) then the centroid is given by,
3 + 2 + 7, 3 + 6 + 9
( ) = (4,6)
3
The clusters should not overlap and every cluster should represent only one class. Therefore, clustering
algorithms use trial and error method form clusters that can be converted to labels
One of the popular algorithms of Centroid-based clustering is

K-Means Clustering
K-Means Clustering is an Unsupervised Machine Learning algorithm which groups unlabeled dataset into
different clusters. It is used to organize data into groups based on their similarity.
How k-means clustering works?
Consider the data set of items with certain features and values for these features like a vector. The task is
to categorize those items into groups. To achieve this we will use the K-means algorithm. 'K' in the name
of the algorithm represents the number of groups/clusters we want to classify our items into.

k_means_clustering

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

The algorithm will categorize the items into k groups or clusters of similarity. To calculate that similarity
we will use the Euclidean distance as a measurement. The algorithm works as follows:

Step 1: Determine the number of clusters before the algorithm is started. This is called k.
Step 2: Choose k instances randomly. These are initial cluster centers.
Step 3: Compute the mean of the initial clusters and assign the remaining sample to the closest
cluster based on Euclidean distance or any other distance measure between the instances and the
centroid of the clusters.
Step 4: Compute new centroid again considering the newly added samples.
Step 5: Perform the steps 3-4 till the algorithm becomes stable with no more changes in
assignment of instances and clusters.

Example: Consider the following set of data given in below Table. Cluster it using k-means algorithm with
the initial value of objects 2 and 5 with the coordinate values (4, 6) and (12, 4) as initial seeds.

Objects X - Coordinates Y - Coordinates

1 2 4
2 4 6
3 6 8
4 10 4
5 12 4

Solution:
As per the problem, choose the objects 2 and 5 with the coordinate values. Hereafter, the objects' id
is not important. The samples or data points (4, 6) and (12, 4) are started as two clusters as shown in Table

Cluster 1 Cluster 2
(4,6) (12,4)
Centroid 1 (4,6) Centroid 2 (12,4)
Iteration 1: Compare all the data points or samples with the centroid and assign to the nearest sample. Take
the sample object 1 (2, 4) from Table and compare with the centroid of the clusters in the table. The distance
is 0. Therefore, it remains in the same cluster. Similarly, consider the remaining samples. For the object 1
(2, 4), 3 (6, 8), 4 (10, 4) the Euclidean distance between it and the centroid is given as:

Object Coordinate Distance Close to

𝐷𝑖𝑠𝑡(1, 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 1) = √(2 − 4)2 + (4 − 6)2 = √8

1 (2,4) Cluster 1
𝐷𝑖𝑠𝑡(1, 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 2) = √(2 − 12)2 + (4 − 4)2 = 10

𝐷𝑖𝑠𝑡(3, 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 1) = √(6 − 4)2 + (8 − 6)2 = √8

3 (6,8) Cluster 1
𝐷𝑖𝑠𝑡(3, 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 2) = √(6 − 12)2 + (8 − 4)2 = √52
𝐷𝑖𝑠𝑡(4, 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 1) = √(10 − 4)2 + (4 − 6)2 = √40
4 (10,4) Cluster 2
𝐷𝑖𝑠𝑡(4, 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 2) = √(10 − 12)2 + (4 − 4)2 = 2

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Cluster Table after Iteration 1.

Cluster Table 1

Cluster 1 Cluster 2
(4,6) (10,4)
(2,4) (12,4)
(6,8)

4 + 2 + 6, 6 + 4 + 8 10 + 12, 4 + 4
𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 1 = ( ) = (4,6) 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 2 = ( ) = (11,4)
3 2

Iteration 2: Second iteration started with the above table.

Obviously, the point (4, 6) remains in cluster 1, as the distance of it with itself is 0. The remaining objects
can be checked. Take the sample object 1 (2, 4), 3 (6, 8), 4 (10, 4) the Euclidean distance between it and
the centroid is given as:

Object Coordinate Distance Close to

𝐷𝑖𝑠𝑡(1, 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 1) = √(2 − 4)2 + (4 − 6)2 = √8

1 (2,4) Cluster 1
𝐷𝑖𝑠𝑡(1, 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 2) = √(2 − 11)2 + (4 − 4)2 = 9

𝐷𝑖𝑠𝑡(3, 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 1) = √(6 − 4)2 + (8 − 6)2 = √8

3 (6,8) Cluster 1
𝐷𝑖𝑠𝑡(3, 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 2) = √(6 − 11)2 + (8 − 4)2 = 41
𝐷𝑖𝑠𝑡(4, 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 1) = √(10 − 4)2 + (4 − 6)2 = √40
4 (10,4) Cluster 2
𝐷𝑖𝑠𝑡(4, 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 2) = √(10 − 11)2 + (4 − 4)2 = 1

The final cluster table is given below

Cluster Table 2

Cluster 1 Cluster 2
(4,6) (10,4)
(2,4) (12,4)
(6,8)

4 + 2 + 6, 6 + 4 + 8 10 + 12, 4 + 4
𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 1 = ( ) = (4,6) 𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 2 = ( ) = (11,4)
3 2

There is no change in the cluster Table 1 & 2. It is exactly the same; therefore, the k-means algorithm
terminates with two clusters with data points as shown in the Table 2.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Core Strengths:
• Simplicity and Interpretability: K-Means is easy to understand and implement. The cluster
centroids provide a simple representation of each cluster.
• Efficiency: It is computationally efficient and scales relatively well to large datasets, especially
compared to hierarchical methods, when K is small. Its complexity is approximately O(n⋅K⋅i⋅d),
where n is data points, K clusters, i iterations, d dimensions.
• Effectiveness: Often produces good results for spherical clusters that are well-separated.
• Guaranteed Convergence: The algorithm is guaranteed to converge to a local optimum.

Objective and drawback of K-Means Clustering

Objective:
The primary objective of the K-means clustering algorithm is to partition a dataset into K distinct, non-
overlapping clusters, where K is a pre-defined number. This partitioning is done in such a way as to
minimize the within-cluster sum of squares (WCSS), also known as inertia.
Mathematically WCSS is represented as

𝑊𝐶𝑆𝑆 = ∑𝑘𝑖=1 ∑𝑥∈𝑐𝑖 ∥ 𝑥 − 𝜇𝑖 ∥2

Where:

• K is the number of clusters.

• Ci represents the ith cluster.
• x is a data point belonging to cluster Ci .
• μi is the centroid (mean) of cluster Ci .
• ∥ 𝑥 − 𝜇𝑖 ∥2 is the squared Euclidean distance between data point x and centroid 𝜇𝑖 .

The algorithm iteratively moves centroids and reassigns points to reduce this total sum of squared distances,
aiming for compact and well-separated clusters

In simpler terms, the goals are:

• Group similar data points together: Data points within the same cluster should be as similar as
possible to each other.
• Separate dissimilar data points: Data points in different clusters should be as dissimilar as possible.
• Minimize the distance between data points and their cluster centroid: Each cluster is represented
by a "centroid" (its mean), and the algorithm aims to make data points as close as possible to their
assigned cluster's centroid. This effectively makes the clusters compact.
• Maximize the distance between different cluster centroids: This ensures that the clusters are well-
separated and distinct from one another.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Drawback (Curse)

Curse of Dimensionality: This refers to various phenomena that arise when analysing and organizing data
in high-dimensional spaces that do not occur in low-dimensional settings. For K-Means, its impact includes:

• Distance Metric Effectiveness: In high dimensions, the concept of "distance" becomes less
meaningful. The relative difference between the nearest and farthest points becomes less pronounced.
Most points tend to be roughly equidistant from each other, making it hard for algorithms like K-Means
(which rely on distance) to distinguish between similar and dissimilar points.

• Sparsity: Data points become extremely sparse in high-dimensional spaces. There's less "local"
density, making it difficult to form compact clusters.

• Increased Computational Cost: Calculating distances in high dimensions takes more time.

• Increased Noise Sensitivity: Irrelevant dimensions can dominate the distance calculations, masking
the true underlying clusters.

Advantages and disadvantages of K-Means clustering

Advantages of K-Means Clustering:

• Simplicity and Speed: Relatively easy to understand and implement. It is computationally

efficient, especially for large datasets and a moderate number of clusters. O(n⋅K⋅i⋅d) complexity.
• Scalability: Can be applied to large datasets with many data points.
• Guaranteed Convergence: The algorithm is guaranteed to converge to a local optimum.
• Well-Defined Centroids: The cluster centroids provide a natural and intuitive representation of
each cluster.
• Easy to Interpret: The resulting clusters are often straightforward to interpret if the data is well-
behaved.

Disadvantages of K-Means Clustering:

• Requires Pre-defined Number of Clusters (K): This is often the biggest challenge. Determining
the optimal K is subjective and can require external methods (like the Elbow method or Silhouette
analysis).
• Sensitive to Initial Centroid Placement: Can converge to suboptimal local optima depending on
the initial centroid selection. Running multiple initializations is often recommended.
• Struggles with Non-Globular Cluster Shapes: K-Means implicitly assumes clusters are
spherical and equal in size and density. It performs poorly with clusters of irregular shapes (e.g.,
elongated, crescent-shaped) or varying densities, as it tries to find circular boundaries around
centroids.
• Sensitive to Outliers: Outliers can significantly skew the centroid positions, leading to
misleading clusters, as squared distances amplify their effect.
• Requires Numerical Data: Typically works only with numerical data. Categorical or mixed-type
data requires specific encoding or specialized K-Means variants.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Hierarchical clustering
Hierarchical clustering is used to group similar data points together based on their similarity creating
a hierarchy or tree-like structure. The key idea is to begin with each data point as its own separate cluster
and then progressively merge or split them based on their similarity. Let’s understand this with the help of
an example
Imagine you have four fruits with different weights: an apple (100g), a banana (120g), a cherry (50g) and
a grape (30g). Hierarchical clustering starts by treating each fruit as its own group.
• It then merges the closest groups based on their weights.
• First the cherry and grape are grouped together because they are the lightest.
• Next the apple and banana are grouped together.

Finally, all the fruits are merged into one large group, showing how hierarchical clustering progressively
combines the most similar data points.

Hierarchical clustering is an unsupervised machine learning algorithm that groups similar data points into
clusters, forming a hierarchy of clusters rather than a flat partitioning (like K-means). The result of
hierarchical clustering is typically visualized as a dendrogram, a tree-like diagram that illustrates the
arrangement of clusters and the sequence in which they were merged or split.

Dendrogram
A dendrogram is like a family tree for clusters. It shows how individual data points or groups of data
merge together. The bottom shows each data point as its own group, and as you move up, similar groups
are combined. The lower the merge point, the more similar the groups are. It helps you see how things are
grouped step by step. The working of the dendrogram can be explained using the below diagram:

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

In the above image on the left side there are five points labeled P, Q, R, S and T. These represent individual
data points that are being clustered. On the right side there’s a dendrogram which show how these points
are grouped together step by step.
• At the bottom of the dendrogram the points P, Q, R, S and T are all separate.
• As you move up, the closest points are merged into a single group.
• The lines connecting the points show how they are progressively merged based on similarity.
• The height at which they are connected shows how similar the points are to each other; the shorter
the line the more similar they are.

Types of Hierarchical Clustering

There are two main approaches to hierarchical clustering:

Agglomerative (Bottom-Up) Clustering:

• This is the most common type.
• It starts by treating each data point as an individual cluster.
• It then iteratively merges the two closest clusters until all data points belong to a single, large
cluster or a stopping condition is met.
• Think of it like building a tree from its leaves up to its root.

Workflow for Hierarchical Agglomerative clustering

1. Start with individual points: Each data point is its own cluster. For example, if you have 5 data
points you start with 5 clusters each containing just one data point.
2. Calculate distances between clusters: Calculate the distance between every pair of clusters.
Initially since each cluster has one point this is the distance between the two data points.
3. Merge the closest clusters: Identify the two clusters with the smallest distance and merge them
into a single cluster.
4. Update distance matrix: After merging you now have one less cluster. Recalculate the distances
between the new cluster and the remaining clusters.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

5. Repeat steps 3 and 4: Keep merging the closest clusters and updating the distance matrix until
you have only one cluster left.
6. Create a dendrogram: As the process continues you can visualize the merging of clusters using a
tree-like diagram called a dendrogram. It shows the hierarchy of how clusters are merged.
Python implementation of the above algorithm using the scikit-learn library:
from [Link] import AgglomerativeClustering
import numpy as np

X = [Link]([[1, 0], [1, 2], [1, 4],

[4, 0], [4, 2], [4, 4]])

clustering = AgglomerativeClustering(n_clusters = 2).fit(X)

print(clustering.labels_)

Output :
[1, 1, 1, 0, 0, 0]
This means:
• Points [1,0], [1,2], [1,4] → assigned to cluster 1
• Points [4,0], [4,2], [4,4] → assigned to cluster 0

Divisive (Top-Down) Clustering:

• This approach works in the opposite direction of agglomerative clustering.
• It starts with all data points in a single, large cluster.
• It then recursively splits the most dissimilar clusters into smaller ones until each data point forms
its own individual cluster or a stopping criterion is met.
• This is like starting with the tree's root and progressively splitting it into smaller branches and
leaves. Divisive clustering is less common in practice due to its higher computational complexity.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Workflow for Hierarchical Divisive clustering:

1. Start with all data points in one cluster: Treat the entire dataset as a single large cluster.
2. Split the cluster: Divide the cluster into two smaller clusters. The division is typically done by
finding the two most dissimilar points in the cluster and using them to separate the data into two
parts.
3. Repeat the process: For each of the new clusters, repeat the splitting process:
1. Choose the cluster with the most dissimilar points.
2. Split it again into two smaller clusters.
4. Stop when each data point is in its own cluster: Continue this process until every data point is its
own cluster, or the stopping condition (such as a predefined number of clusters) is met.

Advantages of Hierarchical Clustering:

1. No need to pre-specify the number of clusters (K): Unlike K-means, you don't need to choose
'K' beforehand. You can decide on the number of clusters by observing the dendrogram.
2. Provides a hierarchical view: The dendrogram offers insights into the relationships and nested
structures between clusters at different levels of granularity. This is very useful for exploratory
data analysis.
3. Flexibility in distance and linkage metrics: Allows customization based on the nature of the
data and the desired cluster shapes.
4. Can uncover nested clusters: Its hierarchical nature allows for the identification of sub-clusters
within larger clusters.

Disadvantages of Hierarchical Clustering:

1. Computationally Intensive: For N data points, it requires calculating an N x N distance matrix,
leading to a time complexity of at least O(N2logN) or O(N3) depending on the linkage method.
This makes it less scalable for very large datasets compared to K-means.
2. Memory Requirements: Storing the distance matrix can be memory-intensive (O(N2) space
complexity).
3. Sensitive to Noise and Outliers: Because it starts with individual points, outliers can
significantly influence the cluster formation and dendrogram structure.
4. Irrevocable Decisions: Once a merger or split is made, it cannot be undone in subsequent steps. A
poorly chosen merger early on can propagate and affect the final clustering structure.
5. Difficulty in identifying optimal number of clusters: While the dendrogram is helpful, choosing
the "right" cut-off point can still be subjective.
6. No objective function directly optimized: Unlike K-means which minimizes WCSS,
hierarchical clustering doesn't explicitly optimize a global objective function in the same way,
making it harder to quantitatively compare different hierarchical clustering solutions.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Evaluation of clustering results

Evaluating clustering results involves assessing how well the data has been grouped into clusters. This can
be done using both external and internal validation methods. External validation compares the clustering
results to known ground truth labels, while internal validation uses only the data itself to assess the quality
of the clusters. Common metrics include silhouette score, Davies-Bouldin index, and Adjusted Rand
Index(ARI).
Here's a breakdown of how to evaluate clustering results, categorized into internal, external, and relative
validation, along with techniques for interpreting them:

Internal Validation (No Ground Truth)

These metrics assess the quality of the clustering based solely on the data itself, without relying on external
labels. They generally focus on two main aspects:
• Compactness (Cohesion): How close data points within a cluster are to each other.
• Separation (Isolation): How distinct or far apart different clusters are from each other.
Common internal metrics are:
1. Silhouette Score
2. Davies-Bouldin Index (DBI)

Silhouette Score
The Silhouette Score is a way to measure how good the clusters are in a dataset. It helps us understand
how well the data points have been grouped. The score ranges from -1 to 1.
• A score close to 1 means a point fits really well in its group (cluster) and is far from other groups.
• A score close to 0 means the point is on the border between two clusters.
• A score close to -1 means the point might be in the wrong cluster.
Silhouette Score (S) for a data point i is calculated as:

𝑏(𝑖) − 𝑎(𝑖)
𝑆(𝑖) =
max( 𝑎(𝑖), 𝑏(𝑖) )
where,
• 𝑎(𝑖) is the average distance from i to other data points in the same cluster.
• 𝑏(𝑖) is the smallest average distance from i to data points in a different cluster.

Davies-Bouldin Index (DBI)

The Davies-Bouldin Index (DBI) helps us measure how good the clustering is in a dataset. It looks at how
tight each cluster is (compactness), and how far apart the clusters are (separation).
• Lower DBI = better, clearer clusters
• Higher DBI = messy, overlapping clusters

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

A lower score is better, because it means:

• Points in the same cluster are close to each other.
• Different clusters are far apart from one another.
Davies-Bouldin Index (DB) is calculated as:
𝑘
1 𝑅𝑖𝑖 + 𝑅𝑗𝑗
𝐷𝐵 = ∑ 𝑚𝑎𝑥𝑗≠𝑖 ( )
𝑘 𝑅𝑖𝑗
𝑖=1
where,
• k is the total number of clusters.
• 𝑅𝑖𝑖 is the compactness of cluster i.
• 𝑅𝑗𝑗 is the compactness of cluster j.

• 𝑅𝑖𝑗 is the dissimilarity (distance) between cluster i and cluster j.

External Validation (With Ground Truth/True Labels)

When you have some prior knowledge about the true grouping of your data (e.g., class labels from a
classification task), you can use external metrics to compare your clustering results against this "ground
truth."
The important external metric is,

Adjusted Rand Index (ARI)

The Adjusted Rand Index (ARI) helps us measure how accurate a clustering result is by comparing it to
the true labels (ground truth).
It checks how well the pairs of points are grouped:
• Are the same pairs together in both the real and predicted clusters?
• Are different pairs also kept apart correctly?
The score ranges from -1 to 1:
• 1 means perfect match - the clustering is exactly right.
• 0 means random guess - no better than chance.
• Below 0 means worse than random - very poor clustering.
Adjusted Rand Index (ARI) is calculated as:
𝑅𝐼 − 𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑𝑅𝐼
𝐴𝑅𝐼 =
max(𝑅𝐼) − 𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑𝑅𝐼
where,
• RI is the Rand Index.
• 𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑𝑅𝐼 is the expected value of the Rand Index.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

DIMENSIONALITY REDUCTION

Dimensionality reduction is a fundamental technique in machine learning and data analysis that involves
transforming data from a high-dimensional space into a low-dimensional space while retaining as much
meaningful information as possible. It's crucial for addressing various challenges posed by high-
dimensional datasets.
Dimensionality reduction is a process that simplifies complex dataset by combining similar or correlated
features. It helps in improving analysis and computational efficiency.

Example: when you are building a model to predict house prices with features like bedrooms, square
footage and location. If you add too many features such as room condition or flooring type, the dataset
becomes large and complex.
Why is Dimensionality Reduction Necessary?
• Curse of Dimensionality: As the number of dimensions (features) in a dataset increases, the data
points become increasingly sparse. This makes it difficult for machine learning algorithms to find
meaningful patterns, leading to poor model performance and generalization.
• Computational Efficiency: High-dimensional data requires more computational resources (time
and memory) for storage, processing, and model training. Reducing dimensions can significantly
speed up these processes.
• Overfitting Prevention: Models trained on high-dimensional data are more prone to overfitting,
where they learn the noise in the training data rather than the true underlying patterns.
Dimensionality reduction can help mitigate overfitting by removing redundant or irrelevant features.
• Visualization Challenges: It's practically impossible to visualize data with more than three
dimensions. Dimensionality reduction allows us to project data into 2D or 3D, making it easier to
explore patterns, clusters, and relationships.
• Noise Reduction: High-dimensional datasets often contain noisy or irrelevant features that can
obscure the true signal. Dimensionality reduction can help filter out this noise, improving the data's
signal-to-noise ratio.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Methods of Dimensionality Reduction

Dimensionality reduction techniques generally fall into two main categories:
1. Feature Selection
2. Feature Extraction
• Principal Component Analysis (PCA)
• Linear Discriminant Analysis (LDA)
• t-Distributed Stochastic Neighbour Embedding (t-SNE)

Principal Component Analysis (PCA)

PCA (Principal Component Analysis) is a dimensionality reduction technique used in data analysis and
machine learning. It helps you to reduce the number of features in a dataset while keeping the most
important information. It changes original features into new features these new features don’t overlap
with each other and the first few keep most of the important differences found in the original data.

How Principal Component Analysis Works

PCA uses linear algebra to transform data into new features called principal components. It finds these by
calculating eigenvectors (directions) and eigenvalues (importance) from the covariance matrix. PCA selects
the top components with the highest eigenvalues and projects the data onto them simplify the dataset.
The Core Idea: Finding New Axes of Maximum Variance
Imagine your data points scattered in a multi-dimensional space. PCA's goal is to find a new set of
orthogonal (uncorrelated) axes, called principal components (PCs), along which the data varies the most.
• First Principal Component (PC1): This component points in the direction of the greatest variance
in the data. If you project all your data points onto this axis, they will be most spread out. This
means PC1 captures the most significant patterns or information.
• Second Principal Component (PC2): This component is orthogonal to PC1 and captures the next
greatest amount of variance remaining in the data.
• Subsequent Principal Components: Each subsequent principal component is orthogonal to all
preceding ones and captures decreasing amounts of variance.
By selecting only the top few principal components, you can project your high-dimensional data onto a
lower-dimensional space, effectively reducing the number of features while minimizing information loss.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Steps Involved in PCA:

1. Standardize the Data:
• PCA adjusts each measurement so they all have an average of zero and a consistent
spread, making sure every feature contributes fairly to finding the main patterns.
2. Compute the Covariance Matrix:
• The covariance matrix describes the relationships (covariances) between all pairs of
features in your standardized dataset.
3. Compute Eigenvalues and Eigenvectors of the Covariance Matrix:
• This is the core mathematical step of PCA.
• Eigenvectors: These are the principal components themselves. Each eigenvector
represents a direction (axis) in the data space. They are orthogonal to each other.
• Eigenvalues: Each eigenvalue corresponds to an eigenvector and represents the amount
of variance captured by that principal component. A larger eigenvalue means the
corresponding eigenvector captures more variance.
4. Sort Eigenvalues and Select Principal Components:
• Sort the eigenvalues in descending order. This also sorts their corresponding
eigenvectors, so the first eigenvector (PC1) will have the largest eigenvalue, and so on.
5. Transform the Data:
• Create a projection matrix (also called a "feature vector" or "weights matrix") using the
selected top k eigenvectors. Repeat the process.

Advantages of PCA in Dimensionality Reduction:

• Reduces Redundancy: By transforming correlated features into uncorrelated principal
components, PCA addresses multicollinearity.
• Computational Efficiency: Lower-dimensional data requires less storage and faster computation
for machine learning algorithms.
• Noise Reduction: By focusing on the components with high variance, PCA can effectively filter
out noise, which often resides in lower-variance dimensions.
• Improved Model Performance: Reduced dimensionality can prevent overfitting and lead to more
robust models, especially with high-dimensional datasets.
• Visualization: It allows for the visualization of high-dimensional data by projecting it onto 2D or
3D principal components, making patterns and clusters more apparent.
Limitations of PCA:
• Linearity Assumption: PCA is a linear technique. If the underlying relationships in the data are
highly non-linear, PCA might not effectively capture the true structure.
• Interpretability: The principal components are linear combinations of the original features, which
can make them difficult to interpret in terms of their real-world meaning.
• Sensitivity to Outliers: PCA is sensitive to outliers, as they can disproportionately affect the
calculation of variance and covariance

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

• Requires Standardization: As mentioned, data must be standardized before applying PCA, which
adds an extra preprocessing step.
• Information Loss: While PCA aims to minimize information loss, some information is inevitably
lost when reducing dimensions, especially if not enough components are retained.

Applications:
PCA is widely used across various domains:
• Image Compression and Processing: Reducing the number of pixels or features in images while
preserving visual quality.
• Face Recognition (Eigenfaces): Representing faces as a combination of principal components.
• Bioinformatics: Analysing gene expression data, which often has thousands of features.
• Financial Data Analysis: Identifying principal factors driving stock market movements.
• Anomaly Detection: Detecting unusual patterns in high-dimensional data by projecting it into a
lower-dimensional space where anomalies might become more apparent.
• Exploratory Data Analysis: Gaining insights into the structure and relationships within complex
datasets.
In essence, PCA is a powerful and versatile tool for simplifying complex datasets, making them more
manageable for analysis, visualization, and machine learning tasks.

Linear Discriminant Analysis (LDA)

When working with high-dimensional datasets it is important to apply dimensionality reduction techniques
to make data exploration and modeling more efficient. Linear Discriminant Analysis (LDA) also known
as Normal Discriminant Analysis is unsupervised classification problem that helps separate two or
more classes by converting higher-dimensional data space into a lower-dimensional space. It is used
to identify a linear combination of features that best separates classes within a dataset.

For example we have two classes that need to be separated efficiently. Each class may have multiple features
and using a single feature to classify them may result in overlapping. To solve this LDA is used as it uses
multiple features to improve classification accuracy. LDA works by some assumptions and we are
required to understand them so that we have a better understanding of its working.

How Does LDA Work (Intuition)?

Imagine we have data points from two different classes scattered in a 2D plane. If we try to classify them
directly, they might overlap. LDA tries to find a line (or a hyperplane in higher dimensions) such that when
you project the data points onto this line, the points from different classes are as far apart as possible, and
the points within the same class are as close together as possible.
Image shows an example where the classes (black and green circles) are not linearly separable. LDA
attempts to separate them using red dashed line. It uses both axes (X and Y) to generate a new axis in

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

such a way that it maximizes the distance between the means of the two classes while minimizing the
variation within each class. This transforms the dataset into a space where the classes are better separated.
After transforming the data points along a new axis LDA maximizes the class separation. This new axis
allows for clearer classification by projecting the data along a line that enhance the distance between the
means of the two classes.

Perpendicular distance between the decision boundary and the data points helps us to visualize how LDA
works by reducing class variation and increasing separability.

After generating this new axis using the above-mentioned criteria all the data points of the classes are
plotted on this new axis and are shown in the figure given below.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

It shows how LDA creates a new axis to project the data and separate the two classes effectively along a
linear path. But it fails when the mean of the distributions are shared as it becomes impossible for LDA to
find a new axis that makes both classes linearly separable. In such cases we use non-linear discriminant
analysis.

Advantages of LDA
• Simple and computationally efficient.
• Works well even when the number of features is much larger than the number of training samples.
• Can handle multicollinearity.

Disadvantages of LDA
• Assumes Gaussian distribution of data which may not always be the case.
• Assumes equal covariance matrices for different classes which may not hold in all datasets.
• Assumes linear separability which is not always true.
• May not always perform well in high-dimensional feature spaces.

Applications of LDA
1. Face Recognition: It is used to reduce the high-dimensional feature space of pixel values in face
recognition applications helping to identify faces more efficiently.
2. Medical Diagnosis: It classifies disease severity in mild, moderate or severe based on patient
parameters helping in decision-making for treatment.
3. Customer Identification: It can help identify customer segments most likely to purchase a specific
product based on survey data.

T-Distributed Stochastic Neighbour Embedding (t-SNE)

T-Distributed Stochastic Neighbour Embedding (t-SNE) is a powerful, non-linear dimensionality reduction
technique widely used for visualizing high-dimensional data in a lower-dimensional space, typically 2D
or 3D.
What is t-SNE?
• Dimensionality Reduction: It's a method to transform data from a high-dimensional space (many
features/variables) into a lower-dimensional space (fewer features) while preserving the underlying
structure.
• Non-linear: Unlike linear methods like Principal Component Analysis (PCA), t-SNE can capture
complex, non-linear relationships within the data, making it particularly effective for visualizing
datasets where simple straight lines cannot separate clusters.
• Visualization Tool: Its primary purpose is to help humans understand and explore complex datasets
by providing an intuitive visual representation of how data points relate to each other. Similar data

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

points are mapped to nearby points in the low-dimensional space, and dissimilar points are mapped
far apart.

How does it work?

t-SNE operates in two main stages:
1. Constructing Probability Distributions in High-Dimensional Space:

• For each data point, t-SNE calculates the probability that other data points are its neighbors. This
is done using a Gaussian distribution, where points closer to the central point have a higher
probability of being considered neighbors.
• The "perplexity" parameter plays a crucial role here. It can be thought of as a smooth measure of
the effective number of neighbors each point has. A higher perplexity value considers more
neighbors and balances attention between local and global aspects of the data.
2. Creating a Low-Dimensional Embedding:

• t-SNE then creates a similar probability distribution over the points in a low-dimensional map
(e.g., 2D).
• It uses a Student's t-distribution (which has heavier tails than a Gaussian) to measure similarities
in the low-dimensional space. The heavy tails help to alleviate the "crowding problem" (where
distant points in high-dimensional space might be squeezed together in low-dimensional space).
• The algorithm then iteratively adjusts the positions of the points in the low-dimensional space to
minimize the difference (Kullback-Leibler divergence) between the two probability distributions
(high-dimensional and low-dimensional). This optimization is typically performed using gradient
descent.
• The goal is to ensure that similar objects in high-dimensional space are modelled by nearby points
in the low-dimensional map, and dissimilar objects are modelled by distant points.

Advantages of t-SNE
• Great for Visualization: t-SNE is particularly used to convert complex high-dimensional data into
2D or 3D for visualization making patterns and clusters easy to observe.
• Preserve Local Structure: Unlike linear techniques like PCA t-SNE focus on maintaining the local
relationships between data points meaning similar data points remain close in the lower-dimensional
space.
• Non-Linear Capability: It captures non-linear dependencies in the data which makes it suitable for
complex datasets where linear methods fail.
• Cluster Separation: Helps in clearly visualizing clusters and class separability in datasets like
MNIST making it easier for interpretation and exploration.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Disadvantages of t-SNE
• Computationally Intensive: t-SNE is slower and more computationally expensive compared to
linear methods especially on large datasets.
• Non-deterministic Output: The output can vary with each run due to its randomness unless a fixed
random_state is used.
• Not Scalable for Large Datasets: It struggles with very large datasets (e.g., millions of points)
unless optimized or approximated versions are used.
• No Global Structure Preservation: It keeps nearby things close together but might make far-apart
things seem weirdly spaced.

Applications in Machine Learning:

• Data Visualization: This is the most common and powerful application. It helps in
understanding the underlying structure of complex datasets, identifying clusters, and spotting
outliers.
• Clustering Analysis: While not a clustering algorithm itself, t-SNE can visually reveal natural
groupings in data, which can then guide or validate clustering algorithms.
• Anomaly Detection: Outliers, which are distinct from the main clusters, often stand out in t-
SNE plots.
• Natural Language Processing (NLP): Visualizing word embeddings to understand semantic
relationships between words.
• Image Analysis: Visualizing high-dimensional image features (e.g., from Convolutional Neural
Networks) to understand how images are grouped.
• Bioinformatics and Genomics: Visualizing gene expression data or single-cell RNA
sequencing data to identify cell types or disease states.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

ASSOCIATION RULE LEARNING

Association Rule Learning is an unsupervised machine learning technique used for discovering interesting
relationships and patterns between variables in large datasets. It's a type of unsupervised learning that aims
to find patterns and correlations within data. It's particularly powerful for identifying "if-then" relationships,
often expressed as rules of the form X→Y, where X and Y are disjoint sets of items. This means that if a
customer buys items in set X, they are likely to also buy items in set Y
The primary goal is to find strong rules that reveal how items frequently co-occur in transactions or
observations.

How it Works:
Association rule learning typically involves three main stages:
1. Identifying Frequent Itemsets: This first step involves finding all combinations of items (itemsets)
that appear together in a dataset with a frequency above a predefined threshold. This threshold is
called Support.

• Support: For an itemset, support is the proportion of transactions in the dataset that contain
that itemset. A higher support means the itemset appears more frequently.
2. Generating Association Rules: Once the frequent itemsets are identified, association rules are
generated from these itemsets. To determine the "strength" or "interestingness" of these rules,
additional metrics are used:
3. Evaluate Rule Strength: Metrics like confidence and lift are used to evaluate the reliability and
strength of the generated rules.

• Confidence: For a rule X→Y, confidence measures the reliability of the inference. It's the
conditional probability that Y will be in a transaction, given that X is already in the transaction.
In simpler terms, it tells you how often itemset Y is purchased when itemset X is also
purchased.
• Lift: Lift measures how much more likely the consequent item (Y) is to be purchased when
the antecedent item (X) is present, compared to its individual occurrence rate.
▪ A lift value of 1 indicates that X and Y are independent.
▪ A lift value greater than 1 suggests a positive association (the presence of X increases the
likelihood of Y).
▪ A lift value less than 1 indicates a negative association (the presence of X decreases the
likelihood of Y).

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Example of Conditional Probability

Suppose you draw one card at random from a standard deck of 52 playing cards.
Let’s define two events:
• A: The card drawn is a King
• B: The card drawn is a face card (Jack, Queen, or King)
There are:
• 4 Kings in the deck
• 12 face cards in total (4 Jacks, 4 Queens, 4 Kings)
Now, what is the conditional probability that the card is a King, given that it is a face
card?
This is written as:
𝑃(𝐴 ∩ 𝐵)
𝑃(𝐴|𝐵) =
𝑃(𝐵)
Since every King is also a face card:
4
𝑃(𝐴 ∩ 𝐵) = 𝑃(𝐾𝑖𝑛𝑔) =
52
12
𝑃(𝐵) = 𝑃(𝐹𝑎𝑐𝑒 𝐶𝑎𝑟𝑑) =
52
4⁄ 1
𝑃(𝐴|𝐵) = 52 =
12⁄ 3
52

Key Algorithms:
Several algorithms are used to efficiently find association rules, most important is:
• Apriori Algorithm: This is one of the most well-known algorithms. It uses an iterative, breadth-
first search approach. Its core principle (Apriori property) is that if an itemset is frequent, then all
of its subsets must also be frequent. This helps to prune the search space and reduce computational
cost.
Applications of Association Rule Learning:
Association Rule Learning has a wide range of practical applications across various domains:
• Market Basket Analysis: This is the most classic application. Retailers use it to understand
customer purchasing habits by identifying which products are frequently bought together (e.g.,
"customers who buy bread and butter also tend to buy milk"). This information is used for:

• Optimizing store layout and product placement.

• Cross-selling and up-selling strategies.
• Designing promotional offers and product bundles.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

• Web Usage Mining (Clickstream Analysis): Analysing user navigation patterns on websites to
improve website design, personalize content, and make recommendations.
• Healthcare and Medical Diagnosis: Identifying co-occurrence patterns in symptoms, diseases, and
treatments to aid in diagnosis, predict complications, and understand drug interactions.
• Fraud Detection: Detecting unusual patterns in transactions (e.g., credit card fraud) that deviate
from established norms.
• Recommendation Systems: Suggesting items to users based on their past behaviour or the
behaviour of similar users (e.g., "people who watched this movie also watched...").
• Bioinformatics: Discovering relationships between genes, proteins, or other biological entities.
• Customer Segmentation: Grouping customers based on their purchasing habits to tailor marketing
campaigns.

Apriori algorithm
Apriori Algorithm is a basic method used in data analysis to find groups of items that often appear together
in large sets of data. It helps to discover useful patterns or rules about how items are related which is
particularly valuable in market basket analysis.
Like in a grocery store if many customers buy bread and butter together, the store can use this information
to place these items closer or create special offers. This helps the store sell more and make customers happy.

How the Apriori Algorithm Works?

The Apriori Algorithm operates through a systematic process that involves several key steps:
1. Identifying Frequent Itemsets
• The Apriori algorithm starts by looking through all the data to count how many times each single
item appears. These single items are called 1-itemsets.
• Next it uses a rule called minimum support this is a number that tells us how often an item or group
of items needs to appear to be important. If an item appears often enough meaning its count is above
this minimum support it is called a frequent itemset.
2. Creating Possible item group
• After finding the single items that appear often enough (frequent 1-item groups) the algorithm
combines them to create pairs of items (2-item groups). Then it checks which pairs are frequent by
seeing if they appear enough times in the data.
• This process keeps going step by step making groups of 3 items, then 4 items and so on. The
algorithm stops when it can’t find any bigger groups that happen often enough.
3. Removing Infrequent Item groups
• The Apriori algorithm uses a helpful rule to save time. This rule says: If a small group of items
isn't common, then any bigger group containing them won't be common either.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

• Because of this, the algorithm does not check those larger groups. This way it avoids wasting time
looking at groups that won’t be important make the whole process faster.
4. Generating Association Rules
• The algorithm makes rules to show how items are related.
• It checks these rules using support, confidence and lift to find the strongest ones.

Key Metrics of Apriori Algorithm

• Support: This metric measures how frequently an item appears in the dataset relative to the total
number of transactions. A higher support indicates a more significant presence of the itemset in the
dataset. Support tells us how often a particular item or combination of items appears in all the
transactions ("Bread is bought in 20% of all transactions.")
• Confidence: Confidence assesses the likelihood that an item Y is purchased when item X is
purchased. It provides insight into the strength of the association between two items. Confidence
tells us how often items go together. ("If bread is bought, butter is bought 75% of the time.")
• Lift: Lift evaluates how much more likely two items are to be purchased together compared to being
purchased independently. A lift greater than 1 suggests a strong positive association. Lift shows how
strong the connection is between items. ("Bread and butter are much more likely to be bought
together than by chance.")

Example:
Let’s understand the concept of Apriori Algorithm with the help of an example. Consider the following
dataset and we will find frequent itemsets and generate association rules for them:

Transactions of a Grocery Shop

Step 1 : Setting the parameters
• Minimum Support Threshold: 50% (item must appear in at least 3/5 transactions). This threshold
is formulated from this formula:
• Minimum Confidence Threshold: 70% (You can change the value of parameters as per the use
case and problem statement).

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Step 2: Find Frequent 1-Itemsets

Lets count how many transactions include each item in the dataset (calculating the frequency of each item).

Frequent 1-Itemsets
All items have support% ≥ 50%, so they qualify as frequent 1-itemsets. if any item has support% < 50%, It
will be omitted out from the frequent 1- itemsets.
Step 3: Generate Candidate 2-Itemsets
Combine the frequent 1-itemsets into pairs and calculate their support. For this use case we will get 3 item
pairs ( bread,butter) , (bread,ilk) and (butter,milk) and will calculate the support similar to step 2

Candidate 2-Itemsets
Frequent 2-itemsets: {Bread, Milk} meet the 50% threshold but {Butter, Milk} and {Bread ,Butter}
doesn't meet the threshold, so will be committed out.
Step 4: Generate Candidate 3-Itemsets
Combine the frequent 2-itemsets into groups of 3 and calculate their support. for the triplet we have only
got one case i.e {bread,butter,milk} and we will calculate the support.

Candidate 3-Itemsets
Since this does not meet the 50% threshold, there are no frequent 3-itemsets.
Step 5: Generate Association Rules
Now we generate rules from the frequent itemsets and calculate confidence.
Rule 1: If Bread → Butter (if customer buys bread, the customer will buy butter also)

• Support of {Bread, Butter} = 2.

• Support of {Bread} = 4.
• Confidence = 2/4 = 50% (Failed threshold).
Rule 2: If Butter → Bread (if customer buys butter, the customer will buy bread also)

• Support of {Bread, Butter} = 3.

• Support of {Butter} = 3.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

• Confidence = 3/3 = 100% (Passes threshold).

Rule 3: If Bread → Milk (if customer buys bread, the customer will buy milk also)

• Support of {Bread, Milk} = 3.

• Support of {Bread} = 4.
• Confidence = 3/4 = 75% (Passes threshold).
The Apriori Algorithm, as demonstrated in the bread-butter example, is widely used in modern startups like
Zomato, Swiggy and other food delivery platforms. These companies use it to perform market basket
analysis which helps them identify customer behaviour patterns and optimise recommendations.

Applications of Apriori Algorithm

Below are some applications of Apriori algorithm used in today's companies and startups
1. E-commerce: Used to recommend products that are often bought together like laptop + laptop bag,
increasing sales.
2. Food Delivery Services: Identifies popular combos such as burger + fries, to offer combo deals to
customers.
3. Streaming Services: Recommends related movies or shows based on what users often watch
together like action + superhero movies.
4. Financial Services: Analyses spending habits to suggest personalised offers such as credit card
deals based on frequent purchases.
5. Travel & Hospitality: Creates travel packages like flight + hotel by finding commonly purchased
services together.
6. Health & Fitness: Suggests workout plans or supplements based on users' past activities like
protein shakes + workouts.

Advantages of Apriori:
• Simplicity: Easy to understand and implement.
• Well-defined: The process is systematic and clear.
• Foundation: Forms the basis for many other association rule mining algorithms.
Disadvantages of Apriori:
• High Computational Cost: Can be very slow and computationally expensive when dealing with
large datasets and a large number of unique items, as it generates a huge number of candidate
itemsets.
• Multiple Scans of Database: Requires multiple passes over the database to count the support of
candidate itemsets, which can be inefficient.
• Memory Intensive: Storing candidate itemsets can consume a lot of memory.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Market Basket Analysis

A data mining technique that is used to uncover purchase patterns in any retail setting is known as Market
Basket Analysis. Basically, market basket analysis in data mining involves analyzing the combinations of
products that are bought together.
This is a technique that gives the careful study of purchases done by a customer in a supermarket. This
concept identifies the pattern of frequent purchase items by customers. This analysis can help to promote
deals, offers, sale by the companies, and data mining techniques helps to achieve this analysis task.

Example:
• Data mining concepts are in use for Sales and marketing to provide better customer service, to
improve cross-selling opportunities, to increase direct mail response rates.
• Customer Retention in the form of pattern identification and prediction of likely defections is
possible by Data mining.
• Risk Assessment and Fraud area also use the data-mining concept for identifying inappropriate or
unusual behaviour etc.

Market basket analysis mainly works with the ASSOCIATION RULE {IF} -> {THEN}.
• IF means Antecedent: An antecedent is an item found within the data
• THEN means Consequent: A consequent is an item found in combination with the antecedent.

Let's see ASSOCIATION RULE {IF} -> {THEN} rules used in Market Basket Analysis in Data Mining.
For example, customers buying a domain means they definitely need extra plugins/extensions to make it
easier for the users.
Like we said above Antecedent is the item sets that are available in data. By formulating from the rules
means {if} component and from the example is the domain.
Same as Consequent is the item that is found with the combination of Antecedents. By formulating from
the rules means {THEN} component and from the example is extra plugins/extensions.
With the help of these, we are able to predict customer behavioural patterns. From this, we are able to make
certain combinations with offers that customers will probably buy those products. That will automatically
increase the sales and revenue of the company.
With the help of the Apriori Algorithm, we can further classify and simplify the item sets which are
frequently bought by the consumer.
There are three components in APRIORI ALGORITHM:

• SUPPORT
• CONFIDENCE
• LIFT

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Now take an example, suppose 5000 transactions have been made through a popular eCommerce website.
Now they want to calculate the support, confidence, and lift for the two products, let's say pen and notebook
for example out of 5000 transactions, 500 transactions for pen, 700 transactions for notebook, and 1000
transactions for both.
SUPPORT: It is been calculated with the number of transactions divided by the total number of transactions
made,
CONFIDENCE: It is been calculated for whether the product sales are popular on individual sales or
through combined sales. That is calculated with combined transactions/individual transactions.
Confidence=freq(A,B)/freq(A)Confidence=freq(A,B)/freq(A)
Confidence = combine transactions/individual transactions
i.e confidence-> 1000/500=20 percent
LIFT: Lift is calculated for knowing the ratio for the sales.
Lift=confidencepercent/supportpercentLift=confidencepercent/supportpercent
Lift-> 20/10=2
When the Lift value is below 1 means the combination is not so frequently bought by consumers. But in
this case, it shows that the probability of buying both the things together is high when compared to the
transaction for the individual items sold.
With this, we come to an overall view of the Market Basket Analysis in Data Mining and how to calculate
the sales for combination products.
Types of Market Basket Analysis
There are three types of Market Basket Analysis. They are as follow:
1. Descriptive market basket analysis: This sort of analysis looks for patterns and connections in the
data that exist between the components of a market basket. This kind of study is mostly used to
understand consumer behavior, including what products are purchased in combination and what the
most typical item combinations. Retailers can place products in their stores more profitably by
understanding which products are frequently bought together with the aid of descriptive market
basket analysis.
2. Predictive Market Basket Analysis: Market basket analysis that predicts future purchases based
on past purchasing patterns is known as predictive market basket analysis. Large volumes of data
are analysed using machine learning algorithms in this sort of analysis in order to create predictions
about which products are most likely to be bought together in the future. Retailers may make data-
driven decisions about which products to carry, how to price them, and how to optimize shop layouts
with the use of predictive market basket research.
3. Differential Market Basket Analysis: Differential market basket analysis analyses two sets of
market basket data to identify variations between them. Comparing the behavior of various client
segments or the behavior of customers over time is a common usage for this kind of study. Retailers
can respond to shifting consumer behavior by modifying their marketing and sales tactics with the
help of differential market basket analysis.
Benefits of Market Basket Analysis

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

1. Enhanced Customer Understanding: Market basket research offers insights into customer
behavior, including what products they buy together and which products they buy the most
frequently. Retailers can use this information to better understand their customers and make
informed decisions.
2. Improved Inventory Management: By examining market basket data, retailers can determine
which products are sluggish sellers and which ones are commonly bought together. Retailers can
use this information to make well-informed choices about what products to stock and how to manage
their inventory most effectively.
3. Better Pricing Strategies: A better understanding of the connection between product prices and
consumer behavior might help merchants develop better pricing strategies. Using this knowledge,
pricing plans that boost sales and profitability can be created.
4. Sales Growth: Market basket analysis can assist businesses in determining which products are most
frequently bought together and where they should be positioned in the store to grow sales. Retailers
may boost revenue and enhance customer shopping experiences by improving store layouts and
product positioning.
Applications of Market Basket Analysis
1. Retail: Market basket research is frequently used in the retail sector to examine consumer buying
patterns and inform decisions about product placement, inventory management, and pricing tactics.
Retailers can utilize market basket research to identify which items are sluggish sellers and which
ones are commonly bought together, and then modify their inventory management strategy
accordingly.
2. E-commerce: Market basket analysis can help online merchants better understand the customer
buying habits and make data-driven decisions about product recommendations and targeted
advertising campaigns. The behaviour of visitors to a website can be examined using market basket
analysis to pinpoint problem areas.
3. Finance: Market basket analysis can be used to evaluate investor behaviour and forecast the types
of investment items that investors will likely buy in the future. The performance of investment
portfolios can be enhanced by using this information to create tailored investment strategies.
4. Telecommunications: To evaluate consumer behaviour and make data-driven decisions about
which goods and services to provide, the telecommunications business might employ market basket
analysis. The usage of this data can enhance client happiness and the shopping experience.
5. Manufacturing: To evaluate consumer behaviour and make data-driven decisions about which
products to produce and which materials to employ in the production process, the manufacturing
sector might use market basket analysis. Utilizing this knowledge will increase effectiveness and
cut costs.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

Evaluation metrics for association rules

Evaluating association rules is crucial to ensure that the discovered patterns are not only statistically
significant but also practically useful and "interesting." While support, confidence, and lift are the
fundamental metrics, there are others that provide a more nuanced understanding of the rule's strength and
implications.
Here's a breakdown of the key evaluation metrics for association rules:
1. Support
• Definition: The proportion of transactions in the dataset that contain both the antecedent (A) and
the consequent (B) of a rule A⇒B.

• Interpretation:

• High support: Indicates that the itemset (A∪B) appears frequently in the dataset. It means
the rule is applicable to a substantial portion of the transactions.
• Low support: The itemset is rare. Rules with very low support might be statistically
significant but may not be practically actionable due to infrequent occurrence.
• Purpose: Primarily used to identify frequent itemsets. A minimum support threshold (min_sup) is
set to filter out infrequent items and itemsets, reducing the search space for rules.
2. Confidence
• Definition: The conditional probability that a transaction contains the consequent (B), given that it
already contains the antecedent (A).
• Interpretation:
o High confidence: Suggests a strong likelihood that if a customer buys A, they will also buy
B. It measures the reliability of the rule.
o Low confidence: The occurrence of A does not strongly imply the occurrence of B.
• Purpose: To assess the reliability or predictive power of an association rule. A minimum confidence
threshold (min_conf) is set to filter out unreliable rules.
3. Lift
• Definition: Measures how much more likely the consequent (B) is to be purchased when the
antecedent (A) is purchased, compared to the general likelihood of purchasing B. It compares the
observed support of A∪B with the expected support if A and B were statistically independent.

• Interpretation:

• Lift = 1: A and B are independent. The occurrence of A has no impact on the occurrence of
B.
• Lift > 1: Positive association. A and B occur together more often than expected by chance.
The higher the lift, the stronger the positive association.
• Lift < 1: Negative association. A and B occur together less often than expected by chance.
The presence of A decreases the likelihood of B.

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by
 lOMoARcPSD|69216461

Machine learning and Data analytics using Python MMC201

• Purpose: To determine the true strength and significance of the association, beyond just co-
occurrence. It helps filter out rules that might have high support and confidence simply because the
individual items are very popular.
Considerations for Evaluation:
• Thresholds: The interpretation of these metrics heavily depends on the chosen minimum support
and confidence thresholds. What's "high" or "low" is relative to the domain and dataset.
• Domain Knowledge: Statistical significance doesn't always equate to business usefulness. Domain
experts are crucial for interpreting rules and identifying truly actionable insights.
• Redundancy: A large number of rules can be generated. Pruning redundant or very similar rules is
important for focusing on the most valuable ones.
• Actionability: The ultimate goal of MBA is to find actionable insights. A rule might have high
metrics but be impractical to implement. Conversely, a rule with slightly lower metrics might be
highly actionable.
By using a combination of these evaluation metrics, analysts can gain a comprehensive understanding of
the strength, reliability, and interestingness of the association rules discovered, leading to more effective
business strategies.

In brief,
Rules are evaluated using:

• Support: How frequently the rule occurs in the dataset

The proportion of transactions that contain an itemset
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑇𝑟𝑎𝑛𝑠𝑎𝑐𝑡𝑖𝑜𝑛 𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑖𝑛𝑔 𝑋
𝑆𝑢𝑝𝑝𝑜𝑟𝑡(𝑋) =
𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑟 𝑜𝑓 𝑇𝑟𝑎𝑛𝑠𝑎𝑐𝑡𝑖𝑜𝑛

• Confidence: How often B is bought when A is bought

𝑆𝑢𝑝𝑝𝑜𝑟𝑡(𝐴 𝑎𝑛𝑑 𝐵)
𝐶𝑜𝑛𝑓𝑖𝑑𝑒𝑛𝑐𝑒 (𝐴 → 𝐵) =
𝑆𝑢𝑝𝑝𝑜𝑟𝑡(𝐴)

• Lift: How much more likely B is bought when A is bought compared to random chance
𝑆𝑢𝑝𝑝𝑜𝑟𝑡(𝐴 𝑎𝑛𝑑 𝐵)
𝐿𝑖𝑓𝑡 (𝐴 → 𝐵) =
𝑆𝑢𝑝𝑝𝑜𝑟𝑡(𝐴) × 𝑆𝑢𝑝𝑝𝑜𝑟𝑡(𝐵)

o Lift > 1: Positive association

o Lift = 1: Independent
o Lift < 1: Negative association

Dr. Srinivasa Rao K SVCE, B’lore

messages.downloaded_by