Chapter 3
Unsupervised learning
❖ Unsupervised learning is a branch of machine learning that deals
with unlabeled data.
❖ Unlike supervised learning, where the data is labeled with a specific
category or outcome, unsupervised learning algorithms are tasked
with finding patterns and relationships within the data without any
prior knowledge of the data’s meaning.
❖ Unsupervised machine learning algorithms find hidden patterns and
data without any human intervention, i.e., we don’t give output to
our model.
❖ The training model has only input parameter values and discovers
the groups or patterns on its own.
❖ To give you a simple example, think of a student who has
textbooks and all the required material to study but has no
teacher to guide.
❖ Ultimately, the student will have to learn by himself or
herself to pass the exams. This sort of self-learning is what
we have scaled into Unsupervised Learning for machines.
❖ There are mainly 3 types of Algorithms which are used
for Unsupervised dataset.
1. Clustering
2. Association Rule Learning
3. Dimensionality Reduction
1. Clustering Algorithm
What is clustering?
❖ The task of grouping data points based on their similarity with
each other is called Clustering or Cluster Analysis.
❖ This method is defined under the branch of unsupervised
learning, which aims at gaining insights from unlabelled data
points.
❖ Think of it as you have a dataset of customers shopping habits.
❖ Clustering can help you group customers with similar
purchasing behaviors, which can then be used for targeted
marketing, product recommendations, or customer segmentation
Types of Clustering
❖ Broadly, there are 2 types of clustering that can be performed
to group similar data points: These are hard clustering and
soft clustering.
Hard Clustering
❖ In this type of clustering, each data point belongs to a cluster
completely or not. For example, Let’s say there are 4 data
point and we have to cluster them into 2 clusters. So each data
point will either belong to cluster 1 or cluster 2.
Data Points Clusters
A C1
B C2
C C2
D C1
Soft Clustering
❖ In this type of clustering, instead of assigning each data point
into a separate cluster, a probability or likelihood of that point
being that cluster is evaluated.
❖ For example, Let’s say there are 4 data point and we have to
cluster them into 2 clusters.
❖ So we will be evaluating a probability of a data point
belonging to both clusters.
❖ This probability is calculated for all data points.
Data Points Probability of C1 Probability of C2
A 0.91 0.09
B 0.3 0.7
C 0.17 0.83
D 1 0
Uses of Clustering
Clustering algorithms are majorly used for:
❖ Market Segmentation: Businesses use clustering to group
their customers and use targeted advertisements to attract
more audience.
❖ Market Basket Analysis: Shop owners analyze their sales
and figure out which items are majorly bought together by
the customers.
❖ Social Network Analysis: Social media sites use your data to
understand your browsing behavior and provide you with
targeted friend recommendations or content
recommendations.
❖ Medical Imaging: Doctors use Clustering to find out
diseased areas in diagnostic images like X-rays.
❖ Anomaly Detection: To find outliers in a stream of real-time
dataset or forecasting fraudulent transactions we can use
clustering to identify them.
Grouping Unlabeled Items Using K-means
Clustering
❖ K-Means clustering is an Unsupervised machine learning algorithm
which groups the unlabeled dataset into different clusters.
❖ K-means clustering is a technique used to organize data into groups
based on their similarity.
❖ The algorithm will categorize the items into k groups or clusters
of similarity.
The algorithm works as follows:
1. First, we randomly initialize k points, called means or cluster
centroids.
2. We categorize each item to its closest mean, and we update the
mean’s coordinates, which are the averages of the items categorized
in that cluster so far.
3. We repeat the process for a given number of iterations and at the
end, we have our clusters.
Example 1
Design two cluster based on the following data points using K-
Means clustering algorithm.
Given data points {2,4,6,9,12,16,20,24,26}
▪ STEP-1: Initially we select or assume two data points or
centroids values are {4,12}
▪ We need to find out the nearest values of new centroids value i.e
{4,12}
k1={2,4,6} k2={9,12,16,20,24,26}
▪ This one find out based on the distance between the new centroids
points and other data points. k1 and k2 is clusters and contain the
above data points .
Now find out the mean value
k1=2+4+6/3=4
k2=9+12+16+20+24+26/6=17.8=18
▪ Therefore new data points for clustering is (4,18) and consider
this data points and again repeat the above process up to get same
value.
STEP-2:
▪ k1={2,4,6,9} k2={12,16,20,24,26}
▪ Mean value for k1=2+4+6+9/4=5.25=5
▪ Mean value for k2=12+16+20+24+26/5=19.6=20
▪ Therefore new data points for clustering is (5,20) and
consider this data points and again repeat the above process.
STEP-3
▪ k1={2,4,6,9,12} k2={16,20,24,26}
▪ Mean value for k1=2+4+6+9+12/5=6.6=7
▪ Mean value for k2=16+20+24+26/4=21.5=22
Therefore new data points for clustering is (7,22) and consider
this data points and again repeat the above process.
STEP-4:
▪ k1={2,4,6,9,12} k2={16,20,24,26}
▪ Mean value for k1=2+4+6+9+12/5=6.6=7
▪ Mean value for k2=16+20+24+26/4=21.5=22
▪ Therefore, both sides yield the same values compared to the
previous one, so the final mean value or data points are (7, 22).
▪ These data points represent the centroid values for the new
cluster. The new cluster is designed based on these data points.
Example: 2
Design two cluster based on following data points using k-
means clustering algorithm and Euclidean distance
formula. d = √[(x – x1)2 + (y – y1)2].
Given data is
[Link] X Y
1 170 60
2. 160 55
3 180 75
4 150 50
5 175 65
6 190 85
STEP:1
First we assume that two centroids for cluster i.e as per given
data we are considering row-1 and row -2 .so k=2.
initial centroids.
C X Y
C1 170 60
C2 160 55
STEP-2
For applying the Euclidean distance formula,
d = √[(x – x₁)² + (y – y₁)²].
Start with the initial centroids table C₁, C₂, and row 3. Then, we
will get new centroids for clustering.
Formula Applying For C1 Formula Applying For C2
here x=170 y= 60 and x1 =180 y1 =75 here x=160 y= 55 and x1 =180 y1 =75
d = √[(180 – 170)2 + (75 – 60)2] d = √[(180 – 160)2 + (75 – 55)2]
=√[(10)2 + (15)2] =√[(20)2 + (20)2]
=√[(100) + (225) =√[(400) + (400)
=√[325] =√[800]
=18.02 =28.28
=18 =28
Therefore, based on the above values, we are designing clusters using the new
centroid values.
The modified row is derived from the two points (18, 28), where 18 is smaller
compared to 28. By using the following formula, we obtain the new centroid value
for clustering.
X = 170 + 180/2=175 ( first row first value + third row first value) / 2
Y=60+75/2=67.5 ( first row second value + third row second value) / 2
Above step repeated up to all rows finished.
New centroids is given below
C X Y
C1 175 67.5
C2 160 55
Now New Cluster with centroids or data points are
C1={1,3........}
C2={2,.........}
STEP-3
To apply the Euclidean distance formula:
d = √[(x – x₁)² + (y – y₁)²].
Start with the previous centroids, represented in table C₁, C₂, and
row 4. Then, we can calculate the new centroids for clustering.
Formula Applying For C1 Formula Applying For C2
here x=175 y= 67.5 and x1 =150 y1 =50 here x=160 y= 55 and x1 =150 y1 =50
d = √[(175– 150)2 + (67.5 – 50)2] d = √[(160 – 150)2 + (55 – 50)2]
=√[(25)2 + (17.5)2] =√[(10)2 + (5)2]
=√[931.25] =√[125]
=30.51 =11.18
Therefore, based on the above values, we are designing clusters using the new
centroid values.
In the table below, the second centroid value has been changed.
The modified row is derived from the two points (30.51, 11.18), where 11.18 is
smaller compared to 30.51. By using the following formula, we can calculate the new
centroid value for clustering.
X = 160 + 150/2=155 (previous centroid table C2 first value + fourth row first
value) / 2
Y=55+50/2=52.5 (previous centroid table C2 row second value + fourth row
second value) / 2
New centroids is given below
C X Y
C1 175 67.5
C2 155 52.5
Now new cluster with centroids or data points are
C1={1,3........}
C2={2,4,.........}
STEP 4:
To apply the Euclidean distance formula:
d = √[(x – x₁)² + (y – y₁)²].
Start with the previous centroids in table C₁, C₂, and row
5. Then, we can calculate the new centroids for clustering.
Formula Applying For C1 Formula Applying For C2
here x=175 y= 67.5 and x1 =175 y1 =65 here x=155 y= 52.5 and x1 =175 y1 =65
d = √[(175– 175)2 + (67.5 – 65)2] d = √[(155 – 175)2 + (65 – 52.5)2]
=√[(0)2 + (2.5)2] =√[(25)2 + (12.5)2]
=√[(2.5)2] =√[625+169]
=2.5 =23.58
Therefore, based on the above values, we are designing clusters using the new centroid
values from the table below.
The first centroid value (C₁) has been changed.
The modified row is derived from the two points (2.5, 23.58), where 2.5 is smaller
compared to 23.58. By using the following formula, we can calculate the new centroid
value for clustering.
X = 175 + 175/2=175 ( previous centroid table C1 row first value + Fifth row first
value) / 2
Y=67.5+65/2=66.5 (previous centroid table C2 row second value + Fifth row
second value) / 2
New centroids is given below
C X Y
C1 175 66.25
C2 155 52.5
Now new cluster with centroids or data points are c1={1,3,5........}
C2={2,4,.........}
STEP-5:
To apply the Euclidean distance formula:
d = √[(x – x₁)² + (y – y₁)²].
Start with the previous centroids in table C₁, C₂, and row 6. Then, we
can calculate the new centroids for clustering.
Formula Applying For C1 Formula Applying For C2
here x=175 y= 66.25 and x1 =190 y1 =85 here x=155 y= 52.5 and x1 =190 y1 =85
d = √[(190– 175)2 + (85 – 66.25)2] d = √[(155 – 190)2 + (52.5 – 85)2]
=√[(15)2 + (18.75)2] =√[(35)2 + (32.5)2]
=√[225+351.56] =√[1225+1056.25]
=24.01 =47.76
Therefore, based on the above values, we are designing clusters using the new centroid
values from the table below. The first C₁ centroid value has been changed.
The modified row is derived from the two points (24.01, 47.76), where 24.01 is the
smallest value compared to 47.76. By using the following formula, we can calculate
the new centroid value for clustering.
X = 175 + 190/2=182.5 ( previous centriod table C1 row first value + Sixth row first
value) / 2
Y=62.5+85/2=73.75 (previous centriod table C2 row second value + sixth row
second value) / 2
New centroids is given below
C X Y
C1 182.5 73.75
C2 155 52.5
Now new cluster with centroids or data points are
C1={1,3,5,6........}
C2={2,4,.........}
2. Association Rule Learning
❖ Association rule mining, at a basic level, involves the use
of machine learning models to analyze data for patterns, in a
database.
❖ It identifies frequent if-then associations, which themselves are
the association rules.
❖ An association rule has two parts:
❖ an antecedent (if) and a consequent (then).
❖ An antecedent is an item found within the data. A consequent is an
item found in combination with the antecedent.
❖ Association rules are created by searching data for frequent if-then
patterns and using the criteria support and confidence to identify
the most important relationships
❖ Support is an indication of how frequently the items
appear in the data.
❖Confidence indicates the number of times the if-then
statements are found true.
Apriori Algorithm :
➢ It is one of the most well-known algorithms for association
rule mining.
Apriori principle (Main observation):
▪ If an itemset is frequent, then all of its subsets must also be
frequent.
▪ If an itemset is not frequent, then none of its supersets can be
frequent.
X⊆Y⇒s(X)≥s(Y)
▪ The support of a superset never exceeds the support of its
subsets. –This property is called the anti-monotone property
of support.
Important Points About Apriori Algorithm
❖ The Apriori algorithm was the first algorithm proposed for
frequent itemset mining.
❖ This algorithm uses two main steps, called the join step and the
prune step, to reduce the search space.
❖ It is an iterative approach used to discover frequent itemsets in a
transaction database.
Apriori says:
The probability that item I is not frequent is if:
❖ P(I) < minimum support threshold, then I is not frequent.
❖ P (I+A) < minimum support threshold, then I+A is not frequent,
where A also belongs to item set.
❖ If an item set has value less than minimum support then all of
its supersets will also fall below min support, and thus can be
ignored. This property is called the Antimonotone property.
The steps followed in the Apriori Algorithm of data mining are:
• Join Step: This step generates (K+1) item set from K-item sets by
joining each item with itself.
• Prune Step: This step scans the count of each item in the database.
If the candidate item does not meet minimum support, then it is
regarded as infrequent and thus it is removed. This step is performed
to reduce the size of the candidate item sets.
❖ Example problem on Apriori Algorithm.
❖ let us consider the transaction database D as shown in below. there
are 9 transaction in the database use Apriori Algorithm for finding
frequent itemsets in D.
❖ NOTE : Minimum support count=2
T-ID LIST OF ITEM -ID'S
T1 I1,I2,I5
T2 I2,I4
T3 I2,I3
T4 I1,I2,I4
T5 I1,I3
T6 I2,I3
T7 I1,I3
T8 I1,I2,I3,I5
T9 I1,I2,I3
STEP:1
SCAN D for count of each candidate "C1".
C1 L1
ITEMS SUPPORT COUNT
ITEMS SUPPORT COUNT
{I1} 6
{I1} 6
{I2} 7
{I2} 7
{I3} 6
{I3} 6
{I4} 2
{I4} 2
{I5} 2
{I5} 2
STEP-2:
Compare candidate support count with minimum support count "L1".
STEP-3 Generate C2 from L1
"C2― ―L2‖
ITEMS SUPPORT COUNT
ITEMS SUPPORT COUNT
{I1,I2} 4
{I1,I2} 4
{I1,I3} 4
{I1,I4} 1
{I1,I3} 4
{I1,I5} 2 {I1,I5} 2
{I2,I3} 4 {I2,I3} 4
{I2,I4} 2 {I2,I4} 2
{I2,I5} 2 {I2,I5} 2
{I3,I4} 0
{I3,I5} 1
{I4,I5} 0
STEP-4: Compare Candidate support count with minimum support
count."L2―
STEP-5: Generate C3 from L2
"C3― "L3"
ITEMS SUPPORT COUNT ITEMS SUPPORT COUNT
{I1,I2,I3} 2 {I1,I2,I3} 2
{I1,I2,I5} 2 {I1,I2,I5} 2
{I1,I2,I4} 1
{I1,I3,I5} 1
{I2,I3,I4} 0
{I2,I3,I5} 1
{I2,I4,I5} 0
STEP-6 : COMPARE candidate support count with minimum support
count "L3"
STEP-7: Generate " C4" From L3
"C4"
ITEMS SUPPORT COUNT
{I1,I2,I3,I5} 1
STEP-8
Compare candidate support count with minimum support count "L4"
L4
as per apriori Algorithm whenever L4=0 The
ITEMS SUPPORT algorithm terminated.
Lk=0 i.e
COUNT as per Algorithm Rule we know that Lk-1 L 4-1=L3
0 0 frequent item sets.
the item sets in L3 are frequent item sets i.e
{I1,I2,I3} and {I1,I2,I5}
Below are a few real-world use cases for association rules:
❖ Medicine. Doctors can use association rules to help diagnose
patients. There are many variables to consider when making a
diagnosis, as many diseases share symptoms.
❖ By using association rules and machine learning-fueled data
analysis, doctors can determine the conditional probability of a
given illness by comparing symptom relationships in the data from
past cases.
❖ Retail. Retailers can collect data about purchasing patterns,
recording purchase data as item barcodes are scanned by point-of-
sale systems.
❖ Machine learning models can look for co-occurrence in this data to
determine which products are most likely to be purchased together.
The retailer can then adjust marketing and sales strategy to take
advantage of this information.
❖ User experience (UX) design. Developers can collect data on how
consumers use a website they create. They can use associations in
the data to optimize the website user interface by analyzing where
users tend to click and what maximizes the chance that they engage
with a call to action.
❖ Entertainment. Services like Netflix and Spotify can use
association rules to fuel their content recommendation engines.
Machine learning models analyze past user behavior data for
frequent patterns, develop association rules and use those rules to
recommend content that a user is likely to engage with, or organize
content in a way that is likely to put the most interesting content for
a given user first.
3. Dimensionality Reduction
❖ Dimensionality reduction is the task of reducing the number
of inputs.
❖ To reduce the noise and improve the performance of the
Machine Learning algorithm.
❖ Principal Component Analysis (PCA)
❖ Singular Value Decomposition (SVD)
❖ In data analysis we always face high dimensional data. (In real
world)
❖ The process of finding these narrow matrices from the high
dimensional data is called dimensionality reduction.
❖ Why Dimensionality Reduction (DR)?
❖ Each disciplines may have its specific demand to perform
dimensionality reduction.
❖ DR focus on the observed data set and ignores the
generalization performance as used in data compression.
Principal Component Analysis
❖ In PCA, the dataset is transformed from its original coordinate
system to a new coordinate system.
❖ The new coordinate system is chosen by the data itself.
❖ The first new axis is chosen in the direction of the most variance
in the data.
❖ The second axis is orthogonal to the first axis and in the
direction of an orthogonal axis with the largest variance.
❖ This procedure is repeated for as many features as we have in
the original data.
Principal Component Analysis
❖ The first axis has to be rotated to cover the largest variation in
the data: line B in the figure.
❖ The largest variation is the data telling us what’s most
important.
Principal Component Analysis
❖ Pros: Reduces complexity of data, identifies most important
features.
❖ Cons: May not be needed, could throw away useful information.
❖ Works with: Numerical values.
Singular Value Decomposition (SVD)
❖ SVD is a powerful tool used to distill information in a number
of applications, from bioinformatics to finance.
❖ We can use the SVD to represent our original data set with a
much smaller data set.
❖ When we do this, we’re removing noise and redundant
information.
❖ Pros: Simplifies data, removes noise, may improve algorithm
results.
❖ Cons: Transformed data may be difficult to understand.
❖ Works with: Numeric values.
Reinforcement Learning
❖ Reinforcement learning: is a type of machine learning where an
agent learns to behave in an environment by performing actions
and seeing the results.
Reinforcement Learning
❖ Example of RL:
❖A robot cleaning a room,
❖Game playing,
❖Learning how to fly a helicopter,
❖Scheduling planes to their destinations,
❖And so on…
Reinforcement Learning Process
❖ Reinforcement learning system is composed of two main
components:
❖Agent
❖Environment
❖ The action influences the state of the world which determines its
reward.
Reinforcement Learning Definitions
❖ Agent: The RL algorithm that learns from trial and error.
❖ Environment: The world through which the agent moves.
❖ Action (A): All the possible steps that the agent can take.
❖ State (S): Current condition returned by the environment.
❖ Reward (R): An instant return from the environment to appraise
the last action.
Reinforcement Learning with Analogy
1. Initial Situation: A baby is in a safe room and wants to reach a
favorite toy placed a few feet away.
2. Action: The baby attempts to crawl or stand up in order to reach
the toy.
3. Responses:
❖If the baby successfully crawls or takes a few steps and
reaches the toy: The baby gets excited and smiles (a form
of reward).
❖If the baby tries to stand but falls down or encounters an
obstacle: The baby might become frustrated or confused
(no reward).
Reinforcement Learning with Analogy
4. Learning Process:
❖Positive Reinforcement: Each time the baby successfully
reaches the toy, it feels happy and rewarded. This motivates
the baby to try crawling or standing up again in the future,
reinforcing that behavior.
❖Adjusting Behavior: If the baby falls or doesn't manage to
reach the toy, it might change its approach next time—like
crawling faster, using furniture to pull up, or crawling around
obstacles.
In this analogy:
❖ The baby represents the agent (like an AI model).
❖ The action is the baby crawling or trying to walk.
❖ The toy is the reward the baby is aiming for.
❖ The positive emotions or smiles received upon reaching the toy
are the rewards that encourage further attempts.
❖ The baby learns from both successes (reaching the toy) and
failures (falling or not reaching the toy), adjusting its actions to
improve in future attempts.