Machine Learning –Sem-V

Module: 04
Un-Supervised Learning & Semi-supervised Learning
Motivation:

This chapter helps us in understanding about unsupervised and semi-supervised learning algorithm

Syllabus:
Lecture Content Duration Self-Study
no (Hr) (Hrs)
Distance based clustering Algorithms
1 1 1
K-Means
2 1 1
K-medoids
3 1 1
Hierarchical clustering
4 1 1
Hierarchical Agglomerative Clustering
5 1 1
Density Based Scan Clustering (DBSCAN)
6 1 1
Gaussian Clustering Model
7 1 1
Semi- supervised learning using labelled and unlabelled data
8 1 1
Semi- supervised learning using labelled and unlabelled data
9 1 1

Theoretical Background:
Unsupervised learning and semi-supervised learning are two fundamental branches of machine
learning. Unsupervised learning involves analyzing datasets without labeled outputs, aiming to reveal
underlying patterns or groupings within the data. Common techniques like clustering group similar
data points, while dimensionality reduction methods reduce complex data to its essential components.
In contrast, semi-supervised learning operates in scenarios where limited labeled data is available
alongside unlabeled data. This approach leverages the additional unlabeled data to enhance model
performance. Techniques like self-training and graph-based methods help the model learn from both
labeled and unlabeled instances. Unsupervised learning is crucial for uncovering insights from raw
data, while semi-supervised learning bridges the gap between supervised and unsupervised methods,
proving valuable when obtaining labeled data is difficult.

Key Definitions:
Unsupervised learning :Unsupervised learning is a category of machine learning where an algorithm
is trained on a dataset that lacks explicit labels or target outputs. In unsupervised learning, the
algorithm's objective is to discover inherent patterns, structures, or relationships within the data
without the guidance of predefined target values.
Semi-supervised learning:Semi-supervised learning is a machine learning paradigm that combines
elements of both supervised and unsupervised learning. In semi-supervised learning, the training

1
Reinforcement Learning -Sem-VI

2
dataset includes a mix of labeled and unlabeled data. Labeled data points have explicit target labels,
while unlabeled data points lack these labels.

Lecture 1
Distance based clustering Algorithms
Learning Objective:
● Learners shall be able to understand Distance based clustering Algorithms

Course Content:
Distance-based clustering algorithms are a class of unsupervised machine learning techniques that
group data points into clusters based on their proximity or similarity in a multidimensional space.
These algorithms are commonly used in various fields, including data analysis, image processing, and
natural language processing. Here are some notable distance-based clustering algorithms:

K-Means Clustering:

K-Means is one of the most widely used clustering algorithms.

It partitions data into 'k' clusters, where 'k' is predefined.
It minimizes the sum of squared distances between data points and their assigned cluster centroids.
Simple to implement and computationally efficient, but sensitive to initial cluster centroids.

Hierarchical Clustering:

Hierarchical clustering builds a hierarchy of clusters by iteratively merging or splitting existing

clusters.
Two main approaches are Agglomerative (bottom-up) and Divisive (top-down).
It doesn't require specifying the number of clusters beforehand.
Can produce dendrogram plots for visual representation.

DBSCAN (Density-Based Spatial Clustering of Applications with Noise):

DBSCAN groups data points into clusters based on their density.

It defines clusters as regions of high data point density separated by regions of low density.
Able to find clusters of arbitrary shape and handle noise effectively.
It iteratively shifts data points towards the mode of the nearest data points until convergence.
Suitable for applications like image segmentation.

Gaussian Mixture Models (GMM):

GMM models data as a mixture of multiple Gaussian distributions.

It estimates the parameters of these Gaussians to fit the data.

2
Machine Learning –Sem-V

Each Gaussian corresponds to a cluster, and a data point belongs to the cluster with the highest
probability.
Let’s check the take away from this lecture
Exercise:

1. How does the K-Means clustering algorithm work, and what are some practical applications
where it is commonly used?
2. Explain the concept of hierarchical clustering, and provide an example of when hierarchical
clustering might be preferable over other clustering techniques.
3. What are some commonly used distance metrics in distance-based clustering algorithms, and
how do they impact the clustering results?

Learning from this lecture: Learners will be able to understand Distance based clustering Algorithms

Lecture: 2
K-Means

Learning objective:
● Learners shall be able to learn about K-Means algorithm

Course Content

K-Means clustering is a popular unsupervised machine learning algorithm used for partitioning a
dataset into 'K' distinct, non-overlapping subgroups or clusters. These clusters are determined based
on the similarity or proximity of data points in a high-dimensional space. K-Means is widely applied
in various fields, including data analysis, image processing, and customer segmentation.

Euclidean Distance

Clustering works on the basis of grouping the observations which are the most similar to each other.
What does this exactly mean?

In simple terms, the algorithm needs to find data points whose values are similar to each other and
therefore these points would then belong to the same cluster. The method in which any clustering
algorithm goes about doing that is through the method of finding something called a “distance
measure”. The distance measure that is used in K-means clustering is called the Euclidean Distance
measure.

In two-dimensional space, Euclidean distance between two points (x1, y1) and (x2, y2) can be
calculated using the following formula:

3
Reinforcement Learning -Sem-VI

4
Distance = sqrt((x1 - x2)^2 + (y1 - y2)^2)

The observations which are closer or more similar to each other would have a low Euclidean distance

The observations which are farther or less similar to each other would have a higher Euclidean
distance

Centroids are the centre points of the clusters that are being [Link] you have the following
clusters of the marks of a group of students in Mathematics and Biology and someone asks you to
explain them. From a glance, you can easily interpret the 4 clusters that are being [Link] the four
clusters that are being formed are as follows:

Cluster 1: Students who have scored high marks in Bio, but poor marks in Maths

Cluster 2: Students who have scored average marks in Bio and Maths

Cluster 3: Students who have scored high marks in both Bio and Maths.

Cluster 4: Students who have scored high marks in Maths, but poor marks in Bio

4
Machine Learning –Sem-V

Now the representation is fine and correct, but it is missing one crucial information - the numerical
order. For example, when we want to compare two clusters say Cluster 1 and Cluster 2 can you say by
how much marks on average do the students from Cluster 1 outperform or underperform the Cluster 2
students in a particular subject just by taking a look at the above visualisation alone? Is it by 10 marks?
Or 15?

Centroids are essentially the cluster centres of a group of observations that help us in summarising the
cluster's properties

Calculation of Centroid

5
Reinforcement Learning -Sem-VI

6
Machine Learning –Sem-V

Here's how the K-Means algorithm works:

Initialization:

Begin by selecting 'K,' the number of clusters you want to create. This is often determined based on
domain knowledge or by using techniques like the elbow method.

Randomly initialize 'K' cluster centroids (points at the center of each cluster). These centroids serve as
the initial representatives of the clusters.

Assignment Step (Cluster Assignment):

For each data point in the dataset, calculate the distance between that point and each of the 'K' centroids.
Common distance metrics include Euclidean distance, Manhattan distance, or others.

Assign the data point to the cluster represented by the nearest centroid.

Update Step (Centroid Recalculation):

Once all data points are assigned to clusters, calculate the new centroids for each cluster by computing
the mean of all data points assigned to that cluster.

These new centroids represent the updated centers of the clusters.

Convergence Check:

Repeat the assignment and update steps iteratively until one of the stopping conditions is met. Common
stopping conditions include a maximum number of iterations, a small change in centroids, or the
absence of data point reassignment.

Result:

When the algorithm converges, the final cluster centroids represent the centers of the 'K' clusters.

Each data point belongs to the cluster associated with the nearest centroid.

K-Means has several advantages, such as its simplicity, computational efficiency, and effectiveness in
identifying clusters with a roughly spherical shape. However, it also has limitations. It requires you to
specify the number of clusters 'K' in advance, is sensitive to initial centroid placement, and may not
perform well when clusters have irregular shapes or varying densities.

Let’s check the take away from this lecture

Exercise:
1. Could you explain the role of the "K" parameter in K-Means, and how do you determine the
optimal value of K for a given dataset?

7
Reinforcement Learning -Sem-VI

8
2. What are the steps involved in the K-Means clustering process, from initialization to
convergence?
3. What are some advantages of using K-Means clustering, and in what types of data analysis or
applications is it most commonly utilized?
4. Can you discuss any challenges or limitations of the K-Means algorithm, and provide strategies
or alternatives to address these issues when they arise during clustering tasks?
Learning from this lecture: Learners will be able to understand K-Means algorithm

Lecture: 3
K-medoids

Learning objective: Learners shall be able to understand K-medoids

Course Content
The K-means clustering algorithm is sensitive to outliers, because a mean is easily influenced by
extreme values. K-medoids clustering is a variant of K-means that is more robust to noises and outliers.
Instead of using the mean point as the center of a cluster, K-medoids uses an actual point in the cluster
to represent it. Medoid is the most centrally located object of the cluster, with minimum sum of
distances to other points.

Algorithm

Initialize: select k random points out of the n data points as the medoids.
Associate each data point to the closest medoid by using any common distance metric methods.
While the cost decreases: For each medoid m, for each data o point which is not a medoid:
Swap m and o, associate each data point to the closest medoid, and recompute the cost.
If the total cost is more than that in the previous step, undo the swap.

Let’s check the take away from this lecture

Exercise:
1. What is the fundamental difference between K-Means and K-Medoids clustering algorithms,
and why might one choose K-Medoids over K-Means for certain clustering tasks?
2. How does K-Medoids select medoids within clusters, and what advantages does this approach
offer in terms of robustness to outliers compared to K-Means?

8
Machine Learning –Sem-V

3. Similar to K-Means, K-Medoids requires specifying the number of clusters (K). How can one
determine the optimal value of K for a dataset when using K-Medoids?
4. Can you discuss scenarios or types of data where K-Medoids is known to perform exceptionally
well, and provide insights into when it might be less suitable compared to other clustering methods?
5. K-Medoids is often regarded as more computationally intensive than K-Means. What strategies
or techniques can be employed to improve the efficiency and scalability of K-Medoids for large
datasets?
6.
Learning from this lecture: Learners will be able to understand K-medoids

Lecture: 4
Hierarchical clustering

Learning objective: Learners shall be able to understand Hierarchical clustering

Course Content
One of the major considerations in using the K-means algorithm is deciding the value of K beforehand.
The hierarchical clustering algorithm does not have this restriction.
The output of the hierarchical clustering algorithm is quite different from the K-mean algorithm as
well. It results in an inverted tree-shaped structure, called the dendrogram. An example of a
dendrogram is shown below.

In the K-Means algorithm, we divide the data in the first step itself. In the subsequent steps, we refine
our clusters to get the most optimal grouping.
In hierarchical clustering, the data is not partitioned into a particular cluster in a single step. Instead, a
series of partitions/merges take place, which may run from a single cluster containing all objects to n
clusters that each contain a single object or vice-versa.
This is very helpful since we don’t have to specify the number of clusters beforehand.
Given a set of N items to be clustered, the steps in hierarchical clustering are:
Calculate the NxN distance (similarity) matrix, which calculates the distance of each data point from
the other.
Each item is first assigned to its own cluster, i.e. N clusters are formed
The clusters which are closest to each other are merged to form a single cluster

9
Reinforcement Learning -Sem-VI

10
The same step of computing the distance and merging the closest clusters is repeated till all the points
become part of a single cluster

In the dendrogram shown above, samples 4 and 5 are the most similar and join to form the first cluster,
followed by samples 1 and 10. The last two clusters to fuse together to form the final single cluster are
3-6 and 4-5-2-7-1-10-9-8.
Determining the number of groups in a cluster analysis is often the primary goal. Typically, one looks
for natural groupings defined by long stems. Here, by observation, you can identify that there are 3
major groupings: 3-6, 4-5-2-7 and 1-10-9-8.

Cutting the dendrogram

Once we obtain the dendrogram, the clusters can be obtained by cutting the dendrogram at an
appropriate level. The number of vertical lines intersecting the cutting line represents the number of
clusters.

Hierarchical clustering is a popular unsupervised machine learning technique used for grouping similar
data points into nested clusters or a hierarchical structure. Unlike other clustering methods, hierarchical

10
Machine Learning –Sem-V

clustering produces a visual representation of the relationships between data points in the form of a
tree-like structure called a dendrogram.

Here's how the Hierarchical Clustering algorithm works:

Initialization:

Each data point initially represents a single cluster.

Agglomeration (Bottom-Up Approach) or Division (Top-Down Approach):

There are two main approaches to hierarchical clustering: Agglomerative (bottom-up) and Divisive
(top-down).

In Agglomerative Hierarchical Clustering:

Start with each data point as a separate cluster.

Repeatedly merge the two closest clusters into a single cluster based on a specified distance metric
(e.g., Euclidean distance).
Continue this merging process until all data points belong to a single cluster or until a predefined
number of clusters is reached.
The result is a dendrogram representing the hierarchy of clusters.
In Divisive Hierarchical Clustering:

Start with all data points in a single cluster.

Recursively divide the cluster into two sub-clusters based on some criteria, such as maximizing the
distance between the sub-clusters.
Continue dividing until each data point forms its own cluster or until a stopping criterion is met.
Dendrogram Construction:

As the merging or division process proceeds, a dendrogram is constructed. This dendrogram visually
represents the hierarchy of clusters and shows how clusters are combined or divided at each step.
Selecting the Number of Clusters:

One of the advantages of hierarchical clustering is that it doesn't require specifying the number of
clusters beforehand. The number of clusters can be chosen by cutting the dendrogram at an appropriate
level based on the problem's requirements or using techniques like the elbow method.
Result:

Once you've chosen the desired number of clusters by cutting the dendrogram, you obtain the final
clustering assignments for your data points.
Hierarchical clustering has several advantages, including its ability to reveal the hierarchical structure
of the data, flexibility in choosing the number of clusters, and suitability for visual exploration of data
relationships. However, it can be computationally intensive for large datasets, and the choice of linkage
criteria (e.g., single, complete, average linkage) and distance metrics can impact the clustering results.

Let’s check the take away from this lecture

Exercise:
1. Could you explain the key differences between agglomerative and divisive hierarchical
clustering methods, and provide scenarios where one might be more appropriate than the other?
11
Reinforcement Learning -Sem-VI

12
2. How do dendrograms generated by hierarchical clustering algorithms help in understanding the
hierarchical relationships between clusters, and how can one use them to determine the optimal number
of clusters in a dataset?
3. What are some strategies or techniques to make hierarchical clustering more computationally
efficient and scalable when dealing with large datasets, considering that it can be computationally
intensive?
Learning from this lecture: Learners will be able to understand Hierarchical clustering

Lecture: 5
Hierarchical Agglomerative Clustering

Learning objective: Learners shall be able to understand Hierarchical Agglomerative Clustering

Course Content
Agglomerative clustering is a hierarchical clustering technique used to group similar data points
together. The basic idea of agglomerative clustering is to start with each data point as its own cluster
and then iteratively merge pairs of clusters based on their similarity, until all the data points belong to
a single cluster.
The algorithm works as follows:
● Assign each data point to its own cluster.
● Compute the distance or similarity between all pairs of clusters.
● Merge the two closest clusters into a single cluster.
● Recompute the distance or similarity between the new cluster and all the remaining clusters.
● Repeat steps 3 and 4 until all the data points belong to a single cluster.

Linkages

Single
Average
Complete

Single Linkage
In single linkage hierarchical clustering, the distance between two clusters is defined as the shortest
distance between two points in each cluster. For example, the distance between clusters “r” and “s” to
the left is equal to the length of the arrow between their two closest points. This method tends to create
long, elongated clusters and is sensitive to outliers and noise.

12
Machine Learning –Sem-V

In complete linkage hierarchical clustering, the distance between two clusters is defined as the longest
distance between two points in each cluster. For example, the distance between clusters “r” and “s” to
the left is equal to the length of the arrow between their two furthest points. This method tends to create
compact, spherical clusters and is less sensitive to outliers.
In average linkage hierarchical clustering, the distance between two clusters is defined as the average
distance between each point in one cluster to every point in the other cluster. For example, the distance
between clusters “r” and “s” to the left is equal to the average length each arrow between connecting
the points of one cluster to the other. This method strikes a balance between single and complete
linkage and is less sensitive to outliers compared to single linkage.

Example

13
Reinforcement Learning -Sem-VI

14

Let’s check the take away from this lecture

Exercise:
1. How does the dendrogram generated by Hierarchical Agglomerative Clustering help in
understanding the hierarchical relationships between clusters, and what factors should be considered
when determining the optimal level to cut the dendrogram to obtain a specific number of clusters?
2. Could you explain the differences between commonly used linkage methods in Agglomerative
Clustering, such as single linkage, complete linkage, and average linkage? How does the choice of
linkage method affect the resulting clusters?
3. Hierarchical Agglomerative Clustering is known to be computationally intensive, especially
for large datasets. What techniques or strategies can be employed to make the algorithm more efficient
and scalable when dealing with high-dimensional or large data sets?
Learning from this lecture: Learners will be able to understand Hierarchical Agglomerative Clustering.

Lecture: 6
Density Based Scan Clustering (DBSCAN)

Learning objective: Learners shall be able to understand Density Based Scan Clustering
(DBSCAN)
Course Content
DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a density-based clustering
algorithm widely used in data mining and machine learning. Unlike centroid-based algorithms like K-
Means, DBSCAN identifies clusters based on the density of data points in the feature space. It's

14
Machine Learning –Sem-V

particularly effective at discovering clusters of arbitrary shapes and handling noise in the data.

Here's how the DBSCAN algorithm works:

Core Points and Density: DBSCAN categorizes data points into three categories: core points, border
points, and noise points.

Core Points: A data point is considered a core point if there are at least 'minPts' data points (a user-
defined parameter) within a specified distance (epsilon or ε) from it.
Border Points: A data point is classified as a border point if it's within ε distance of a core point but
doesn't have enough neighbors to be a core point itself.
Noise Points (Outliers): Data points that are neither core nor border points are treated as noise points
or outliers.
Cluster Formation:

DBSCAN begins by selecting an arbitrary data point that has not been visited and checks if it's a core
point.
If it's a core point, DBSCAN starts a new cluster and expands it by adding all reachable core points
(and their reachable core points) to the cluster.
This process continues iteratively until no more core points can be added to the cluster.
When all core points in the cluster are exhausted, the algorithm selects another unvisited point and
repeats the process, forming additional clusters.
Result:

After processing all data points, DBSCAN has created clusters of varying shapes and sizes.
Border points are assigned to the clusters of their respective core points.
Noise points are not assigned to any cluster.
DBSCAN's advantages include its ability to find clusters of arbitrary shapes and its robustness to noise
and outliers. It doesn't require specifying the number of clusters beforehand and is relatively insensitive
to parameter choices. However, it may struggle with clusters of varying densities and can be
computationally expensive for large datasets.

Users typically define two main parameters when applying DBSCAN: ε (epsilon), which controls the
neighborhood size, and 'minPts,' which sets the minimum number of data points in a neighborhood for
a point to be considered a core point. The choice of these parameters can affect the clustering results.

15
Reinforcement Learning -Sem-VI

16

Let’s check the take away from this lecture

Exercise:
1. How do the choice of parameters, specifically the epsilon (ε) neighborhood size and the minimum
number of points (minPts), impact the clustering results in DBSCAN? Can you provide examples of how
adjusting these parameters can lead to different clusterings of the same dataset?
2. DBSCAN is known for its ability to handle noise and outliers effectively. Can you explain how
DBSCAN distinguishes noise points from valid clusters, and how can users interpret and utilize noise points in
real-world data analysis or applications?
3. While DBSCAN is a powerful clustering algorithm, it may have limitations. What are some scenarios
or types of data where DBSCAN might not perform well, and are there any specific preprocessing steps or
alternative clustering methods that can be considered in such cases?
Learning from this lecture: Learners will be able to understand Density Based Scan Clustering
(DBSCAN)

Lecture:7
Gaussian Clustering Model

Learning objective: Learners shall be able to understand Gaussian Clustering Model

Course Content

K-Means algorithm inner-loop iterates over two steps:

Assign each observation Xi to the closest cluster centroid μk
Update each centroid to the mean of the points assigned to it.
So, for K-Means, every data point is assigned to any of the one clusters, this is known as Hard
Clustering or Hard cluster assignment.
Hard Clustering: In hard clustering, the data points are assigned to any one cluster completely. That
means for hard clustering there is no consideration of uncertainty in the assignment of a data points.
The limitations with hard clustering are that we tend to clusters even those data points in one of the
clusters, even if the data point doesn't follow the clustering trend completely.
So to overcome this issue, we have a concept of Soft Clustering.
Soft Clustering: In soft clustering, the assignment of the data point to a cluster is based on the
probability or likelihood of that data point to existing in that cluster.

16
Machine Learning –Sem-V

For soft clustering, we have an algorithm called GMMs or Gaussian Mixture Models.

Algorithm

1. Initialize the model: Choose the number of clusters (K) and initialize the parameters of each
Gaussian distribution (mean and covariance matrix) and the mixture weights.
2. E-step (Expectation step): Estimate the probability of each data point belonging to each of the
K Gaussian distributions using Bayes' rule
3. M-step (Maximization step): Update the parameters of each Gaussian distribution (mean and
covariance matrix) and the mixture weights based on the probabilities estimated in the E-step.
4. Repeat steps 2 and 3 until convergence: Repeat the E-step and M-step until the change in the
log-likelihood of the data is below a certain threshold. This ensures that the model has converged to a
stable solution.
5. Assign clusters to data points: Once the model has converged, assign each data point to the
cluster with the highest probability of belonging to that cluster.
6. Evaluate the quality of clustering: Use various metrics (e.g., silhouette score) to evaluate the
quality of the clustering and select the optimal number of clusters.

The Gaussian distribution has a bell-shaped curve with its peak at the mean, and it is symmetric around
the mean. The standard deviation determines the width of the curve, and the larger the standard
deviation, the wider the curve.

Let’s check the take away from this lecture

Exercise:
1. What's the fundamental concept behind DBSCAN, and how does it determine clusters in data?
2. What are the two main parameters in DBSCAN, and what do they control in the clustering process?
3. Can you provide a practical example or scenario where DBSCAN would be a suitable clustering
method, and explain why it's a good choice for that situation?
Learning from this lecture: Learners will be able to understand Gaussian Clustering Model

Lecture: 8
Semi- supervised learning using labelled and unlabelled data

Learning objective: Learners shall be able to understand Semi- supervised learning using labelled and
unlabelled data

17
Reinforcement Learning -Sem-VI

18
Course Content
Semi-supervised learning is a machine learning paradigm that leverages both labeled and unlabeled
data to build predictive models. In this approach, labeled data consists of examples where each input
is associated with a known output (e.g., in a binary classification task, labeled data would have inputs
and corresponding class labels). Unlabeled data, on the other hand, contains inputs without associated
outputs. This paradigm is particularly valuable when acquiring labeled data is costly or time-
consuming.

Here's a detailed description of semi-supervised learning using labeled and unlabeled data:

**1. Semi-Supervised Learning Overview:

Semi-supervised learning is a middle ground between supervised learning (using only labeled data)
and unsupervised learning (using only unlabeled data).
The goal is to improve model performance by harnessing the additional information from the unlabeled
data.
**2. Labeled Data Usage:

Labeled data is used in the same way as in supervised learning. It's used to train the model, evaluate
its performance, and guide its learning process.
**3. Unlabeled Data Usage:

Unlabeled data, which is often more abundant, is used to assist the model in learning the underlying
patterns or structure of the data.
Self-training: One common technique involves using the model trained on labeled data to make
predictions on the unlabeled data. The predictions become pseudo-labels, effectively turning some
unlabeled data into pseudo-labeled data. This process can be iterated.
Consistency regularization: Another approach is to enforce consistency between the model's
predictions on different views of the same unlabeled data point, enhancing the model's robustness.
Clustering: Unlabeled data can be clustered into groups, and these clusters can serve as additional
supervision for the model. This approach is known as cluster-assisted semi-supervised learning.
**4. Benefits and Applications:

Semi-supervised learning is particularly useful when acquiring labeled data is expensive, such as in
medical diagnosis, where labeling requires expert knowledge or human annotation.
It's also valuable in cases where there is a lot of unlabeled data readily available, as it can boost model
performance with minimal additional labeling effort.
**5. Challenges and Considerations:

The effectiveness of semi-supervised learning relies on the quality of the labeled and unlabeled data.
It assumes that the distribution of labeled data is similar to that of the unlabeled data, which may not
always hold true.
Care must be taken to handle errors introduced by pseudo-labeling or other techniques used with
unlabeled data.
**6. State-of-the-Art Models:

Recent advances in deep learning have led to semi-supervised models that can effectively leverage
large amounts of unlabeled data. These models often incorporate self-training, consistency
regularization, and other techniques to improve performance.
**7. Ethical Considerations:

18
Machine Learning –Sem-V

Semi-supervised learning can also raise ethical questions, especially when dealing with sensitive data.
Ensuring privacy and security is crucial when using unlabeled data, and data anonymization techniques
may be necessary.

Let’s check the take away from this lecture

Exercise:
1. What are the potential benefits of co-training, and in what scenarios does it excel in improving
model performance?
2. What challenges might arise in co-training, and how can these challenges be addressed to
ensure the effectiveness of the approach?
3. Could you provide a practical example or use case where co-training could be applied to
enhance machine learning model performance, and describe how it would be implemented in that
context?

Learning from this lecture: Learners will be able to understand Semi- supervised learning using
labelled and unlabelled data

Lecture: 9
Semi- supervised learning using labelled and unlabelled data

Learning objective: Learners shall be able to understand Semi- supervised learning using labelled
and unlabelled data

Course Content

Types of Semi-Supervised Learning:

Self-Training:

Self-training is a fundamental technique in semi-supervised learning. It involves iteratively using a

model trained on labeled data to predict labels for unlabeled data, effectively expanding the labeled
dataset. These pseudo-labeled data points are then used for model retraining.

Co-Training:
- Co-training is a semi-supervised approach where multiple views or representations of data are used.
Two (or more) models are trained on different subsets of features or data, and they co-train each other
by providing additional context from their own views.

Self-training is a popular technique in semi-supervised learning that leverages the predictions of a

model trained on labeled data to expand the labeled dataset. It's a simple yet effective approach for
utilizing unlabeled data to improve the performance of a machine learning model.

Here's a more detailed explanation of self-training:

19
Reinforcement Learning -Sem-VI

20

1. Initialization:

Self-training begins with a limited set of labeled data, typically acquired through manual annotation
or other means.
2. Initial Model Training:

An initial model is trained using the available labeled data. This model provides predictions for
unlabeled data points.
3. Pseudo-Labeling:

The trained model is then used to make predictions on the unlabeled data. These model-generated
labels are referred to as "pseudo-labels."
For example, if it's a binary classification problem, the model assigns each unlabeled data point a
probability score or class label.
4. Expanding the Labeled Dataset:

Data points with high-confidence pseudo-labels (i.e., those for which the model is relatively certain
about the prediction) are added to the labeled dataset.
The model can also be conservative by setting a threshold for pseudo-label confidence, only adding
data points where the model's confidence exceeds a certain level.
5. Model Re-Training:

The labeled dataset is now expanded with the newly pseudo-labeled data.
The model is re-trained using this augmented dataset, which now includes both the original labeled
data and the pseudo-labeled data.
6. Iterative Process:

The process of pseudo-labeling, expanding the labeled dataset, and re-training the model is typically
iterative. Multiple rounds of self-training can be performed to further improve the model's
performance.
7. Benefits and Considerations:

Self-training is particularly useful when there's a scarcity of labeled data but an abundance of unlabeled
data.
It allows the model to learn from the unlabeled data, potentially capturing patterns and information
that might not be present in the small labeled dataset.
However, it comes with a caveat: pseudo-labels can be noisy, leading to errors in the expanded labeled
dataset.
8. Challenges and Thresholding:

One challenge in self-training is setting an appropriate threshold for pseudo-label confidence. If the
threshold is too low, noisy pseudo-labels can hurt model performance. If it's too high, valuable
information from unlabeled data may be missed.

Co-Training in Semi-Supervised Learning:

Co-training is a semi-supervised learning technique that utilizes multiple sources of data or multiple
views of the same data to improve the performance of a machine learning model. Co-training is
particularly effective when dealing with datasets where each data point is represented by multiple

20
Machine Learning –Sem-V

feature sets or when multiple sources of information are available.

Here's a more detailed explanation of co-training:

1. Multiple Views or Feature Sets:

Co-training assumes that each data point in the dataset is described by multiple feature sets or views.
These views can represent different perspectives or sources of information about the same underlying
data.
2. Initialization:

Co-training begins with a limited set of labeled data points.

3. Multiple Models:

Two (or more) machine learning models are initialized, each focusing on learning from a different
view or feature set of the data.
For example, in a text classification task, one model might learn from the text content, while another
model learns from meta-data like timestamps or author information.
4. Initial Model Training:

The initial models are trained using the labeled data and their respective feature sets or views.
5. Iterative Process:

The core of co-training is an iterative process:

Each model uses its own view to make predictions on the unlabeled data.
Data points with high-confidence predictions (i.e., where the model is confident about the prediction)
are pseudo-labeled and added to the labeled data of the other model(s).
The models are re-trained using their expanded labeled datasets.
This process is repeated for a specified number of iterations or until convergence.
6. Knowledge Transfer:

Co-training leverages the diversity of information present in the multiple views or feature sets to
improve the models' performance.
Knowledge transfer occurs as each model learns from the labeled data and the pseudo-labeled data
generated by other models.
7. Benefits and Considerations:

Co-training can be highly effective when there is redundancy or complementary information across
the multiple views.
It's particularly valuable when labeled data is scarce but multiple sources of information are available.
However, co-training assumes that the different views are somewhat independent and that errors made
by one model can be corrected by the other(s).
8. Challenges and Model Synchronization:

One challenge in co-training is ensuring that the multiple models stay synchronized and do not drift
apart in their predictions. Techniques to achieve model synchronization are crucial for co-training's
success.

Let’s check the take away from this lecture

Exercise:

21
Reinforcement Learning -Sem-VI

22
1. What is co-training in semi-supervised learning, and what distinguishes it from other semi-
supervised techniques?
2. Can you explain why co-training is particularly useful when dealing with datasets that have
multiple feature sets or views for each data point?
3. How does the iterative process in co-training work, and why is it a key component of this
technique?

Learning from this lecture: Learners will be able to understand Semi- supervised learning using labelled and
unlabelled data

Conclusion: Thus, the learners successfully summarized UnSupervised and Semi-supervised

Learning Algorithm.

Short Answer Questions:

1. Compare and contrast the basic principles of K-Means clustering and Hierarchical
Agglomerative Clustering. What are the key differences in their approaches to clustering data?
2. In what situations would you choose DBSCAN over K-Means clustering, and vice versa? Can
you provide examples to illustrate the strengths of each method?
3. Explain the concept of self-training in the context of semi-supervised learning. How does it use
unlabeled data to improve model performance, and what are its potential advantages and drawbacks?
4. Describe co-training in semi-supervised learning, emphasizing its use of multiple views or
feature sets. How does co-training leverage diverse sources of information to enhance model
performance, and what are its main considerations?

Long Answer Questions:

1. Can you provide an example of a real-world application where clustering methods, such as K-
Means or DBSCAN, could be integrated with semi-supervised learning techniques like self-training
or co-training to solve a specific problem? What benefits could such an integration offer in that
context?
2. How does hierarchical clustering differ from K-Means clustering, and what is the significance
of dendrograms in hierarchical clustering?
3. Describe the key feature of DBSCAN, which distinguishes it from other clustering algorithms,
and how it identifies clusters based on density.

Set of Questions for FA/IA/ESE

1. What is self-training in semi-supervised learning, and how does it leverage unlabeled data to
improve model performance?
2. Can you explain the concept of co-training in semi-supervised learning and its primary
advantage when multiple sources of data are available?

22
Machine Learning –Sem-V

3. Provide an example of a real-world application where clustering (e.g., K-Means) and semi-
supervised learning (e.g., self-training or co-training) could be used together to solve a practical
problem.

References:
Machine Learning – An Algorithmic Perspective Stephen Marsland Chapman and Hall/CRC
Machine Learning and Pattern Recognition Series ,First 2014

Self-evaluation
Name of
Student
Class
Roll No.
Subject Machine Learning
3
Module No.
[Link] Tick
Your choice
Do you understand Distance based clustering o Yes
1.
Algorithms? o No
Do you understand K means? o Yes
2.
o No
Do you understand K-medoids? o Yes
3.
o No
Do you understand Hierarchical clustering o Yes
4.
o No
Do you understand Hierarchical Agglomerative o Yes
5
Clustering o No
Do you understand Density Based Scan Clustering o Yes
6.
(DBSCAN) o No
Do you understand Gaussian Clustering Model? o Yes
7
o No
Do you understand Conditional Probability? o Yes
8
o No
Do you understand Semi- supervised learning using o Yes
9 labelled and unlabelled data?
o No

23
Reinforcement Learning -Sem-VI

24
o Yes, Completely.
10 Do you understand module 3 ?
o Partially.
o No, Not at all.

24