MODULE III – UNSUPERVISED LEARNING

UNSUPERVISED LEARNING:
Unsupervised learning is a type of machine learning that analyzes and models data without
labelled responses or predefined categories. Unlike supervised learning, where the algorithm
learns from input-output pairs, unsupervised learning algorithms work solely with input data
and aim to discover hidden patterns, structures or relationships within the dataset
independently, without any human intervention or prior knowledge of the data's meaning.

The image shows set of animals like elephants, camels and cows that represents raw
data that the unsupervised learning algorithm will process.
• The "Interpretation" stage signifies that the algorithm doesn't have predefined labels or
categories for the data. It needs to figure out how to group or organize the data based
on inherent patterns.
• An algorithm represents unsupervised learning process which can be clustering,
dimensionality reduction or anomaly detection to identify patterns in the data.
• The processing stage shows the algorithm working on the data.
The output shows the results of the unsupervised learning process. In this case, the algorithm
might have grouped the animals into clusters based on their species (elephants, camels, cows).
Working of Unsupervised Learning
The working of unsupervised machine learning can be explained in these steps:
Collect Unlabeled Data
• Gather a dataset without predefined labels or categories.
• Example: Images of various animals without any tags.
2. Select an Algorithm
 • Choose a suitable unsupervised algorithm such as clustering like K-Means, association
rule learning like Apriori or dimensionality reduction like PCA based on the goal.
3. Train the Model on Raw Data
• Feed the entire unlabeled dataset to the algorithm.
• The algorithm looks for similarities, relationships or hidden structures within the data.
4. Group or Transform Data
• The algorithm organizes data into groups (clusters), rules or lower-dimensional forms
without human input.
• Example: It may group similar animals together or extract key patterns from large
datasets.
5. Interpret and Use Results
• Analyze the discovered groups, rules or features to gain insights or use them for further
tasks like visualization, anomaly detection or as input for other models.
Applications of Unsupervised learning:
Unsupervised learning has diverse applications across industries and domains. Key
applications include:

• Customer Segmentation: Algorithms cluster customers based on purchasing behavior

or demographics, enabling targeted marketing strategies.
• Anomaly Detection: Identifies unusual patterns in data, aiding fraud detection,
cybersecurity and equipment failure prevention.
• Recommendation Systems: Suggests products, movies or music by analyzing user
behavior and preferences.
• Image and Text Clustering: Groups similar images or documents for tasks like
organization, classification or content recommendation.
• Social Network Analysis: Detects communities or trends in user interactions on social
media platforms.
Advantages:
• No need for labeled data: Works with raw, unlabeled data hence saving time and effort
on data annotation.
• Discovers hidden patterns: Finds natural groupings and structures that might be
missed by humans.
• Handles complex and large datasets: Effective for high-dimensional or vast amounts
of data.
• Useful for anomaly detection: Can identify outliers and unusual data points without
prior examples.
 Challenges:
Here are the key challenges of unsupervised learning:
• Noisy Data: Outliers and noise can distort patterns and reduce the effectiveness of
algorithms.
• Assumption Dependence: Algorithms often rely on assumptions (e.g., cluster shapes)
which may not match the actual data structure.
• Overfitting Risk: Overfitting can occur when models capture noise instead of
meaningful patterns in the data.
• Limited Guidance: The absence of labels restricts the ability to guide the algorithm
toward specific outcomes.
• Cluster Interpretability: Results such as clusters may lack clear meaning or alignment
with real-world categories.
• Sensitivity to Parameters: Many algorithms require careful tuning of hyperparameters
such as the number of clusters in k-means.
• Lack of Ground Truth: Unsupervised learning lacks labeled data making it difficult
to evaluate the accuracy of results.
DIFFERENCE BETWEEN SUPERVISED AND UNSUPERVISED LEARNING:

Aspect Supervised Learning Unsupervised Learning

Uses labeled data (input features + Uses unlabeled data (only input
Input Data
corresponding outputs). features, no outputs).

Predicts outcomes or classifies data based Discovers hidden patterns, structures,

Goal
on known labels. or groupings in data.

Computational Less complex, as the model learns from More complex, as the model must find
Complexity labeled data with clear guidance. patterns without any guidance.

Two types : Classification (for discrete

Types outputs) or regression (for continuous Clustering and association
outputs).

Model can be tested and evaluated using Cannot be tested in the traditional
Testing the Model
labeled test data. sense, as there are no labels.
TYPES OF UNSUPERVISED LEARNING:
1. Clustering Algorithms
Clustering is an unsupervised machine learning technique that groups unlabeled data into
clusters based on similarity. Its goal is to discover patterns or relationships within the data
without any prior knowledge of categories or labels.
• Groups data points that share similar features or characteristics.
• Helps find natural groupings in raw, unclassified data.
• Commonly used for customer segmentation, anomaly detection and data organization.
• Works purely from the input data without any output labels.
• Enables understanding of data structure for further analysis or decision-making.
Some common clustering algorithms:
• K-means Clustering: Groups data into K clusters based on how close the points are to
each other.
• Hierarchical Clustering: Creates clusters by building a tree step-by-step, either
merging or splitting groups.
• Density-Based Clustering (DBSCAN): Finds clusters in dense areas and treats
scattered points as noise.
• Mean-Shift Clustering: Discovers clusters by moving points toward the most crowded
areas.
• Spectral Clustering: Groups data by analyzing connections between points using
graphs.
2. Association Rule Learning
Association rule learning is a rule-based unsupervised learning technique used to discover
interesting relationships between variables in large datasets. It identifies patterns in the form of
“if-then” rules, showing how the presence of some items in the data implies the presence of
others.
• Finds frequent item combinations and the rules connecting them.
• Commonly used in market basket analysis to understand product purchase
relationships.
• Helps retailers design promotions and cross-selling strategies.
Some common Association Rule Learning algorithms:
• Apriori Algorithm: Finds patterns by exploring frequent item combinations step-by-
step.
 • FP-Growth Algorithm: An Efficient Alternative to Apriori. It quickly identifies
frequent patterns without generating candidate sets.
• Eclat Algorithm: Uses intersections of itemsets to efficiently find frequent patterns.
• Efficient Tree-based Algorithms: Scales to handle large datasets by organizing data
in tree structures.
3. Dimensionality Reduction
Dimensionality reduction is the process of decreasing the number of features or variables in a
dataset while retaining as much of the original information as possible. This technique helps
simplify complex data making it easier to analyze and visualize. It also improves the efficiency
and performance of machine learning algorithms by reducing noise and computational cost.
• It reduces the dataset’s feature space from many dimensions to fewer, more meaningful
ones.
• Helps focus on the most important traits or patterns in the data.
• Commonly used to improve model speed and reduce overfitting.
Here are some popular Dimensionality Reduction algorithms:
• Principal Component Analysis (PCA): Reduces dimensions by transforming data
into uncorrelated principal components.
• Linear Discriminant Analysis (LDA): Reduces dimensions while maximizing class
separability for classification tasks.
• Non-negative Matrix Factorization (NMF): Breaks data into non-negative parts to
simplify representation.
• Locally Linear Embedding (LLE): Reduces dimensions while preserving the
relationships between nearby points.
• Isomap: Captures global data structure by preserving distances along a manifold.

CLUSTERING:
Clustering is an unsupervised machine learning technique that groups similar data points
together into clusters based on their characteristics, without using any labeled data. The
objective is to ensure that data points within the same cluster are more similar to each other
than to those in different clusters, enabling the discovery of natural groupings and hidden
patterns in complex datasets.
• Goal: Discover the natural grouping or structure in unlabeled data without
predefined categories.
• How: Data points are assigned to clusters based on similarity or distance measures.
 • Similarity Measures: Can include Euclidean distance, cosine similarity or other
metrics depending on data type and clustering method.
• Output: Each group is assigned a cluster ID, representing shared characteristics within
the cluster.

For example, if we have customer purchase data, clustering can group customers with similar
shopping habits. These clusters can then be used for targeted marketing, personalized
recommendations or customer segmentation.
TYPES OF CLUSTERING
Let's see the types of clustering,
1. Hard Clustering: In hard clustering, each data point strictly belongs to exactly one cluster,
no overlap is allowed. This approach assigns a clear membership, making it easier to interpret
and use for definitive segmentation tasks.
• Example: If clustering customer data into 2 segments, each customer belongs fully to
either Cluster 1 or Cluster 2 without partial memberships.
• Use cases: Market segmentation, customer grouping, document clustering.
• Limitations: Cannot represent ambiguity or overlap between groups; boundaries are
crisp.
Let's see an example to see the difference between the hard and soft clustering using a
distribution,
 int Hard Clustering Soft Clustering

A Cluster 1 Cluster 1: 0.91, Cluster 2: 0.09

B Cluster 2 Cluster 1: 0.30, Cluster 2: 0.70

C Cluster 3 Cluster 1: 0.17, Cluster 2: 0.83

D Cluster 4 Cluster 1: 1.00, Cluster 2: 0.00

2. Soft Clustering: Soft clustering assigns each data point a probability or degree of
membership to multiple clusters simultaneously, allowing data points to partially belong to
several groups.
• Example: A data point may have a 70% membership in Cluster 1 and 30% in Cluster
2, reflecting uncertainty or overlap in group characteristics.
• Use cases: Situations with overlapping class boundaries, fuzzy categories like customer
personas or medical diagnosis.
• Benefits: Captures ambiguity in data, models gradual transitions between clusters.
Types of Clustering Methods:
Clustering methods can be classified on the basis of how they form clusters,
1. Centroid-based Clustering (Partitioning Methods)
Centroid-based clustering organizes data points around central prototypes called centroids,
where each cluster is represented by the mean (or medoid) of its members. The number of
clusters is specified in advance and the algorithm allocates points to the nearest centroid,
making this technique efficient for spherical and similarly sized clusters but sensitive to outliers
and initialization.
Algorithms:
• K-means: Iteratively assigns points to nearest centroid and recalculates centroids to
minimize intra-cluster variance.
• K-medoids: Similar to K-means but uses actual data points (medoids) as centers, robust
to outliers.
Pros:
• Fast and scalable for large datasets.
 • Simple to implement and interpret.
Cons:
• Requires pre-knowledge of kk.
• Sensitive to initialization and outliers.
• Not suitable for non-spherical clusters.
2. Density-based Clustering (Model-based Methods)
Density-based clustering defines clusters as contiguous regions of high data density separated
by areas of lower density. This approach can identify clusters of arbitrary shapes, handles noise
well and does not require predefining the number of clusters, though its effectiveness depends
on chosen density parameters.
Algorithms:
• DBSCAN (Density-Based Spatial Clustering of Applications with Noise): Groups
points with sufficient neighbors; labels sparse points as noise.
• OPTICS (Ordering Points To Identify Clustering Structure): Extends DBSCAN to
handle varying densities.
Pros:
• Handles clusters of varying shapes and sizes.
• Does not require cluster count upfront.
• Effective in noisy datasets.
Cons:
• Difficult to choose parameters like epsilon and min points.
• Less effective for varying density clusters (except OPTICS).
3. Connectivity-based Clustering (Hierarchical Clustering)
Connectivity-based (or hierarchical) clustering builds nested groupings of data by evaluating
how data points are connected to their neighbors. It creates a dendrogram a tree-like structure
that reflects relationships at various granularity levels and does not require specifying cluster
numbers in advance, but can be computationally intensive.
Approaches:
• Agglomerative (Bottom-up): Start with each point as a cluster; iteratively merge closest
clusters.
• Divisive (Top-down): Start with one cluster; iteratively split into smaller clusters.
Pros:
 • Provides a full hierarchy, easy to visualize.
• No need to specify number of clusters upfront.
Cons:
• Computationally intensive for large datasets.
• Merging/splitting decisions are irreversible.
4. Distribution-based Clustering
Distribution-based clustering assumes data is generated from a mixture of probability
distributions, such as Gaussian distributions and assigns points to clusters based on statistical
likelihood. This method supports clusters with flexible shapes and overlaps, but usually
requires specifying the number of distributions.
Algorithm:
• Gaussian Mixture Model (GMM): Fits data as a weighted mixture of Gaussian
distributions; assigns data points based on likelihood.
Pros:
• Flexible cluster shapes.
• Provides probabilistic memberships.
• Suitable for overlapping clusters.
Cons:
• Requires specifying number of components.
• Computationally more expensive.
• Sensitive to initialization.
5. Fuzzy Clustering
Fuzzy clustering extends traditional methods by allowing each data point to belong to multiple
clusters with varying degrees of membership. This approach captures ambiguity and soft
boundaries in data and is particularly useful when the clusters overlap or boundaries are not
clear-cut.
Algorithm:
• Fuzzy C-Means: Similar to K-means but with fuzzy memberships updated iteratively.
Pros:
• Models data ambiguity explicitly.
• Useful for complex or imprecise data.
Cons:
 • Choosing fuzziness parameter can be tricky.
• Computational overhead compared to hard clustering.
Use Cases:
• Customer Segmentation: Grouping customers based on behavior or demographics for
targeted marketing and personalized services.
• Anomaly Detection: Identifying outliers or fraudulent activities in finance, network
security and sensor data.
• Image Segmentation: Dividing images into meaningful parts for object detection,
medical diagnostics or computer vision tasks.
• Recommendation Systems: Clustering user preferences to recommend movies,
products or content tailored to different groups.
• Market Basket Analysis: Discovering products frequently bought together to optimize
store layouts and promotions.
HIERARCHICAL CLUSTERING :
Hierarchical clustering is an unsupervised learning technique used to group similar data points
into clusters by building a hierarchy (tree-like structure). Unlike flat clustering like k-
means hierarchical clustering does not require specifying the number of clusters in advance.
The algorithm builds clusters step by step either by progressively merging smaller clusters or
by splitting a large cluster into smaller ones. The process is often visualized using
a dendrogram, which helps to understand data similarity.

Imagine we have four fruits with different weights: an apple (100g), a banana (120g), a cherry
(50g) and a grape (30g). Hierarchical clustering starts by treating each fruit as its own group.
• Start with each fruit as its own cluster.
• Merge the closest items: grape (30g) and cherry (50g) are grouped first.
 • Next, apple (100g) and banana (120g) are grouped.
• Finally, these two clusters merge into one.
Finally all the fruits are merged into one large group, showing how hierarchical clustering
progressively combines the most similar data points.
Dendrogram:
A dendrogram is like a family tree for clusters. It shows how individual data points or groups
of data merge together. The bottom shows each data point as its own group and as we move up,
similar groups are combined. The lower the merge point, the more similar the groups are. It
helps us see how things are grouped step by step.

• At the bottom of the dendrogram the points P, Q, R, S and T are all separate.
• As we move up, the closest points are merged into a single group.
• The lines connecting the points show how they are progressively merged based on
similarity.
• The height at which they are connected shows how similar the points are to each other;
the shorter the line the more similar they are
Types of Hierarchical Clustering:

Now we understand the basics of hierarchical clustering. There are two main types of
hierarchical clustering.
1. Agglomerative Clustering
2. Divisive clustering

1. Hierarchical Agglomerative Clustering:

It is also known as the bottom-up approach or hierarchical agglomerative clustering (HAC).
Bottom-up algorithms treat each data as a singleton cluster at the outset and then successively
agglomerate pairs of clusters until all clusters have been merged into a single cluster that
contains all data.
Workflow for Hierarchical Agglomerative clustering:

1. Start with individual points: Each data point is its own cluster. For example if
we have 5 data points we start with 5 clusters each containing just one data point.
2. Calculate distances between clusters: Calculate the distance between every pair of
clusters. Initially since each cluster has one point this is the distance between the two
data points.
3. Merge the closest clusters: Identify the two clusters with the smallest distance and
merge them into a single cluster.
4. Update distance matrix: After merging we now have one less cluster. Recalculate the
distances between the new cluster and the remaining clusters.
5. Repeat steps 3 and 4: Keep merging the closest clusters and updating the distance
matrix until we have only one cluster left.
6. Create a dendrogram: As the process continues we can visualize the merging of
clusters using a tree-like diagram called a dendrogram. It shows the hierarchy of how
clusters are merged.

2. Hierarchical Divisive clustering:

Divisive clustering is also known as a top-down approach. Top-down clustering requires a

method for splitting a cluster that contains the whole data and proceeds by splitting clusters
recursively until individual data have been split into singleton clusters.

Workflow for Hierarchical Divisive clustering :

1. Start with all data points in one cluster: Treat the entire dataset as a single large
cluster.
2. Split the cluster: Divide the cluster into two smaller clusters. The division is typically
done by finding the two most dissimilar points in the cluster and using them to
separate the data into two parts.
3. Repeat the process: For each of the new clusters, repeat the splitting process: Choose
the cluster with the most dissimilar points and split it again into two smaller clusters.
4. Stop when each data point is in its own cluster: Continue this process until every
data point is its own cluster or the stopping condition (such as a predefined number
of clusters) is met.
DIFFERENCE BETWEEN AGGLOMERATIVE CLUSTERING AND DIVISIVE CLUSTERING

Parameters Agglomerative Clustering Divisive Clustering

Approach Bottom-up: Starts with individual Top-down: Starts with all data in one
points and merges them. cluster and splits.

Complexity Level More computationally expensive Less computationally expensive but

due to pairwise distance requires careful cluster splitting.
calculations.

Handling Outliers Better at handling outliers, as Outliers may lead to inefficient

outliers can be absorbed into larger splitting and suboptimal results.
clusters.

Interpretability More interpretable due to clear Can be harder to interpret due to

cluster merging in the dendrogram. recursive splitting decisions.

Implementation Scikit-learn provides multiple Not widely implemented in major

linkage methods such as "ward," libraries like Scikit-learn and SciPy.
"complete," "average," and
"single."
Image segmentation, customer Less common but can be used in
Example Applications
segmentation, document clustering, hierarchical data analysis.
etc.
HIERARCHICAL CLUSTERING IN CLOUD MANAGEMENT:

Resource Allocation and Load Balancing

 Efficient resource allocation is a major challenge in cloud environments, where

multiple virtual machines (VMs) and applications compete for CPU, memory, storage,
and network resources.

 Functions:

❖ Dynamic resource allocation

❖ Load redistribution

❖ Efficient scheduling
Anomaly Detection and Fault Management

 Cloud environments generate vast amounts of operational data, including metrics from
servers, networks, and applications.
 Hierarchical clustering can be used to group normal behavior patterns of system
components based on features like response time, network traffic, and resource
utilization.

This helps in early detection of:

❖ Hardware or software failures

❖ Network issues

❖ Security breaches or DDoS attacks

This approach reduces downtime and improves fault tolerance in cloud operations.

Application Profiling and Deployment Optimization

 Cloud platforms often host a wide variety of applications, each with different
performance requirements.

 Hierarchical clustering can be applied to profile applications based on metrics such as

execution time, data transfer rate, memory needs, and response latency.

 Data-intensive analytics applications can be clustered together and deployed on nodes

with high storage capacity.

 This leads to more efficient cloud deployment strategies and better SLA (Service Level
Agreement) compliance.

Cost Optimization and Pricing Strategy

 Hierarchical clustering can also be used by cloud service providers to analyze customer
usage data and identify groups of users with similar consumption patterns.
  For example, some users may have steady, predictable workloads, while others may
have bursty, seasonal usage.

 For example, a cluster of users with high storage usage but low compute can be offered
specialized storage packages at lower costs, while users with fluctuating compute needs
can be offered auto-scaling plans

 This benefits both the provider and the customer by reducing operational costs and
improving customer satisfaction.
K MEANS ALGORITHM:

K-means clustering is a way of grouping data based on how similar or close the data points are
to each other. Imagine you have a bunch of points, and you want to group them into clusters.
The algorithm works by first randomly picking some central points (called centroids) and then
assigning every data point to the nearest centroid.

Objective:

• Grouping Similar Data Points

K-Means is designed to cluster data points that share common traits, allowing patterns or trends
to emerge. Whether analyzing customer behavior or images, the method helps reveal hidden
relationships within your dataset.

• Minimizing Within-Cluster Distance

Another objective is to keep data points in each group as close to the cluster's centroid as
possible. Reducing this internal distance results in compact, cohesive clusters, enhancing the
accuracy of your results.

• Maximizing Between-Cluster Distance

K-Means also aims to maintain clear separation between different clusters. By maximizing the
distance between groups, the algorithm ensures that each cluster remains distinct, providing a
better understanding of data categories without overlap.
Working principle:

K-MEANS CLUSTERING ALGORITHM:

Step 1: Choose K random points as cluster centers called centroids.

We randomly pick K (centroids). We name them c1,c2,..... ck, and we can say that

Where C is the set of all centroids.

Step 2: Assign each x(i) to the closest cluster by implementing euclidean distance (i.e.,
calculating its distance to each centroid)

We assign each data point to its nearest center, which is accomplished by calculating the
euclidean distance.

Where dist() is the Euclidean distance.

Here, we calculate each x value's distance from each c value, i.e. the distance between x1-c1,
x1-c2, x1-c3, and so on. Then we find which is the lowest value and assign x1 to that centroid.
Similarly, we find the minimum distance for x2, x3, etc.
Step 3: Identify new centroids by taking the average of the assigned points.

We identify the actual centroid by taking the average of all the points assigned to that cluster.

Where Si is the set of all points assigned to the ith cluster.

It means the original point, which we thought was the centroid, will shift to the new position,
which is the actual centroid for each of these groups.

Step 4:

Keep repeating step 2 and step 3 until convergence is achieved

Advantages :
1. Simple and easy to implement: The k-means algorithm is easy to understand and
implement, making it a popular choice for clustering tasks.
2. Fast and efficient: K-means is computationally efficient and can handle large datasets
with high dimensionality.
3. Scalability: K-means can handle large datasets with many data points and can be easily
scaled to handle even larger datasets.
4. Flexibility: K-means can be easily adapted to different applications and can be used
with varying metrics of distance and initialization methods.
Applications :

1. Data Segmentation: One of the most common uses of K-Means is segmenting data
into distinct groups. For example, businesses use K-Means to group customers based
on behavior, such as purchasing patterns or website interaction.

2. Image Compression: K-Means can be used to reduce the complexity of images by

grouping similar pixels into clusters, effectively compressing the image. This is useful
for image storage and processing.
3. Anomaly Detection: K-Means can be applied to detect anomalies or outliers by
identifying data points that do not belong to any of the clusters.
4. Document Clustering: In natural language processing (NLP), K-Means is used to
group similar documents or articles together. It’s often used in applications like
recommendation systems or news categorization.

5. Organizing Large Datasets: When dealing with large datasets, K-Means can help in
organizing the data into smaller, more manageable chunks based on similarities,
improving the efficiency of data analysis.

Challenges:

K-Means algorithm has the following limitations:

• Choosing the Right Number of Clusters (kk): One of the biggest challenges is
deciding how many clusters to use.

• Sensitive to Initial Centroids: The final clusters can vary depending on the initial
random placement of centroids.

• Non-Spherical Clusters: K-Means assumes that the clusters are spherical and equally
sized. This can be a problem when the actual clusters in the data are of different shapes
or densities.

• Outliers: K-Means is sensitive to outliers, which can distort the centroid and,
ultimately, the clusters.

K-MEDOIDS CLUSTERING:

K-Medoids (also called Partitioning Around Medoid) algorithm was proposed in 1987 by
Kaufman and Rousseeuw. A medoid can be defined as a point in the cluster, whose
dissimilarities with all the other points in the cluster are minimum. The dissimilarity of the
medoid(Ci) and object(Pi) is calculated by using E = |Pi - Ci|

Algorithm:
 1. Initialize: select k random points out of the n data points as the medoids.

2. Associate each data point to the closest medoid by using any common distance metric
methods.

3. While the cost decreases: For each medoid m, for each data o point which is not a
medoid:

• Swap m and o, associate each data point to the closest medoid, and recompute
the cost.

• If the total cost is more than that in the previous step, undo the swap.

Example:

If a graph is drawn using the above data points, we obtain the following:

Step 1: Let the randomly selected 2 medoids, so select k = 2, and let C1 -(4, 5) and C2 -(8,
5) are the two medoids.
Step 2: Calculating cost. The dissimilarity of each non-medoid point with the medoids is
calculated and tabulated:

Here we have used Manhattan distance formula to calculate the distance matrices between
medoid and non-medoid points. That formula tell that Distance = |X1-X2| + |Y1-Y2|.

Each point is assigned to the cluster of that medoid whose dissimilarity is less. Points 1, 2, and
5 go to cluster C1 and 0, 3, 6, 7, 8 go to cluster C2. The Cost = (3 + 4 + 4) + (3 + 1 + 1 + 2 +
2) = 20

Step 3: randomly select one non-medoid point and recalculate the cost. Let the randomly
selected point be (8, 4). The dissimilarity of each non-medoid point with the medoids - C1 (4,
5) and C2 (8, 4) is calculated and tabulated.

Each point is assigned to that cluster whose dissimilarity is less. So, points 1, 2, and 5 go to
cluster C1 and 0, 3, 6, 7, 8 go to cluster C2. The New cost = (3 + 4 + 4) + (2 + 2 + 1 + 3 + 3) =
22 Swap Cost = New Cost - Previous Cost = 22 - 20 and 2 >0 As the swap cost is not less than
zero, we undo the swap. Hence (4, 5) and (8, 5) are the final medoids.
Advantages:

1. It is simple to understand and easy to implement.

2. K-Medoid Algorithm is fast and converges in a fixed number of steps.
3. PAM is less sensitive to outliers than other partitioning algorithms.

Disadvantages:

1. The main disadvantage of K-Medoid algorithms is that it is not suitable for clustering
non-spherical (arbitrarily shaped) groups of objects. This is because it relies on
minimizing the distances between the non-medoid objects and the medoid (the cluster
center) - briefly, it uses compactness as clustering criteria instead of connectivity.

2. It may obtain different results for different runs on the same dataset because the first k
medoids are chosen randomly.

OUTLIERS:
Outliers are data point that is essentially a statistical anomaly, a data point that significantly
deviates from other observations in a dataset. Outliers can arise due to measurement errors,
natural variation, or rare events, and they can have a disproportionate impact on statistical
analyses and machine learning models if not appropriately handled.
Example: If you have the following dataset of student test scores:
[85, 87, 90, 88, 92, 89, 45]

The score 45 is an outlier—it’s much lower than the others.

Note: Outliers can be valid observations or errors in data entry, mesurement, or processing.

Types of Outliers:
1. Global Outliers: Also known as point anomalies, these data points significantly differ
from the rest of the dataset.
 2. Contextual Outliers: These are data points that are considered outliers in a specific
context. For example, a high temperature may be normal in summer but an outlier in
winter.

3. Collective Outliers: A collection of data points that deviate significantly from the rest
of the dataset, even if individual points within the collection are not outliers.

Outlier Detection:

Outlier detection is a process of identifying observations or data points that significantly

deviate from the majority of the data.

Outliers can distort statistical analyses, leading to erroneous conclusions and misleading
interpretations. When calculating means, medians, or standard deviations, outliers can exert
disproportionate influence, skewing the results and undermining the validity of the analysis.
By detecting and appropriately addressing outliers, analysts can mitigate the impact of these
anomalies on statistical measures, ensuring that the insights drawn from the data are
representative and accurate.

Detecting outliers:

• Improving Accuracy: Removing or accurately handling outliers enhances the

performance and predictability of data models.

• Fraud Detection: Outliers can be symptomatic of fraudulent activity, especially in

financial or transaction data.

• Data Quality: Regular outlier detection is crucial to maintain the integrity and quality
of data, which in turn affects the decision-making processes based on this data.

• Model Performance: Outliers can significantly impact the performance of statistical

models, machine learning algorithms, and other analytical techniques. By identifying
and handling outliers appropriately, we can improve the robustness and accuracy of
these models.
 • Insight Generation: Outliers may represent unique or interesting phenomena in the
data. Identifying and analyzing outliers can lead to valuable insights, such as detecting
emerging trends, understanding rare events, or uncovering potential opportunities or
threats.
Methods for Outlier Detection:
Outlier detection is a critical task in data analysis, crucial for ensuring the quality and reliability
of conclusions drawn from data. Different techniques are tailored for varying data types and
scenarios, ranging from statistical methods for general data sets to specialized algorithms for
spatial and temporal data. Some techniques are:

1. Standard Deviation Method

Standard Deviation Method is based on the assumption that the data follows a normal
distribution. Data points outside of three standard deviations from the mean are considered
outliers.

It is commonly used for univariate data analysis where the distribution can be assumed to be
approximately normal.

Step 1: Calculate the average and standard deviation of the data set, if applicable.

Step 2: Define the lower and upper bounds for outliers.

Step 3: Identify outliers as data points that fall outside these bounds:

Example: Dataset: [1, 2, 2, 3, 1, 3, 10]. Find an outlier using the Standard Deviation Method.

2. IQR Method

The Interquartile Range (IQR) method focuses on the spread of the middle 50% of data. It
calculates the IQR as the difference between the 75th and 25th percentiles of the data and
identifies outliers as those points that fall below 1.5 times the IQR below the 25th percentile or
above 1.5 times the IQR above the 75th percentile. This method is robust to outliers and does
not assume a normal distribution.
• Step 1: Find Q1(25th percentage) and Q3(75th percentage)
 • Step 2: IQR = Q3 - Q1.

• Step 3: Find Lower Bound: Q1 - 1.5 × IQR and Upper Bound Lower Bound: Q1 - 1.5
× IQR

It is suitable for datasets with skewed or non-normal distributions. Useful for identifying
outliers in datasets where the spread of the middle 50% of the data is more relevant than the
mean and standard deviation.

Example: Dataset X = {3,5,7,9,11,13,30}, find outlier using the IQR method.

3. Z-Score Method

The Z-score method calculates the number of standard deviations each data point is from the
mean. A Z-score threshold is set, commonly 3, and any data point with a Z-score exceeding
this threshold is considered an outlier. This method assumes a normal distribution and is
sensitive to extreme values in small datasets.

• Step 1: Calculate the mean.

• Step 2: Compute Standard Deviation

• Step 3:Calculate z-scores

• Step 4: Apply Threshold Rule: Mild outlier: |Z| > 2 and Extreme outlier: |Z| > 3

Suitable for datasets with large sample sizes and where the underlying distribution of the data
can be reasonably approximated by a normal distribution.

Example: X={4,5,5,6,7,8,20}, find outlier using the Z-score method.

The choice of outlier detection technique depends on the characteristics of the data, the
underlying distribution, and the specific requirements of the analysis.
Challenges with Outlier Detection:
Detecting outliers effectively poses several challenges:

• Determining the Threshold: Deciding the correct threshold that accurately separates
outliers from normal data is critical and difficult.

• Distinguishing Noise from Outliers: In datasets with high variability or noise, it can
be particularly challenging to differentiate between noise and actual outliers.

• Balancing Sensitivity: An overly aggressive approach to detecting outliers might

eliminate valid data, reducing the richness of the dataset.

Applications of Outlier Detection:

Outlier detection plays a crucial role across various domains, enabling the identification of
anomalies that can indicate errors, fraud, or novel insights. Here are some key applications of
outlier detection with specific examples:

1. Financial Fraud Detection

• Fraud Detection: Outlier detection is extensively used in the financial sector to identify
fraudulent activities. For instance, credit card companies use outlier detection
algorithms to flag unusual spending patterns that may indicate stolen card usage.

• Example: A credit card transaction for a large amount in a foreign country when the
cardholder usually makes small, local purchases could be flagged as an outlier,
triggering a fraud alert.

2. Cybersecurity
• Network Intrusion Detection: Outlier detection is critical in cybersecurity for
identifying unusual patterns of network traffic that could indicate a security breach.
• Example: A sudden increase in data transmission to an external IP address not
previously contacted by the network could be an outlier, suggesting a potential data
exfiltration attack.

3. AI/ML Modeling

• Data Cleaning: To prevent model skewing in training data

• In reducing Bias: Detects biased predictions.

4. Anomaly Detection in Big Data & Cloud Systems

• Cloud Security: To detect unauthorized access in large-scale cloud environments.

• Ensures integrity by flagging corrupted entries.

PRINCIPAL COMPONENT ANALYSIS (PCA):

Principal Component Analysis is an unsupervised learning algorithm that is used for the
dimensionality reduction in machine learning. It is a statistical process that converts the
observations of correlated features into a set of linearly uncorrelated features with the help of
orthogonal transformation. These new transformed features are called the Principal
Components. It is one of the popular tools that is used for exploratory data analysis and
predictive modeling. It is a technique to draw strong patterns from the given dataset by reducing
the variances.

PCA generally tries to find the lower-dimensional surface to project the high-dimensional data.

PCA works by considering the variance of each attribute because the high attribute shows the
good split between the classes, and hence it reduces the dimensionality. Some real-world
applications of PCA are image processing, movie recommendation system, optimizing the
power allocation in various communication channels. It is a feature extraction technique, so
it contains the important variables and drops the least important variable.

The PCA algorithm is based on some mathematical concepts such as:

o Variance and Covariance

o Eigenvalues and Eigen factors

Some common terms used in PCA algorithm:

o Dimensionality: It is the number of features or variables present in the given dataset.

More easily, it is the number of columns present in the dataset.
o Correlation: It signifies that how strongly two variables are related to each other. Such
as if one changes, the other variable also gets changed. The correlation value ranges
from -1 to +1. Here, -1 occurs if variables are inversely proportional to each other, and
+1 indicates that variables are directly proportional to each other.
o Orthogonal: It defines that variables are not correlated to each other, and hence the
correlation between the pair of variables is zero.
o Eigenvectors: If there is a square matrix M, and a non-zero vector v is given. Then v
will be eigenvector if Av is the scalar multiple of v.
o Covariance Matrix: A matrix containing the covariance between the pair of variables
is called the Covariance Matrix.

Properties:

o The principal component must be the linear combination of the original features.
o These components are orthogonal, i.e., the correlation between a pair of variables is
zero.
o The importance of each component decreases when going to 1 to n, it means the 1 PC
has the most importance, and n PC will have the least importance.
General Steps for PCA algorithm:

1. Getting the dataset

Firstly, we need to take the input dataset and divide it into two subparts X and Y,
where X is the training set, and Y is the validation set.
2. Representing data into a structure
Now we will represent our dataset into a structure. Such as we will represent the two-
dimensional matrix of independent variable X. Here each row corresponds to the data
items, and the column corresponds to the Features. The number of columns is the
dimensions of the dataset.
3. Standardizing the data
In this step, we will standardize our dataset. Such as in a particular column, the
features with high variance are more important compared to the features with lower
variance. If the importance of features is independent of the variance of the feature,
then we will divide each data item in a column with the standard deviation of the
column. Here we will name the matrix as Z.
4. Calculating the Covariance of Z
To calculate the covariance of Z, we will take the matrix Z, and will transpose it.
After transpose, we will multiply it by Z. The output matrix will be the Covariance
matrix of Z.
5. Calculating the Eigen Values and Eigen Vectors
Now we need to calculate the eigenvalues and eigenvectors for the resultant
covariance matrix Z. Eigenvectors or the covariance matrix are the directions of the
axes with high information. And the coefficients of these eigenvectors are defined as
the eigenvalues.
6. Sorting the Eigen Vectors
In this step, we will take all the eigenvalues and will sort them in decreasing order,
which means from largest to smallest. And simultaneously sort the eigenvectors
accordingly in matrix P of eigenvalues. The resultant matrix will be named as P*.
7. Calculating the new features Or Principal Components
Here we will calculate the new features. To do this, we will multiply the P* matrix to
the Z. In the resultant matrix Z*, each observation is the linear combination of
original features. Each column of the Z* matrix is independent of each other.
8. Remove less or unimportant features from the new dataset.
The new feature set has occurred, so we will decide here what to keep and what to
 remove. It means, we will only keep the relevant or important features in the new
dataset, and unimportant features will be removed out.

PCA Algorithm:
DBSCAN CLUSTERING:

DBSCAN is a density-based clustering algorithm that groups data points that are closely
packed together and marks outliers as noise based on their density in the feature space. It
identifies clusters as dense regions in the data space separated by areas of lower density. Unlike
K-Means or hierarchical clustering which assumes clusters are compact and spherical,
DBSCAN perform well in handling real-world data irregularities such as:

• Arbitrary-Shaped Clusters: Clusters can take any shape not just circular or convex.
• Noise and Outliers: It effectively identifies and handles noise points without assigning
them to any cluster.
The figure above shows a data set with clustering algorithms: K-Means and Hierarchical
handling compact, spherical clusters with varying noise tolerance while DBSCAN manages
arbitrary-shaped clusters and noise handling.

Key Parameters in DBSCAN

1. eps: This defines the radius of the neighborhood around a data point. If the distance between
two points is less than or equal to eps they are considered neighbors. A common method to
determine eps is by analyzing the k-distance graph. Choosing the right eps is important:
• If eps is too small most points will be classified as noise.

• If eps is too large clusters may merge and the algorithm may fail to distinguish between
them.

2. MinPts: This is the minimum number of points required within the eps radius to form a
dense region. A general rule of thumb is to set MinPts >= D+1 where D is the number of
dimensions in the dataset.

For most cases a minimum value of MinPts = 3 is recommended.

Working of DBSCAN:

DBSCAN works by categorizing data points into three types:

1. Core points which have a sufficient number of neighbors within a specified radius
(eplison)

2. Border points which are near core points but lack enough neighbors to be core points
themselves

3. Noise points which do not belong to any cluster.

By iteratively expanding clusters from core points and connecting density-reachable points,
DBSCAN forms clusters without relying on rigid assumptions about their shape or size

Steps in the DBSCAN Algorithm:

 1. Identify Core Points: For each point in the dataset count the number of points within
its eps neighborhood. If the count meets or exceeds MinPts mark the point as a core
point.

2. Form Clusters: For each core point that is not already assigned to a cluster create a
new cluster. Recursively find all density-connected points i.e points within
the eps radius of the core point and add them to the cluster.

3. Density Connectivity: Two points a and b are density-connected if there exists a

chain of points where each point is within the eps radius of the next and at least one
point in the chain is a core point. This chaining process ensures that all points in a
cluster are connected through a series of dense regions.

4. Label Noise Points: After processing all points any point that does not belong to a
cluster is labeled as noise.

Difference between DBSCAN and K Means:

DBSCAN K-Means

In DBSCAN we need not specify the number It is very sensitive to the number of clusters (k),
of clusters. which must be specified in advance.

Clusters formed in DBSCAN can be of any Clusters formed are spherical or convex in
arbitrary shape. shape

It can work well with datasets having noise It does not work well with outliers data.
and outliers Outliers can skew the clusters in K-Means to a
very large extent.
In DBSCAN two parameters are required for In K-Means only one parameter is required is
training the Model for training the model

MULTIPLE CORRESPONDENCE ANALYSIS (MCA):

• Multiple Correspondence Analysis (MCA) is an extension of Correspondence Analysis

(CA) used for analyzing the pattern of relationships among several categorical
variables.
• It is similar to Principal Component Analysis (PCA) but for categorical data instead of
continuous data.

• MCA helps to visualize and interpret data in a low-dimensional space (usually 2D or

3D).
Purpose:

• To explore relationships between several categorical variables.

• To reduce dimensionality of categorical data.

• To detect patterns, clusters, and associations among categories.

• To visualize individuals (observations) and variable categories simultaneously on a
map.

Concept Description

Variables with discrete categories (e.g., gender, education

Categorical Variables
level, occupation).
A binary matrix representing individuals vs categories (1 if the
Indicator Matrix
individual belongs to a category, 0 otherwise).
Measure of total variance explained in MCA (analogous to
Inertia
variance in PCA).
Each new component formed that explains part of the variance
Dimension (or Axis)
(similar to principal components in PCA).
Represent the amount of inertia (variance) explained by each
Eigenvalues
dimension.

Steps:

Step 1: Data Preparation

• Collect data with categorical variables.

• Convert data into a complete disjunctive table (indicator matrix).

Step 2: Compute the Burt Matrix

• The Burt matrix is the cross-tabulation of all variables.

• It is a square matrix where each block represents the relationship between two
variables.

Step 3: Apply Correspondence Analysis (CA)

• Perform CA on the Burt matrix or the indicator matrix.

• Extract eigenvalues, principal coordinates, and inertia.

Step 4: Interpret Results

• Factor map (plots) show how categories and individuals are positioned.

• Categories close together are similar.

• Dimensions are interpreted based on which categories contribute most to them.
Advantages:

• Handles multiple categorical variables simultaneously.

• Produces easily interpretable visualizations (category maps).

• Reveals hidden patterns and associations among variables.

• Works well for survey data or questionnaire analysis.

Limitations:

• Sensitive to rare categories (low frequencies).

• Interpretation can be subjective.

• Only captures linear relationships between categories.

• Not suitable for mixed data types (use FAMD instead).

Applications
• Market research (customer profiling, brand preference).
• Social sciences (survey analysis).

• Education (student performance by demographic variables).

• Healthcare (disease risk by lifestyle categories).

• Text analysis (categorical features from documents).

SINGULAR VALUE DECOMPOSITION (SVD):

Singular Value Decomposition (SVD) is a factorization method in linear algebra that

decomposes a matrix into three other matrices, providing a way to represent data in terms of
its singular values.

SVD helps you split that table into three parts:

• U: This part tells you about the people (like their general preferences).

• Σ: This part shows how important each factor is (how much each rating matters).
• Vᵀ: This part tells you about the products (how similar they are to each other)

Lets understand this with help of an example: Suppose you have a small table of people’s
ratings for two movies,

Name Movie 1 Rating Movie 2 Rating

mit 5 3
Sanket 4 2
Harsh 2 5
• SVD breaks this table into three smaller parts: one that shows people’s preferences, one
that shows the importance of each movie, and one that shows how similar the movies
are to each other

• Mathematically, the SVD of a matrix AA (of size m×nm×n) is represented

as: A=UΣVTA=UΣVT
This is the Result SVD matrix of matrix A.