Module 5
Module 5
• Unsupervised learning : Grouping Unlabeled Items Using K-Means Clustering:
• The k-means clustering algorithm,
• improving cluster performance with post processing,
• bisecting k-means,
• Examples
• Using Principal Component Analysis to Simplify Data:
• Dimensionality reduction techniques,
• Principal component analysis,
• moving coordinate axes,
• Performing PCA in NumPy,
• Examples
Introduction
• Clustering is a type of unsupervised learning that automatically forms clusters of
similar things. It’s like automatic classification. You can cluster almost anything, and
the more similar the items are in the cluster, the better your clusters are.
• one type of clustering algorithm called k-means. It’s called k-means because it finds k
unique clusters, and the center of each cluster is the mean of the values in that cluster.
• Clustering is sometimes called unsupervised classification because it produces the
same result as classification but without having predefined classes.
Unsupervised Learning
• In unsupervised learning, the data provided to the machine learning
algorithm does not have predefined labels or outcomes. The goal is to
discover hidden patterns, structures, or relationships within the data.
Typical applications include:
- Clustering: Grouping similar data points together.
- Dimensionality reduction: Simplifying data representation (e.g., PCA).
- Association: Finding rules or correlations
Clustering in Unsupervised Learning
• Clustering groups a set of objects such that:
- Objects in the same group (cluster) are more similar to each other.
- Objects in different clusters are less similar.
Applications include
• customer segmentation,
• image compression,
• document classification, and
• social network analysis.
K-Means Clustering Algorithm
• K-Means is a popular partitional clustering algorithm that partitions n data points into k
clusters, where k is predefined. It minimizes intra-cluster distance and maximizes inter-cluster
distance.
Algorithm Steps:
• 1. Select the number of clusters (k).
2. Initialize centroids randomly.
3. Assign each data point to the nearest centroid.
4. Recalculate the centroids.
5. Repeat steps 3 and 4 until convergence.
• Objective Function: J = Σ (||x_j - μ_i||²), minimizing within-cluster sum of squares (WCSS).
• Advantages: Simple, fast, easy to interpret.
Limitations: Requires k, sensitive to initialization, assumes spherical clusters, affected by
outliers.
Improving Cluster Performance with Post-Processing
• Post-processing techniques improve cluster performance and interpretability:
- Normalization or scaling: Ensure features have equal influence.
- Choosing optimal k
- Outlier removal to reduce distortion.
- Cluster merging/splitting for balance.
- K-Means++ for better initialization and faster convergence.
Bisecting K-Means Algorithm
• Bisecting K-Means is a hierarchical variant that iteratively splits
clusters using K-Means (k=2).
• Steps:
1. Start with all data in one cluster.
2. Select a cluster to split (highest SSE).
3. Apply K-Means (k=2) to split it.
4. Repeat until k clusters are obtained.
Advantages: Better clustering, less sensitive to initialization, combines
hierarchical and partitional methods.
Examples of K-Means Clustering
• Example 1: Customer Segmentation
- Dataset: Age, Income, Spending Score
- K-Means creates groups such as premium customers, impulse buyers, and budget-conscious
users.
Example 2: Image Compression
- Each pixel (R,G,B) is a data point. K-Means reduces colors to k representative shades.
Example 3: Document Clustering
- Documents represented as TF-IDF vectors grouped by topics like sports, politics, or tech.
Dimensionality Reduction Techniques
• Dimensionality reduction is the process of reducing the number of features
(variables) in a dataset while retaining as much important information as possible.
It is useful for simplifying models, improving computation, and reducing noise.
• Common techniques include:
• Principal Component Analysis (PCA) – linear method to find uncorrelated features
• Linear Discriminant Analysis (LDA) – supervised, maximizes class separability
• t-SNE – non-linear method for visualization
• Autoencoders – neural network-based reduction
Principal Component Analysis (PCA)
• Principal Component Analysis (PCA) is a statistical technique that transforms correlated
variables into a new set of uncorrelated variables called principal components. These
components are ordered by the amount of variance they capture in the data.
• Mathematical steps:
1. Compute covariance matrix of the dataset.
2. Find eigenvalues and eigenvectors.
3. Sort eigenvectors by decreasing eigenvalues.
4. Select top k eigenvectors (principal components).
5. Project data onto these new axes to form the reduced dataset.
• Objective:
PCA seeks directions that maximize the variance of the projected data, ensuring the
transformed dataset retains maximum information.
Moving Coordinate Axes (Geometric Interpretation)
• PCA can be understood as rotating the original coordinate axes so they align with
the directions of maximum variance in the data.
• The first principal component (PC1) captures the most variance, followed by PC2,
which is orthogonal to PC1.
• By retaining only the top components, we effectively compress the data while
keeping essential information.
Examples of PCA Applications
1. Image Compression – Reduce pixel dimensions while maintaining quality.
2. Face Recognition (Eigenfaces) – Identify key facial features using principal components.
3. Data Visualization – Plot high-dimensional data in 2D or 3D using principal components.
4. Financial Analysis – Simplify correlated market indicators into a few main factors.
• Advantages:
• Reduces complexity while preserving variance
• Removes correlation among features
• Improves model performance
• Enables easy visualization
• Limitations:
• Linear method; not effective for non-linear data
• Reduces interpretability of features
• Sensitive to scaling
• May lose some information
• .