UNSUPERVISED LEARNING – DETAILED NOTES
1. Introduction to Unsupervised Learning
Unsupervised learning refers to a class of machine-learning techniques that analyze and cluster
unlabeled datasets. The goal is to discover hidden structures, patterns, and relationships within
data without predefined categories or outcomes.
Key objectives:
• Identify patterns and structures.
• Reduce data dimensionality.
• Group similar items (clustering).
• Detect anomalies.
• Transform data into useful representations.
Applications:
• Market segmentation
• Image compression
• Anomaly detection
• Recommender systems
• Feature engineering
2. Engineering the Data
Data engineering in unsupervised learning involves preparing raw data so clustering or
transformation algorithms can operate effectively.
Steps:
A. Data Cleaning
• Handle missing values (mean, median, KNN imputation)
• Remove duplicates
• Detect and manage outliers
B. Normalization / Standardization
• Min–Max scaling (0,1)
• Z-score standardization
• Robust scaling
C. Feature Encoding
• One-hot encoding
• Label encoding
D. Feature Selection (filter/relevance-based)
• Variance thresholding
• Correlation-based removal
E. Data Transformation
• Log transformation
• Box–Cox transformation
• Binning/discretization
• Polynomial feature generation
3. Overview of Basic Clustering Methods
Clustering identifies groups of similar data points. Common methods include:
A. Means Clustering
A broad class of clustering methods based on minimizing mean distances between points and
cluster centers.
General idea:
• Assign points to nearest cluster mean.
• Recompute mean.
• Repeat until stable.
B. K-Means Clustering
One of the most widely used clustering algorithms.
Steps:
1. Choose k initial cluster centers.
2. Assign each point to the nearest center.
3. Recompute cluster mean.
4. Repeat until convergence.
Strengths:
• Fast and easy
• Works well for spherical clusters
Limitations:
• Sensitive to initialization
• Fails with non-convex clusters
• Requires k to be known
C. Expectation–Maximization (EM) Algorithm & Gaussian Mixture Model (GMM)
GMM assumes data is generated from multiple Gaussian distributions.
EM steps:
1. E-step: Estimate probability of each point belonging to a Gaussian.
2. M-step: Update Gaussian parameters (mean, covariance, weight).
3. Repeat until convergence.
Advantages:
• Soft clustering (probabilistic)
• Flexible: can model elliptical clusters
Disadvantages:
• Computationally expensive
• Requires number of mixtures beforehand
4. Useful Data Transformations
Transformations help algorithms work efficiently.
A. Standardization
• (x – mean) / std
B. Normalization
• Scale to [0,1]
C. Log Transform
• Helps reduce skewness.
D. Binning / Discretization
• Convert continuous attributes to discrete intervals.
• Useful for entropy-based feature selection.
E. Polynomial Features
• Helps generate new interactions between attributes.
F. Whitening Transform
• Removes correlation between features.
5. Dimensionality Reduction Techniques
A. Principal Component Analysis (PCA)
PCA reduces dimensionality by projecting data onto directions of maximum variance.
Steps:
1. Standardize data
2. Compute covariance matrix
3. Compute eigenvalues/eigenvectors
4. Select top-k principal components
5. Project original data to new subspace
Advantages:
• Reduces computation
• Removes multicollinearity
• Enhances visualization
Disadvantages:
• Components lack interpretability
• Linear method only
B. Random Projections
Random projection reduces dimension by projecting data onto a randomly generated
lower-dimensional space.
Properties:
• Fast and scalable
• Preserves distances approximately (Johnson–Lindenstrauss lemma)
C. Compressed Sensing
Compressed sensing reconstructs signals from fewer samples than traditional methods require.
Idea:
• Data can be reconstructed if it is “sparse” in some domain.
• Uses L1 optimization.
Applications:
• MRI reconstruction
• Signal processing
D. Entropy-based Attribute Discretization
Entropy-based discretization divides continuous attributes into intervals based on information gain.
Steps:
1. Sort values.
2. Select cut point that maximizes information gain.
3. Recursively apply on sub-intervals.
Uses:
• Converts continuous → discrete
• Improves rule-based and tree algorithms
E. PCA for Attribute Reduction
PCA identifies redundant features based on covariance.
Uses:
• Remove correlated attributes
• Reduce noise
• Improve clustering accuracy
F. Rough Sets–Based Attribute Reduction
Rough set theory deals with vagueness & uncertainty.
Key terms:
• Lower approximation: definite members of a concept.
• Upper approximation: possible members.
• Reducts: minimal subset of attributes preserving classification ability.
Steps in attribute reduction:
1. Identify indiscernibility relation.
2. Find redundant attributes.
3. Generate reduct sets.
Advantages:
• Does not require probability distribution
• Handles uncertainty and vagueness
6. Summary
Unsupervised learning is a powerful framework for revealing hidden structures in data. Techniques
such as clustering, PCA, random projections, and rough-set-based attribute reduction allow
analysts to understand patterns, reduce complexity, and improve model performance. Proper data
engineering ensures these methods deliver accurate and meaningful insights.