Module-4 Dimensionality Reduc on and Clustering
Dimensionality Reduc on
Dimensionality reduc on is a crucial technique in data preprocessing, used to reduce the
number of variables or features in a dataset while retaining its essen al structure and
informa on. This simplifies analysis, enhances visualiza on, and improves the performance of
machine learning algorithms.
1. Subset Selec on
Subset selec on involves iden fying and selec ng a subset of relevant features from the original
dataset. The primary goal is to reduce the dimensionality while retaining the most important
informa on. Techniques include:
Filter Methods: Use sta s cal tests to evaluate feature importance (e.g., correla on,
chi-square test).
Wrapper Methods: Employ machine learning models to evaluate subsets of features
(e.g., forward selec on, backward elimina on).
Embedded Methods: Integrate feature selec on within the learning process (e.g., LASSO
regression).
2. Principal Component Analysis (PCA)
PCA is a sta s cal method that transforms a dataset into a new coordinate system, reducing
dimensions while preserving the variance.
Steps:
1. Standardize the dataset.
2. Compute the covariance matrix.
3. Calculate eigenvalues and eigenvectors.
4. Select principal components with the highest eigenvalues.
5. Project data onto the selected components.
Applica ons: Image compression, gene expression analysis, and exploratory data
analysis.
3. Linear Discriminant Analysis (LDA)
LDA is a supervised dimensionality reduc on technique that maximizes the separability
between classes.
Steps:
1. Compute the mean vectors for each class.
2. Calculate the within-class and between-class sca er matrices.
3. Solve the eigenvalue problem for the sca er matrices.
4. Select linear discriminants corresponding to the largest eigenvalues.
Applica ons: Face recogni on, document classifica on, and fraud detec on.
Clustering
Clustering is an unsupervised learning method that groups data points into clusters based on
similarity. It is widely used in pa ern recogni on, market segmenta on, and anomaly detec on.
1. K-Means Clustering
K-Means is a centroid-based clustering algorithm that par ons data into k clusters.
Algorithm Steps:
1. Ini alize k cluster centroids randomly.
2. Assign each data point to the nearest centroid.
3. Update centroids by calcula ng the mean of assigned points.
4. Repeat steps 2-3 un l convergence.
Strengths: Simple and efficient.
Limita ons: Requires predefining k, sensi ve to outliers.
2. Expecta on-Maximiza on (EM) Algorithm
The EM algorithm is used for probabilis c clustering, o en in Gaussian Mixture Models (GMM).
Steps:
1. Ini alize parameters (mean, covariance, and mixing coefficients).
2. Expecta on (E-step): Calculate responsibili es using the current parameters.
3. Maximiza on (M-step): Update parameters to maximize the likelihood func on.
4. Repeat E and M steps un l convergence.
Applica ons: Image segmenta on, customer segmenta on.
3. Hierarchical Clustering
Hierarchical clustering builds a tree-like structure (dendrogram) represen ng nested groupings
of data points.
Types:
o Agglomera ve: Bo om-up approach, merging smaller clusters itera vely.
o Divisive: Top-down approach, spli ng larger clusters itera vely.
Steps:
1. Compute a distance matrix for all data points.
2. Merge or split clusters based on proximity metrics (e.g., single-linkage, complete-
linkage).
3. Repeat un l the desired number of clusters is reached.
Applica ons: Gene expression analysis, social network analysis.
Applica ons of Dimensionality Reduc on and Clustering
Data Preprocessing: Simplifying data for machine learning models.
Visualiza on: Represen ng high-dimensional data in 2D/3D space.
Pa ern Recogni on: Iden fying inherent groupings in data.
Anomaly Detec on: Detec ng outliers or unusual pa erns.
Market Segmenta on: Grouping customers based on behaviors or preferences.