ML
CSC701 Machine Learning
Module No. 06
Dimensionality Reduction
By Prof. Merlin Jacob
ML
Module No. 06
Dimensionality Reduction
By Prof. Merlin Jacob
ML
• Reducing the number of input variables for a predictive model is referred to as dimensionality
reduction.
• Fewer input variables can result in a simpler predictive model that may have better
performance when making predictions on new data.
• The most popular technique for dimensionality reduction in machine learning is Principal
Component Analysis.
• This is a technique that comes from the field of linear algebra and can be used as a data
preparation technique to create a projection of a dataset prior to fitting a model.
By Prof. Merlin Jacob
ML
Principal Component Analysis
By Prof. Merlin Jacob
ML
PCA
• Principal Component Analysis is an unsupervised learning algorithm that is used for the
dimensionality reduction in machine learning.
• It is a statistical process that converts the observations of correlated features into a set of
linearly uncorrelated features with the help of orthogonal transformation.
• These new transformed features are called the Principal Components.
• It is one of the popular tools that is used for exploratory data analysis and predictive
modeling.
• It is a technique to draw strong patterns from the given dataset by reducing the variances.
By Prof. Merlin Jacob
ML
• PCA works by considering the variance of each attribute because the high attribute shows the
good split between the classes, and hence it reduces the dimensionality.
• Some real-world applications of PCA are image processing, movie recommendation system,
optimizing the power allocation in various communication channels.
• It is a feature extraction technique, so it contains the important variables and drops the least
important variable.
By Prof. Merlin Jacob
ML
The PCA algorithm is based on some mathematical concepts such as:
• Variance and Covariance
• Eigenvalues and Eigen factors
By Prof. Merlin Jacob
ML
Some common terms used in PCA algorithm:
• Dimensionality: It is the number of features or variables present in the given dataset. More
easily, it is the number of columns present in the dataset.
• Correlation: It signifies that how strongly two variables are related to each other. Such as if
one changes, the other variable also gets changed. The correlation value ranges from -1 to +1.
Here, -1 occurs if variables are inversely proportional to each other, and +1 indicates that
variables are directly proportional to each other.
By Prof. Merlin Jacob
ML
Some common terms used in PCA algorithm:
• Orthogonal: It defines that variables are not correlated to each other, and hence the correlation
between the pair of variables is zero.
• Eigenvectors: If there is a square matrix M, and a non-zero vector v is given. Then v will be
eigenvector if Av is the scalar multiple of v.
• Covariance Matrix: A matrix containing the covariance between the pair of variables is called
the Covariance Matrix.
By Prof. Merlin Jacob
ML
Principal Components in PCA
As described above, the transformed new features or the output of PCA are the Principal
Components. The number of these PCs are either equal to or less than the original features
present in the dataset. Some properties of these principal components are given below:
• The principal component must be the linear combination of the original features.
• These components are orthogonal, i.e., the correlation between a pair of variables is zero.
• The importance of each component decreases when going to 1 to n, it means the 1 PC has the
most importance, and n PC will have the least importance.
By Prof. Merlin Jacob
ML
Steps for PCA algorithm
Getting the dataset
• Firstly, we need to take the input dataset and divide it into two subparts X and Y, where X is
the training set, and Y is the validation set.
Representing data into a structure
• Now we will represent our dataset into a structure. Such as we will represent the two-
dimensional matrix of independent variable X. Here each row corresponds to the data items,
and the column corresponds to the Features. The number of columns is the dimensions of the
dataset.
By Prof. Merlin Jacob
ML
Steps for PCA algorithm
Standardizing the data
• In this step, we will standardize our dataset. Such as in a particular column, the features with
high variance are more important compared to the features with lower variance.
• If the importance of features is independent of the variance of the feature, then we will divide
each data item in a column with the standard deviation of the column. Here we will name the
matrix as Z.
Calculating the Covariance of Z
• To calculate the covariance of Z, we will take the matrix Z, and will transpose it. After
transpose, we will multiply it by Z. The output matrix will be the Covariance matrix of Z.
By Prof. Merlin Jacob
ML
Steps for PCA algorithm
Calculating the Eigen Values and Eigen Vectors
• Now we need to calculate the eigenvalues and eigenvectors for the resultant covariance
matrix Z. Eigenvectors or the covariance matrix are the directions of the axes with high
information. And the coefficients of these eigenvectors are defined as the eigenvalues.
Sorting the Eigen Vectors
• In this step, we will take all the eigenvalues and will sort them in decreasing order, which
means from largest to smallest. And simultaneously sort the eigenvectors accordingly in
matrix P of eigenvalues. The resultant matrix will be named as P*.
By Prof. Merlin Jacob
ML
Steps for PCA algorithm
Calculating the new features Or Principal Components
• Here we will calculate the new features. To do this, we will multiply the P* matrix to the Z.
In the resultant matrix Z*, each observation is the linear combination of original features.
Each column of the Z* matrix is independent of each other.
Remove less or unimportant features from the new dataset.
• The new feature set has occurred, so we will decide here what to keep and what to remove. It
means, we will only keep the relevant or important features in the new dataset, and
unimportant features will be removed out.
By Prof. Merlin Jacob
ML
Applications of Principal Component Analysis
• PCA is mainly used as the dimensionality reduction technique in various AI applications
such as computer vision, image compression, etc.
• It can also be used for finding hidden patterns if data has high dimensions. Some fields
where PCA is used are Finance, data mining, Psychology, etc.
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
Linear Discriminant Analysis
By Prof. Merlin Jacob
ML
Linear Discriminant Analysis
• Linear Discriminant Analysis or Normal Discriminant Analysis or Discriminant Function
Analysis is a dimensionality reduction technique that is commonly used for supervised
classification problems. It is used for modelling differences in groups i.e. separating two or
more classes. It is used to project the features in higher dimension space into a lower
dimension space.
• For example, we have two classes and we need to separate them efficiently. Classes can have
multiple features. Using only a single feature to classify them may result in some overlapping
as shown in the below figure. So, we will keep on increasing the number of features for
proper classification.
By Prof. Merlin Jacob
ML
Example:
• Suppose we have two sets of data points belonging to two
different classes that we want to classify.
• As shown in the given 2D graph, when the data points are
plotted on the 2D plane, there’s no straight line that can
separate the two classes of the data points completely.
• Hence, in this case, LDA (Linear Discriminant Analysis)
is used which reduces the 2D graph into a 1D graph in
order to maximize the separability between the two
classes.
By Prof. Merlin Jacob
ML
• Here, Linear Discriminant Analysis uses both the axes (X and Y) to create a new axis and
projects data onto a new axis in a way to maximize the separation of the two categories and
hence, reducing the 2D graph into a 1D graph.
Two criteria are used by LDA to create a new axis:
• Maximize the distance between means of the two classes.
• Minimize the variation within each class.
By Prof. Merlin Jacob
ML
• In the graph, it can be seen that a new axis (in red) is
generated and plotted in the 2D graph such that it
maximizes the distance between the means of the two
classes and minimizes the variation within each class.
• In simple terms, this newly generated axis increases the
separation between the data points of the two classes.
By Prof. Merlin Jacob
ML
• After generating this new axis using the above-mentioned criteria, all the data points of
the classes are plotted on this new axis and are shown in the figure given below.
• But Linear Discriminant Analysis fails when the mean of the distributions are shared, as it
becomes impossible for LDA to find a new axis that makes both the classes linearly
separable. In such cases, we use non-linear discriminant analysis.
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
Singular Value Decomposition (SVD)
By Prof. Merlin Jacob
ML
SVD
• The Singular Value Decomposition (SVD) of a matrix is a factorization of that matrix into
three matrices. It has some interesting algebraic properties and conveys important geometrical
and theoretical insights about linear transformations. It also has some important applications
in data science.
By Prof. Merlin Jacob
ML
Mathematics behind SVD
• The SVD of mxn matrix A is given by the formula :
• A = UW𝑉 𝑇
where:
• U: mxn matrix of the orthonormal eigenvectors of A 𝐴𝑇 .
• 𝑉 𝑇 : transpose of a nxn matrix containing the orthonormal eigenvectors of 𝐴𝑇 A.
• W: a nxn diagonal matrix of the singular values which are the square roots of the eigenvalues
of 𝐴𝑇 A
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
By Prof. Merlin Jacob
ML
SVD
• Examples
• [Link]
By Prof. Merlin Jacob
ML
THANK YOU
By Prof. Merlin Jacob