0% found this document useful (0 votes)

4 views44 pages

ML Module 2

The document covers concepts related to bivariate and multivariate data, including statistics such as covariance and correlation, and techniques for feature engineering and dimensionality reduction. It discusses the importance of probability and statistics in machine learning, detailing various probability distributions and density estimation methods. Additionally, it outlines essential mathematical concepts for multivariate data, including linear systems, matrix decompositions, and feature selection strategies.

Uploaded by

utkarshojha578

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

4 views44 pages

ML Module 2

Uploaded by

utkarshojha578

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

Module-2

Understanding Data – 2:
Bivariate Data and Multivariate Data
Multivariate Statistics
Essential Mathematics for Multivariate Data
Feature Engineering and Dimensionality Reduction Techniques.

Basic Learning Theory:

Design of Learning System
Introduction to Concept of Learning
Modelling in Machine Learning.

Chapter-2 (2.6-2.8, 2.10), Chapter-3 (3.3, 3.4, 3.6)

Temperature in a Shop and Sales Data
Line Chart

Bivariate Data

Scatter Plot
Bivariate Data - Statistics
Covariance and Correlation are examples of bivariate statistics.
Covariance : is an indicator of the extent to which 2 random variables are dependent on each other.
• Covariance implies whether the two variables are directly or inversely proportional.
• A higher number denotes higher dependency.
Correlation : is a statistical measure that indicates how strongly two variables are related.
• The value of covariance lies in the range of -∞ and +∞.

• Covariance(X, Y) or COV(X, Y) and is used to measure the variance between two dimensions.
• The formula for finding co-variance for specific x, and y are:
• Here,
𝑥𝑖 and 𝑦𝑖 are data values from X and Y.
E(X) and E(Y) are the mean values of 𝑥𝑖 and 𝑦𝑖.
The COV(X, Y) is same as COV(Y, X).
N is the number of given data.
Bivariate Data - Statistics
Example 2.6: Find the covariance of data X = {1, 2, 3, 4, 5} and Y = {1, 4, 9, 16, 25}.

Solution: Mean(X) = E(X) = 15/5 = 3,

Mean(Y) = E(Y) = 55/5 = 11.
The covariance is computed using COV(X, Y) as:

The covariance between X and Y is 12.

It can be normalized to a value between -1 and +1.
This is done by dividing it by the correlation of variables.
Bivariate Data - Statistics
Correlation
1. If the value is positive, it indicates that the dimensions increase together.
2. If the value is negative, it indicates that while one-dimension increases, the other dimension
decreases.
3. If the value is zero, then it indicates that both the dimensions are independent of each other.
4. If the dimensions are correlated, then it is better to remove one dimension as it is a redundant
dimension.
5. If the given attributes are X = (x1, x2, …, xN) and Y = (y1, y2, …, yN), then the Pearson correlation
coefficient, that is denoted as r, is given as:
Bivariate Data - Statistics
Example 2.7: Find the correlation coefficient of data X = {1, 2, 3, 4, 5} and Y = {1, 4, 9, 16, 25}.

Solution:
The mean values of X and Y are 15/5 = 3 and 55/5 = 11.
The standard deviations of X and Y are 1.41 and 8.6486, respectively.
Therefore, the correlation coefficient is given as ratio of covariance (12 from the
previous problem 2.6) standard deviation of x and y as per the above equation as-
Multivariate Statistics
• Some of the multivariate analysis are regression analysis, principal component
analysis, and path analysis.

• The mean of multivariate data is a mean vector and the mean of the shown three
attributes is given as (2, 7.5, 1.33).
Multivariate Statistics
Heatmap
• Heatmap is a graphical representation of 2D matrix.
• It takes a matrix as input and colours it. The darker colours indicate very large
values and lighter colours indicate smaller values.
• In Figure 2.25, patient data highlighting weight and health status is plotted. Here,
X-axis is weights and Y-axis is patient counts. The dark colour regions highlight
patients’ weights vs patient counts in health status.
Multivariate Statistics

Heatmap for
Patient Data
Multivariate Statistics

Pair plot for

Random Data
Essential Mathematics for Multivariate Data
1. Linear Systems and Gaussian Elimination for Multivariate Data:
• A linear system of equations is a group of equations with unknown variables.
• Let Ax = y, then the solution x is given as:

• This is true if y is not zero and A is not zero. The logic can be extended for N-
set of equations with ‘n’ unknown variables.
Essential Mathematics for Multivariate Data
• If there is a unique solution,
then the system is called
consistent independent.
• If there are various solutions,
then the system is called
consistent dependent.
• If there are no solutions and if
the equations are
contradictory, then the system
is called inconsistent.
• For solving large number of
system of equations, Gaussian
elimination can be used. The
procedure for applying
Gaussian elimination is given
as follows:
Essential Mathematics for Multivariate Data
• To facilitate the application of
Gaussian elimination method,
the following row operations are
applied:
1. Swapping the rows
2. Multiplying or dividing a
row by a constant
3. Replacing a row by adding
or subtracting a multiple of
another row to it These
concepts are illustrated in
Example 2.8.
Essential Mathematics for Multivariate Data
Example 2: Solve the following set of equations
using Gaussian Elimination method.
Essential Mathematics for Multivariate Data
2. Matrix Decompositions:
• It is often necessary to reduce a matrix to its constituent parts so that complex matrix operations can
be performed. These methods are also known as matrix factorization methods.
1. LU decomposition
2. Eigan decomposition

1. LU decomposition
One of the simplest matrix decompositions is LU decomposition where the matrix A can be
decomposed matrices:

A = LU
Here, L is the lower triangular matrix and U is the upper triangular matrix. The decomposition can be
done using Gaussian elimination method.
Essential Mathematics for Multivariate Data
Example 2.9: Find LU decomposition of the given matrix:

Solution: First, augment an identity matrix and apply Gaussian elimination.

The steps are as shown in:

Now, it can be observed that the first matrix is L as it is the lower

triangular matrix whose values are the determiners used in the
reduction of equations above such as 3, 3 and 2/3. The second
matrix is U, the upper triangular matrix whose values are the
values of the reduced matrix because of Gaussian elimination.
Essential Mathematics for Multivariate Data
2. Eigan decomposition
The most popular matrix decomposition is called eigen decomposition. It is a way of reducing the
matrix into eigen values and eigen vectors. Then, the matrix A can be decomposed as:
𝐴 = 𝑄𝛬𝑄 𝑇
where, Q is the matrix of eigen vectors, Λ is the diagonal matrix and 𝑄 𝑇 is the transpose of matrix Q.
Essential Mathematics for Multivariate Data
3. Machine Learning and Importance of Probability and Statistics:

• Machine learning is linked with statistics and probability.

• Like linear algebra, statistics is the heart of machine learning.
• The importance of statistics needs to be stressed as without statistics; analysis of
data is difficult.
• Probability is especially important for machine learning.
• Any data can be assumed to be generated by a probability distribution.
Essential Mathematics for Multivariate Data
Probability Distributions
• A probability distribution of a variable, say X, summarizes the probability associated with X’s events.
• Distribution is a function that describes the relationship between the observations in a sample space.
• Probability distributions are of two types:
1. Continuous probability distribution
2. Discrete probability distribution

PDF - The relationships between the events for a continuous random variable and their probabilities is called a
continuous probability distribution. It is summarized as Probability Density Function (PDF). The plot of PDF
shows the shape of the distribution.

CDF - Cumulative Distributive Function (CDF) computes the probability of an observation ≤ value. Both PDF
and CDF are continuous values.

PMF - The discrete equivalent of PDF in discrete distribution is called Probability Mass Function (PMF).
Essential Mathematics for Multivariate Data
Continuous probability distribution

Probability Expression Description Applications

Distribution Type
Normal PDF of Normal Distribution Here, μ is mean and σ is the The heights of the
distribution/ standard deviation. Normal students, blood
Gaussian distribution is characterized pressure of a
by two parameters – mean population, and
distribution/
and variance. marks scored in a
Bell shaped class can be
distribution approximated using
Z-Score When μ is zero and σ is 1, z- normal distribution.
score is same as x . This is
useful to normalize
the data.
Essential Mathematics for Multivariate Data
Continuous probability distribution

Probability Expression Description Applications

Distribution Type
Rectangular The uniform distribution It has equal probabilities for Random number
Distribution / all values in the range a, b generation, lottery
Uniform draws
distribution
Essential Mathematics for Multivariate Data
Continuous probability distribution

Probability Expression Description Applications

Distribution Type
Exponential PDF of Exponential Distribution This probability distribution is Time between
distribution used to describe the time arrivals in a queue,
between events in a Poisson failure rates of
process machines

x- random variable
 - rate parameter
 - the mean and standard
deviation of exponential
distribution
=1/
Essential Mathematics for Multivariate Data
Discrete probability distribution

Binomial Distribution
• Binomial distribution is another distribution that is often encountered in machine learning. It has only two
outcomes: success or failure. This is also called Bernoulli trial.
• The objective of this distribution is to find probability of getting success ‘k’ out of ‘n’ trials. The way to get success
out of ‘k’ out of ‘n’ number of trials is given as:

The binomial distribution function is given as follows, where ‘p’ is the probability of success and probability of
failure is (1 - p). The probability of success in a certain number of trials is given as:

Combining both, one gets PDF of binomial distribution as:

Here, ‘p’ is the probability of each choice, ‘k’ is the number of choices, and ‘n’ is the total number of choices. The
mean of binomial distribution is given below:

And the variance is given as:

Hence, the standard deviation is given as:
Essential Mathematics for Multivariate Data
Discrete probability distribution

Poisson Distribution
Some of the examples of Poisson distribution are number of emails received, number of customers
visiting a shop and the number of phone calls received by the office

The PDF of Poisson distribution is given as follows:

Given an interval of time, this distribution is used to model the probability of a given number of events
‘k’. The mean rule ‘λ’ is inclusive of previous events.
The standard deviation is (λ).
Essential Mathematics for Multivariate Data
Discrete probability distribution
Essential Mathematics for Multivariate Data
Density Estimation
Density Estimation is a technique used to estimate the probability distribution of a dataset. It helps
in understanding the underlying structure of the data, detecting patterns, and identifying anomalies.
Types –
1. Parametric Density Estimation – Maximum Likelihood Estimation
2. Non-parametric Density Estimation – Parzen Window and k-Nearest Neighbor (KNN)

Example:
The California Housing Dataset contains information about housing prices and associated features
like population, median income, and location. Density estimation on this dataset helps analyze how
housing prices are distributed across different regions.

1. Parametric Density Estimation

• It assumes that the data is from a known probabilistic distribution and can be estimated as p(x | Θ),
where, Θ is the parameter. x = {x1, x2.. , xn}
• Maximum likelihood function is a parametric estimation method.
Essential Mathematics for Multivariate Data
Density Estimation

Feature Parametric Density Estimation Non-Parametric Density Estimation

Assumes that data follows a known probability Estimates the probability density function (PDF)
Definition distribution (e.g., normal, exponential) and directly from data without assuming an
estimates parameters (e.g., mean, variance). underlying distribution.
Less flexible; restricted to the chosen Highly flexible; adapts to any shape of the data
Flexibility
parametric family. distribution.
Assumes a specific form of the distribution (e.g.,
Assumptions No assumption about data distribution.
normal, Poisson).
Data Can work well with smaller datasets if the Requires large sample sizes for accurate
Requirements assumption about distribution is correct. density estimation.
Kernel Density Estimation (KDE), Parzen
Gaussian (Normal) Distribution, Exponential
Examples Window, k-Nearest Neighbors (k-NN) Density
Distribution, Binomial Distribution.
Estimation.
Used when the data is believed to follow a Used when the underlying distribution is
Usage
known distribution. unknown or complex.
Essential Mathematics for Multivariate Data
Maximum Likelihood Estimation

• For a sample of observations, one can estimate the probability distribution. This is called density

estimation. Maximum Likelihood Estimation (MLE) is a probabilistic framework that can be used

for density estimation.

• This involves formulating a function called likelihood function which is the conditional probability

of observing the observed samples and distribution function with its parameters.

• For example, consider a joint probability p(X; θ), where, X = {x1, x2, …, xn}

• The likelihood of observing the data is given as a function L(X; θ).

• The objective of MLE is to maximize this function as max L(X; θ).

Essential Mathematics for Multivariate Data
Maximum Likelihood Estimation

• The joint probability of this problem can be restated as:

• The computation of the above formula is unstable and the hence the problem is restated as

maximum of log conditional probability given θ. This is given as:

• Instead of maximizing, one can minimize this function as:

This is called negative log-likelihood function

Essential Mathematics for Multivariate Data
Maximum Likelihood Estimation

• The relevance of this theory of MLE for machine learning is that, MLE can solve the problem of

predictive modeling in machine earning.

• If MLE is used with Regression problem, the MLE framework is

• If MLE is used with Gaussian Distribution, the MLE applied is

Essential Mathematics for Multivariate Data
Density Estimation

2. Non-Parametric Density Estimation

Essential Mathematics for Multivariate Data
Density Estimation

2. Non-Parametric Density Estimation

Essential Mathematics for Multivariate Data
Feature Engineering and Dimensionality Reduction Techniques
Feature engineering is the process of selecting and transforming relevant attributes (features) to
improve machine learning model performance.

It addresses two key problems:

1. Feature Transformation: Involves extracting and creating new meaningful features.

Example: Combining height and weight to create Body Mass Index (BMI) for health prediction.

2. Feature Selection: Identifies the most important features while removing irrelevant/redundant
ones to reduce computational complexity.
Challenge: A dataset with ‘n’ features has ‘2n’ possible subsets, making feature selection difficult.
Essential Mathematics for Multivariate Data
Feature Engineering and Dimensionality Reduction Techniques
Feature Selection Strategies
1. Feature Relevancy: Determines how useful a feature is for classification.
Example: A mole on a face is more relevant for face recognition than common features like a nose.
Methods: Mutual Information, Correlation Coefficient, Distance Measures.

2. Feature Redundancy
Removes duplicate or unnecessary features that do not provide new information.
Example: If a dataset has Date of Birth, then Age can be removed as it is derived from the Date of
Birth.

Feature Selection Techniques

1. Filter Methods (Fast & Model-Independent): Uses statistical measures to rank feature importance.
Example: Mutual Information, Correlation, Entropy. No machine learning model is required.

2. Wrapper Methods (Slow but More Accurate): Uses machine learning classifiers to evaluate
feature importance. Trains a model on different feature subsets and selects the best one. More
computationally expensive but often performs better.
Essential Mathematics for Multivariate Data
Feature Engineering and Dimensionality Reduction Techniques
Algorithms for feature selection –
1. Stepwise forward selection
2. Stepwise backward elimination
3. Principal Component Analysis
4. Linear Discriminant Analysis
5. Singular Value Decomposition
Essential Mathematics for Multivariate Data
Feature Engineering and Dimensionality Reduction Techniques
1. Stepwise forward selection
This procedure starts with an empty set of attributes. Every time, an attribute is tested for statistical significance
for best quality and is added to the reduced set. This process is continued till a good reduced set of attributes is
obtained.

2. Stepwise backward elimination

This procedure starts with a complete set of attributes. At every stage, the procedure removes the worst attribute
from the set, leading to the reduced set.

3. Principle Component Analysis

• The idea of the principal component analysis (PCA) or KL transform is to transform a given set of
measurements to a new set of features so that the features exhibit high information packing properties.
• This leads to a reduced and compact set of features.
• Consider a group of random vectors of the form: