Module -5

Covariance Matrix

To understand discriminant analysis, it is first necessary to introduce the concept

of covariance between two or more variables. The covariance measures the
relationship between two variables x and z. Denote the mean for each variable
The covariance Sx,y between x and z is given by :

where n is the number of records

A Covariance Matrix is a type of matrix used to describe the covariance values

between two items in a random vector. It is also known as the variance-
covariance matrix because the variance of each element is represented along the
matrix‟s major diagonal and the covariance is represented among the non-diagonal
elements.
It‟s particularly important in fields like data science, machine learning, and
finance, where understanding relationships between multiple variables is crucial
and comes in handy when it comes to stochastic modeling and principal
component analysis.
It helps us to identify the direction of the relationship(positive or negative)
between variables. Covariance matrix are also essential to understand the high-
dimensional datasets.
The covariance matrix Σ for x and z consists of the individual variable variances,
S2x and S2y on the diagonal (where row and column are the same variable) and the
covariances between variable pairs on the off-diagonals:
Fisher's linear discriminant
Fisher's linear discriminant is a statistical method used in machine learning
for dimensionality reduction and classification by finding a linear combination of
features that maximizes the separation between two or more classes. It achieves
this by maximizing the ratio of between-class variance to within-class variance.
This approach is also known as Linear Discriminant Analysis (LDA).

Fisher‟s linear discriminant attempts to find the vector that maximizes the
separation between classes of the projected data. Maximizing “ separation” can be
ambiguous. The criteria that Fisher‟s linear discriminant follows to do this is to
maximize the distance of the projected means and to minimize the projected
within-class variance
Here are two bivariate Gaussians with identical covariance matrices and distinct
means. We want to find the vector that best separates the projections of the data.
Let's draw a random vector and plot the projections.

Remember we are looking at projections of the data onto the vector (the dot
product of the weights vector and the data matrix) and not a decision boundary.
The projections of the data onto this random weights vector can be plotted as a
histogram (image on the right). As you can see when projecting the data onto the
vector and drawing the histogram the two classes of data aren‟t well separated. The
goal is to find the line that best separates the two distributions on the image on the
right.
To separate the two distributions, we could first try to maximize the distance
between the projected means, meaning the distributions are, on average, as far as
possible from each other. Let‟s draw a line between the two means and plot the
histogram of the projections onto that line.
That‟s quite a bit better, but the projections of the data are not fully separated yet.
To fully separate them, Fisher‟s linear discriminant minimizes the within-class
variance of the projections at the same time as maximizing the projections between
the means. It tries to maximize the means as we discussed before to separate them,
but also attempts to make the distributions as tight as possible. This allows for
better separation as you‟ll see below.

As you can see the projections of the data are well separated. We can take an
orthogonal vector from the weights vector to create a decision boundary. The
decision boundary tells us that on either side of the boundary the data can be
predicted to be one class or another. For multivariate gaussian distributions with
identical covariance matrices, this yields an optimal classifier.
generalized linear models
Generalized Linear Model (GLiM, or GLM) is an advanced statistical modelling
technique formulated by John Nelder and Robert Wedderburn in 1972. It is an
umbrella term that encompasses many other models, which allows the response
variable y to have an error distribution other than a normal distribution. The
models include Linear Regression, Logistic Regression, and Poisson Regression.
In a Linear Regression Model, the response (aka dependent/target) variable „y‟ is
expressed as a linear function/linear combination of all the predictors „X‟ (aka
independent/regression/explanatory/observed variables). The underlying
relationship between the response and the predictors is linear (i.e. we can simply
visualize the relationship in the form of a straight line). Also, the error distribution
of the response variable should be normally distributed. Therefore we are building
a linear model.
GLM models allow us to build a linear relationship between the response and
predictors, even though their underlying relationship is not linear. This is made
possible by using a link function, which links the response variable to a linear
model. Unlike Linear Regression models, the error distribution of the response
variable need not be normally distributed. The errors in the response variable are
assumed to follow an exponential family of distribution (i.e. normal, binomial,
Poisson, or gamma distributions). Since we are trying to generalize a linear
regression model that can also be applied in these cases, the name Generalized
Linear Models.
Why GLM?
Linear Regression model is not suitable if,
 The relationship between X and y is not linear. There exists some non-linear
relationship between them. For example, y increases exponentially as X
increases.
 Variance of errors in y (commonly called as Homoscedasticity in Linear
Regression), is not constant, and varies with X.
 Response variable is not continuous, but discrete/categorical. Linear
Regression assumes normal distribution of the response variable, which can
 only be applied on a continuous data. If we try to build a linear regression
model on a discrete/binary y variable, then the linear regression model
predicts negative values for the corresponding response variable, which is
inappropriate.
Assumptions of GLM
Similar to Linear Regression Model, there are some basic assumptions for
Generalized Linear Models as well. Most of the assumptions are similar to Linear
Regression models, while some of the assumptions of Linear Regression are
modified.
 Data should be independent and random (Each Random variable has the
same probability distribution).
 The response variable y does not need to be normally distributed, but the
distribution is from an exponential family (e.g. binomial, Poisson,
multinomial, normal)
 The original response variable need not have a linear relationship with the
independent variables, but the transformed response variable (through the
link function) is linearly dependent on the independent variables
Ex., Logistic Regression Equation, Log odds = β0+β1X1+β2X2 ,
where β0,β1,β2 are regression coefficient, and X1,X2 are the independent variables
 Feature engineering on the Independent variable can be applied i.e instead of
taking the original raw independent variables, variable transformation can be
done, and the transformed independent variables, such as taking a log
transformation, squaring the variables, reciprocal of the variables, can also
be used to build the GLM model.
 Homoscedasticity (i.e constant variance) need not be satisfied. Response
variable Error variance can increase, or decrease with the independent
variables.
 Errors are independent but need not be normally distributed
Components of GLM
There are 3 components in GLM.
 Systematic Component/Linear Predictor:
It is just the linear combination of the Predictors and the regression coefficients.
β0+β1X1+β2X2
 Link Function:
Represented as η or g(μ), it specifies the link between a random and systematic
components. It indicates how the expected/predicted value of the response relates
to the linear combination of predictor variables.
 Random Component/Probability Distribution:
It refers to the probability distribution, from the family of distributions, of the
response variable.
The family of distributions, called an exponential family, includes normal
distribution, binomial distribution, or poisson distribution.
Below summarizes the table of Probability Distribution, and their corresponding

Probability Distribution Link Function

Normal Distribution Identity function

Binomial Distribution Logit/Sigmoid function

Poisson Distribution Log function (aka log-linear, log-link)

Link function

Different Generalized Linear Models

Commonly used models in the GLiM family include:
  Linear Regression, for continuous outcomes with normal distribution:
Here we model the mean expected value of a continuous response variable as a
function of the explanatory variables. Identity link function is used, which is the
simplest link function.
If there is only 1 predictor, then the model is called Simple Linear Regression. If
there are 2 or more explanatory variables, then the model is called Multiple Linear
Regression.
Simple Linear Regression, y= β0+β1X1
Multiple Linear Regression, y = β0+β1X1+β2X2
Response is continuous
Predictors can be continuous or categorical, and can also be transformed.
Errors are distributed normally and variance is constant.
 Binary Logistic Regression, for dichotomous or binary outcomes with
binomial distribution:
Here Log odds is expressed as a linear combination of the explanatory variables.
Logit is the link function. The Logistic or Sigmoid function, returns probability as
the output, which varies between 0 and 1.
Log odds= β0+β1X1+β2X2
Response variable has only 2 outcomes
Predictors can be continuous or categorical, and can also be transformed.
Image source: [Link]
 Poisson Regression, for count based outcomes with poisson distribution:
Here count values are expressed as a linear combination of the explanatory
[Link] link is the link function.
log(λ)=β0+β1x1+β2x2,
where λ is the average value of the count variable
Response variable is a count value per unit of time and space
Predictors can be continuous or categorical, and can also be transformed.

Interpreting the Coefficients and Odds Ratios

Below Figure shows the relationship between the odds ratio and the log-odds ratio
for odds ratios greater than 1. Because the coefficients are on the log scale, an
increase of 1 in the coefficient results in an increase of exp 1 ≈ 2.72 in the odds
ratio.
 Figure . The relationship between the odds ratio and the log-odds ratio
Odds ratios for numeric variables X can be interpreted similarly: they measure the
change in the odds ratio for a unit change in X. For example, the effect of
increasing the payment-to-income ratio from, say, 5 to 6 increases the odds of the
loan defaulting by a factor of exp 0.08244 ≈ 1.09. The variable borrower_score is a
score on the borrowers‟ creditworthiness and ranges from 0 (low) to 1 (high). The
odds of the best borrowers relative to the worst borrowers defaulting on their loans
is smaller by a factor of exp − 4.61264 ≈ 0.01. In other words, the default risk from
the borrowers with the poorest creditworthiness is 100 times greater than that of
the best borrowers!

Feature Linear Regression Logistic Regression

Regression (predicting a
Purpose Classification (predicting a category)
continuous value)

Dependent Continuous (e.g.,

Categorical (e.g., yes/no, 0/1)
Variable temperature, salary)

A continuous value, not

Output A probability between 0 and 1
bounded

A straight line,
A sigmoid (S-shaped) curve, representing
Model representing a linear
a nonlinear relationship
relationship

Equation

Estimation
Least squares estimation Maximum likelihood estimation
Method

Predicting house prices,

Predicting if an email is spam, or if a
Use Case stock prices, or
patient has a disease
temperature
Strategies for Imbalanced Data.
Data imbalance techniques include resampling (oversampling the minority class
with SMOTE or random duplication, and undersampling the majority class by
removing samples), cost-sensitive learning (assigning different misclassification
costs to classes), and ensemble methods (combining multiple models trained on
balanced subsets). Other approaches include using appropriate evaluation metrics
like F1-score instead of accuracy and developing algorithm-level methods.

Undersampling
Undersampling is a technique used to handle imbalanced datasets. When one
class (usually the majority class) has far more samples than the other, a model may
become biased and mostly predict the majority class.
To fix this imbalance, undersampling reduces the size of the majority class by
removing some of its samples.

Why undersampling is used?

 To balance the dataset.
 To prevent the model from learning a biased pattern.
 To reduce training time.

How undersampling works?

You have a dataset like:
 Majority class (0) → 10,000 samples
 Minority class (1) → 1,000 samples
Undersampling selects a smaller number of majority samples:
 Keep 1,000 samples of class 0
 Keep all 1,000 samples of class 1
Now your data becomes balanced.

Types of Undersampling
[Link] Undersampling
Remove samples from the majority class randomly.
2. Cluster Centroid Undersampling
Uses clustering (e.g., K-means) to keep only representative samples.
3. Near-Miss Undersampling
Select majority samples that are closest to minority class samples.
Advantages
 Simple and fast.
 Reduces dataset size → faster training.
 Helps balance the dataset.

Disadvantages
 Loss of information (because majority samples are removed).
 Might reduce model performance if important data points are deleted.

Oversampling
Oversampling means increasing the number of samples in the minority
class so that the dataset becomes balanced.
 Why do it?
Because the model should not ignore the minority class (e.g., fraud cases,
disease cases).

Types of Oversampling
(a) Random Oversampling
Simply duplicate minority class samples.
Example:
 Class 0: 9000
 Class 1: 1000
→ Duplicate class 1 samples until it becomes 9000.
(b) SMOTE (Synthetic Minority Oversampling Technique)
Creates new synthetic samples instead of duplicates.
 Up-Weighting and Down-Weighting
Instead of changing the number of samples, weighting changes the
importance of each class during model training.
Up-weighting
Give more importance to minority class samples.
Example:
 Fraud case (1): weight = 5
 Normal case (0): weight = 1
The model will treat each fraud sample as if it appeared 5 times.

Down-weighting
Give less importance to majority class samples.
Example:
 Fraud case (1): weight = 1
 Normal case (0): weight = 0.2
The model pays far less attention to the majority class
Oversampling vs Up/Down Weighting (Simple Comparison)

What it
Method Risk Use case
changes

When model
Number of Overfitting needs more
Oversampling
samples (duplication) minority
examples

Better than
Synthetic
SMOTE Adds noise random
samples
oversampling

Removes
Loss of Very large
Undersampling majority
information datasets
samples

Up-weighting Importance Harder to Works well

 What it
Method Risk Use case
changes

of samples tune with logistic

regression,
SVM

Reduces Use when

Down-
majority None majority class
weighting
influence dominates