Module – 5
Covariance Matrix
Covariance measures how two variables change together.
• If both increase or decrease together → positive covariance
• If one increases while the other decreases → negative covariance
• If they move independently → covariance close to zero
Understanding covariance
• It uses the deviation of each point from its mean
• Multiplies deviations of both variables
• Averages them to get a measure of joint variation
Variance is a special case: Var(X) = Cov(X, X).
This forms the diagonal of the covariance matrix.
Covariance Matrix
A covariance matrix summarises how all variables in a dataset relate to each other.
• It is a square matrix
• Diagonal contains variance of each variable
• Off-diagonal contains covariance between pairs
• Symmetric matrix → Cov(X, Y) = Cov(Y, X)
Why it matters
• Helps identify relationships between features before classification
• Helps detect multicollinearity
• Used heavily in Fisher’s Linear Discriminant
• Helps understand how data spreads in multiple directions
Real-life example
A bank studies three variables: income, loan amount, and number of missed EMIs.
• Income & Loan Amount → positive covariance
• Income & Missed EMIs → negative covariance
• Loan Amount & Missed EMIs → may have mixed patterns
Understanding the covariance matrix helps in deciding which features help in separating
groups.
Fisher’s Linear Discriminant
Fisher’s Linear Discriminant (FLD) is a method used for classification.
It reduces multi-dimensional data to one dimension, while keeping class separation high.
Goal of FLD
• Maximise separation between class means
• Minimise spread (variance) within each class
• Find a direction (w) that gives the best class distinction
Two important scatter measures
1. Between-class scatter (SB)
Represents how far apart the class means are
2. Within-class scatter (SW)
Represents how spread the samples are inside each class
Fisher’s Criterion
Maximise the ratio:
Between-class variance / Within-class variance
This gives the most discriminative direction.
How FLD works
1. Compute mean of each class
2. Compute scatter (covariance) within each class
3. Add to form SW (within-class scatter matrix)
4. Compute mean difference (m1 – m2)
5. The discrimination vector w = SW⁻¹ (m1 – m2)
6. Project all points onto w
7. Choose a threshold to separate classes
Strengths
• Simple and effective
• Works well when covariance structures of classes are similar
• Reduces dimensionality while keeping crucial information
Limitations
• Assumes linear separability
• Doesn’t work well if classes overlap heavily
• Sensitive to differences in variances
Real-life examples
• Distinguishing genuine vs fraudulent transactions
• Classifying healthy vs unhealthy patients
• Separating high-performing vs low-performing students
Generalized Linear Models (GLM)
GLMs are extensions of linear regression for outcomes that are not continuous.
They allow:
• Binary outcomes
• Count outcomes
• Probabilities
• Non-normal distributions
Components of a GLM
1. Random Component
Distribution of the dependent variable
Examples:
• Normal → continuous
• Binomial → binary
• Poisson → count data
2. Systematic Component
Linear predictor:
β₀ + β₁X₁ + β₂X₂ + …
3. Link Function
Connects predicted mean to the linear predictor
Examples:
• Logit (for binary)
• Log (for count)
• Identity (for normal)
Why link functions are needed
• Probabilities must be between 0 and 1
• Counts cannot be negative
• Logit ensures the mapping is correct
Examples of GLMs
● Logistic regression
Used for binary predictions like disease/no disease
● Poisson regression
Used for count predictions like number of customers per hour
Textbook-style example
Hospital predicting presence of a disease using glucose, BMI, age:
logit(p) = β₀ + β₁ × glucose + β₂ × BMI + β₃ × age
Probability is obtained by converting logit to p.
Why GLMs are useful
• More flexible than linear regression
• Handle different types of data
• Used widely in medical, financial, and marketing applications
Interpreting Coefficients & Odds Ratios
In logistic regression, the model predicts log-odds.
Coefficients describe how log-odds change when a predictor increases by one unit.
Log-odds → odds → probability
Coefficient meaning
• Positive coefficient → higher chance of the event
• Negative coefficient → lower chance
• Zero → no effect
Odds Ratio
Odds Ratio = e^(coefficient)
Interpretation becomes easier.
Examples
Coefficient = 0.3
Odds ratio = e⁰·³ ≈ 1.35
Meaning → 1 unit increase makes event 35% more likely.
Coefficient = -0.6
Odds ratio ≈ 0.55
Meaning → event becomes 45% less likely.
Numerical example (more detailed)
logit(p) = -3 + 0.04 × glucose
If glucose = 120:
logit(p) = -3 + 4.8 = 1.8
Probability = 1 / (1 + e⁻¹·⁸) ≈ 0.86
So chance of disease ≈ 86%.
Why interpretation matters
• Helps explain how each predictor affects the outcome
• Important for healthcare, insurance, banking
• Makes model results transparent
Strategies for Imbalanced Data
Imbalanced data → one class has much more data than the other.
Example:
• 99% non-fraud and 1% fraud
• 90% healthy and 10% diseased
Problems caused
• Model may always predict majority class
• High accuracy but poor minority detection
• Important cases may be missed
Why accuracy is misleading
Example: 95 normal + 5 fraud
Predict all normal → accuracy = 95%
But fraud detection = 0%
So recall, precision, F1-score must be used.
Techniques to handle imbalance
1. Undersampling
Reduces majority class
Simple but may lose information
2. Oversampling
Duplicates minority samples
Good for small datasets
3. SMOTE
Creates synthetic samples
More varied than simple oversampling
4. Class Weighting
Assigns heavier penalty to minority misclassification
Supported by logistic regression and GLMs
Better evaluation metrics
• Precision
• Recall
• F1-score
• ROC-AUC
• PR-AUC
Real-life examples
● Medical diagnosis
Missing a positive case is dangerous → recall is important
● Fraud detection
Minority class is extremely rare
● Customer churn
Need the model to catch “leaving” customers
Final note
Models trained on imbalanced data need both resampling and correct evaluation metrics to
perform reliably.
Quick Summary of Module – 5
Topic Key Idea
Covariance Matrix Shows relationships among variables
Fisher’s LD Finds best direction to separate classes
GLM Regression for binary / count / non-normal data
Coefficients & Odds Interpret log-odds and odds ratios
Imbalanced Data Handle uneven class distribution