by Kunal Sir

What is Machine Learning.

Machine Learning is the science of making computers learn patterns from past data so
they can predict or decide on new, unseen data.

Real-Life Examples

• Netflix / YouTube: Recommends videos you might like.

• Spam filter in Gmail: Learns which emails are spam.
• Voice assistants (Siri, Alexa): Understand your speech.
• Google Maps: Predicts traffic and best routes.
• Medical diagnosis: Detects cancer from X-ray images.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

How Machine Learning Works (Simple Flow)

Step Name What Happens Example (House Price Prediction)

1 Collect Data Gather raw historical Collect data of 1000 houses: size,
data location, rooms, age, price
2 Clean & Fix errors, handle missing Remove houses with negative price,
Preprocess Data values, remove fill missing rooms with average
duplicates, handle
outliers
3 Feature Scaling & Scale features and select Apply Standardization, remove
Selection only useful features (Very useless columns like "House ID"
Important)
4 Split Data Divide data into Training 80% for training, 20% for testing
set and Testing set
5 Train Model Algorithm learns patterns Train Decision Tree / Linear
from training data Regression on training data
6 Test & Evaluate Check how well the Calculate Accuracy, Precision,
model performs on Recall, Confusion Matrix
unseen test data
7 Improve (if Tune hyperparameters, Try Random Forest if Decision Tree
needed) try different algorithms, is overfitting
add more data
8 Predict / Deploy Use the final trained Predict price of a new house that
model on new real-world comes in the market
data

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Step 1: Collect Data

Sample Raw Dataset (Collected from sources like Google Forms, CSV, or
database)

Area Bedrooms Age Location Price

1200 2 5 Pune 5000000
1500 3 10 Mumbai 8000000
1800 3 8 Pune 7500000
? 2 7 Delhi 6000000
200 4 15 Mumbai ?
1200 2 5 Pune 5000000

Problems:

• Missing values
• Duplicate data

Step 2: Clean & Preprocess Data

After Cleaning

Area Bedrooms Age Location Price

1200 2 5 Pune 5000000
1500 3 10 Mumbai 8000000
1800 3 8 Pune 7500000
1500 2 7 Delhi 6000000
2000 4 15 Mumbai 8200000

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

What we did:

• Filled missing Area → 1500 (average)

• Filled missing Price → 8200000
• Removed duplicate row

This step is crucial in Machine Learning

Step 3: Feature Scaling & Selection

Convert Categorical → Numeric

Location Encoded
Pune 0
Mumbai 1
Delhi 2

Final Dataset

Area Bedrooms Age Location Price

1200 2 5 0 5000000
1500 3 10 1 8000000
1800 3 8 0 7500000
1500 2 7 2 6000000
2000 4 15 1 8200000

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Step 4: Split Data (Train + Test)

Training Data (80%)

Area Bedrooms Age Location Price

1200 2 5 0 5000000
1500 3 10 1 8000000
1800 3 8 0 7500000
1500 2 7 2 6000000

Testing Data (20%)

Area Bedrooms Age Location Price

2000 4 15 1 8200000

Step 5: Train Model

• Apply algorithm (e.g., Decision Tree)
• Model learns pattern:

Example:

• More area → higher price

• Mumbai → expensive

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Step 6: Test & Evaluate

Prediction:

Input:

• Area = 2000
• Bedrooms = 4
• Location = Mumbai

Output:

• Predicted Price ≈ 8100000

Compare:

• Actual = 8200000
• Error = small → Good model

Step 7: Improve Model

If accuracy is low:

• Add more data

• Add features (bathrooms, parking)
• Try better algorithms

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Step 8: Predict on New Data

New Input:

Area Bedrooms Age Location

1600 3 6 Pune

Model Output:

• Predicted Price ≈ ₹70,00,000

Main Types of Machine Learning

Type What it does Example

Learns from labeled data

Supervised Predict house price, classify email as spam/ham
(input + correct output)

Finds hidden patterns in

Unsupervised Customer segmentation, anomaly detection
unlabeled data

Learns by trial & error

Reinforcement Game-playing AI (AlphaGo), self-driving cars
with rewards

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Why is Machine Learning Important?

• Handles huge amounts of data

• Improves over time with more data
• Automates tasks that were previously done by humans
• Powers most modern AI applications

Feature Selection

What is a Feature in Machine Learning?

In Machine Learning, a feature is an input variable (column) used to make predictions.

Simple Meaning:

Feature = Input data that helps the model learn

Example: House Price Prediction

Area Bedrooms Location Price

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Features (X):
o Area
o Bedrooms
o Location
• Target (Y):
o Price

So, features are the independent variables

Types of Features

1. Numerical Features
a. Example: Age, Salary, Area
2. Categorical Features
a. Example: City, Gender, Color
3. Derived Features
a. Created from existing data
b. Example: Price per [Link]

What is Feature Selection?

Feature Selection is the process of choosing only important features from the dataset
and removing unnecessary ones.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Definition:
Selecting the most relevant input variables to improve model performance

Example

Original Data:

Area Bedrooms Owner Name Phone Number Price

After Feature Selection:

| Area | Bedrooms | Price |

We have Removed:

• Owner Name
• Phone Number

Because they don’t affect price prediction

Why is Feature Selection Important?

1. Improves Accuracy

• Removes noise (irrelevant data)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

2. Reduces Overfitting

• Model focuses only on useful data

3. Faster Training

• Less data → faster computation

4. Better Understanding

• Easier to interpret model

Methods of Feature Selection

• Here is your clean, formal, exam-ready version (no symbols, numbered points)

1. What is Feature Selection?

Feature Selection is the process of selecting the most relevant input features and removing
irrelevant or redundant ones to improve model performance.

2. Objectives

1. Improve accuracy
2. Reduce overfitting
3. Decrease training time
4. Improve interpretability

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

3. Types of Feature Selection Methods

1. Filter Methods
2. Wrapper Methods
3. Embedded Methods

1. Filter Methods
• Filter methods are based on statistical techniques and do not require a machine learning model.
Each feature is evaluated independently.

1A] Correlation (Numerical Data)

Definition

Correlation measures the linear relationship between two numerical variables.

• Formula

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Meaning of Symbols

1. r

Correlation coefficient (final result)

2. x

Individual value of variable X (feature)

3. y

Individual value of variable Y (target)

4. x̄ (x-bar)

Mean (average) of all X values

x̄ = (Σx) / n

5. ȳ (y-bar)

Mean (average) of all Y values

ȳ = (Σy) / n

6. (x − x̄)

Deviation of X from its mean

7. (y − ȳ)

Deviation of Y from its mean

8. Σ (Sigma)

Summation (add all values)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Step-by-Step Example
Given Data

X (Experience) Y (Salary)
1 10
2 20
3 30

Step 1: Calculate Mean

1. x̄ = (1 + 2 + 3) / 3 = 2
2. ȳ = (10 + 20 + 30) / 3 = 20

Step 2: Find Deviations

X Y (x−x̄) (y−ȳ) (x−x̄)(y−ȳ) (x−x̄)² (y−ȳ)²

1 10 -1 -10 10 1 100
2 20 0 0 0 0 0
3 30 1 10 10 1 100

Step 3: Calculate Summation

1. Σ(x−x̄)(y−ȳ) = 10 + 0 + 10 = 20
2. Σ(x−x̄)² = 1 + 0 + 1 = 2
3. Σ(y−ȳ)² = 100 + 0 + 100 = 200

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Step 4: Apply Formula

r = 20 / √(2 × 200)

r = 20 / √400

r = 20 / 20

r=1

4. Final Answer

1. r = 1
2. This means perfect positive correlation

5. Interpretation

1. r = +1 → perfect positive relationship

2. r = -1 → perfect negative relationship
3. r = 0 → no relationship

Simple Understanding
1. If X increases and Y increases → positive correlation
2. If X increases and Y decreases → negative correlation
3. If no pattern → zero correlation

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

NOTE:

1. Correlation measures linear relationship

2. Value always lies between -1 and +1
3. Uses mean and deviations
4. Numerator shows relationship
5. Denominator normalizes the value

1 B] Chi-Square Test (Categorical Data)

1. Definition

1. Chi-Square Test is a statistical method used to check whether there is a relationship

(dependency) between a categorical feature and the target variable.

2. Purpose

1. To identify important categorical features

2. To check whether variables are independent or dependent

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

3. Formula

4. Meaning of Terms

• O (Observed Value)
• Actual value from dataset
• E (Expected Value)
• Value expected if there is no relationship
• Σ (Sigma)
• Sum of all values

5. Step-by-Step Example

Problem
1. Check whether Gender affects product purchase

• Step 1: Observed Table

Gender Buy Not Buy Total

Male 40 60 100
Female 60 40 100
Total 100 100 200

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Step 2: Calculate Expected Values

• Formula:
• E = (Row Total × Column Total) / Grand Total

Expected Table

Gender Buy Not Buy

Male 50 50
Female 50 50

Step 3: Apply Formula

Male Buy

• (40 − 50)² / 50 = 100 / 50 = 2

Male Not Buy

• (60 − 50)² / 50 = 2

Female Buy

• (60 − 50)² / 50 = 2

Female Not Buy

• (40 − 50)² / 50 = 2

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Step 4: Final Chi-Square Value

• χ² = 2 + 2 + 2 + 2 = 8

• High χ² value → strong relationship → important feature

• Low χ² value → weak relationship → remove feature

2. Wrapper Methods
1. Wrapper methods use a machine learning model to evaluate different combinations of features.

2.1 Forward Selection

• Start with no features
• Add one feature at a time
• Select feature that improves model performance
• Repeat until no improvement

3.2 Backward Elimination

• Start with all features
• Remove least important feature
• Retrain model
• Repeat

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

3. Embedded Methods
Embedded methods perform feature selection inside the model training process. The model itself
decides which features are important while learning.

1. Lasso Regression (L1 Regularization)

Definition

Lasso (Least Absolute Shrinkage and Selection Operator) adds an L1 penalty (sum of absolute values of
coefficients) to the ordinary linear regression cost function.

This penalty shrinks some coefficients exactly to zero, automatically removing those features.

Cost Function

• First part → usual least-squares error

• Second part → L1 penalty (controlled by λ)

How Feature Selection Happens

• λ = 0 → same as normal Linear Regression (no selection).

• As λ ↑ → unimportant coefficients shrink and become exactly 0 → those features are dropped.

Real-Life Example: House Price Prediction

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Dataset has 5 features:

• Area, Location, Age, OwnerName, HouseID (target = Price)

After training Lasso (λ = 0.5):

Feature Coefficient (β) Selected?

Area 0.82 Yes

Location 0.65 Yes

Age 0.31 Yes

OwnerName 0.00 No (dropped)

HouseID 0.00 No (dropped)

Result: Lasso automatically removes OwnerName and HouseID. The final model uses only 3 features →
simpler and less overfitting.

Advantages

• Automatic feature selection (coefficients become zero).

• Produces sparse (simple) models.

Disadvantages

• Unstable when features are highly correlated.

2. Decision Trees (Gini Importance / Tree-based Feature Importance)

Definition

Decision Trees calculate feature importance automatically during tree construction.

Importance is measured by how much a feature reduces impurity (Gini or Entropy) across all splits.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

How It Works

• At every node, the tree picks the feature that gives the largest impurity reduction.
• Feature importance score = total weighted impurity reduction caused by that feature.

Formula (Mean Decrease in Impurity)

Feature Importance Bar Chart (Example)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Real-Life Example: Laptop Purchase Prediction

Target: Will customer buy laptop? (Yes/No)

Features: Age, Income, Gender, Education, City.

After building the Decision Tree:

Feature Gini Importance Selected?

Age 0.62 Yes (highest)

Income 0.28 Yes

Education 0.08 Yes

Gender 0.02 Maybe drop

City 0.00 Drop

Result: The tree automatically gives very low importance to “City”. You can safely remove it without
losing much accuracy.

Advantages

• Captures non-linear relationships.

• No extra code needed — importance is built-in.

Disadvantages

• Can be biased toward features with more categories.

Revision Table

Method Penalty / Score Feature Selection Style Example Use Case

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Linear models with many

Lasso L1 (absolute value) Coefficients → exactly 0
features

Decision Gini / Entropy Built-in importance Any tree-based model (non-

Tree reduction ranking linear)

What is Feature Scaling?

In Machine Learning, feature scaling is the process of bringing all input features to a
similar range or scale.

Definition:

Feature Scaling is used to standardize the range of independent variables so that no

feature dominates others.

Example (Without Scaling)

Feature Value
Age 25
Salary 50,000

Problem:

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Salary is much larger than Age

• Model gives more importance to Salary

Why is Feature Scaling Needed?

1. Avoid Bias

• Large values dominate small ones

2. Faster Training

• Algorithms converge faster

3. Better Accuracy

• Especially for distance-based models

Types of Feature Scaling

1. Normalization (Min-Max Scaling)

Scales values between 0 and 1

Xchanged = X – Xmin / X_max - Xmin

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Example:

• Age: 20 → 0.2
• Age: 50 → 0.8

2. Standardization (Z-Score Scaling)

Centers data around mean = 0

Xchanged = X - μ / σ

Where:

• μ = mean
• σ = standard deviation

Before vs After Scaling

Before:

• Age: 20–60
• Salary: 10,000–1,00,000

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

After:

• Both scaled to similar range (e.g., 0–1)

1. Feature Scaling and Selection – Why Important?

Machine learning models are sensitive to the scale of features.

A feature like “salary” (in thousands) can dominate “age” (0–100) in distance-based
algorithms (KNN, SVM) or gradient descent (Neural Networks).

Normalization (Min-Max Scaling) Scales data to range [0, 1].

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Standardization (Z-score Scaling) means 0 and standard deviation = 1.

Example: House price dataset

• Area: 500 to 5000 [Link].

• Rooms: 1 to 10

After Normalization:

Area becomes 0 to 1,

Rooms become 0 to 1 → both treated equally.

After Standardization:

Both features centered around 0

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

What is Feature Extraction?

Feature Extraction is the process of creating a new set of features by transforming the original
high-dimensional data into a lower-dimensional space while keeping the most important
information.

Simple Meaning:

Instead of using 100 original features, we create 5–10 new combined features that capture almost
the same information.

Key Difference from Feature Selection

• Feature Selection → Picks existing features (Filter/Wrapper/Embedded).

• Feature Extraction → Creates brand new features (e.g., PCA combines many features into
one new component).

Why Feature Extraction is Needed?

• Solves Curse of Dimensionality (too many features → model becomes slow, overfits, and
performance drops).

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Main Feature Extraction Techniques

Uses
Technique Type Goal Best Used When
Labels?

PCA Unsupervised No Maximize variance No class labels, general data

LDA Supervised Yes Maximize class separation Classification with labels

1. Principal Component Analysis (PCA)

Definition

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

PCA is an unsupervised technique that finds new axes (principal components) along which the data
has maximum variance.

[Variance measures how much data values spread out from the mean (average).]

PCA Steps

1. Standardize the data (mean = 0, variance = 1).

2. Compute Covariance Matrix.
3. Calculate Eigenvectors and Eigenvalues.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

4. Sort by eigenvalues (highest variance first).

5. Select top k eigenvectors → Project data.

Mathematical Formula

where ( W ) = matrix of top ( k ) eigenvectors.

Explained Variance (Scree Plot)

Use this to decide how many components to keep.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Example: House price dataset with 50 features (area, location, age, etc.) → PCA creates 3 new
components that explain 95% variance.

PCA Example – Complete Step-by-Step (with Full Std Dev Calculation)

Unit III / Unit V (TLO 3.3 & 5.4)

Dataset

Student Maths Science

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

1 80 85

2 90 88

3 60 65

4 70 72

5 85 90

Goal: Reduce 2 features → 1 Principal Component.

Step 1: Standardize the Data (Z-Score Scaling)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Standardized Data Table (Z-scores)

Student Maths_std = Maths –77/ 12.04 Science_std = Science –80 / 10.93

1 0.249 0.457

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

2 1.079 0.731

3 -1.413 -1.372

4 -0.581 -0.731

5 0.665 0.915

Step 2: Covariance Matrix

Formula (after standardization)

Standardized Data (used for calculation)

Student Maths_std Science_std

1 0.249 0.457

2 1.080 0.732

3 -1.412 -1.372

4 -0.581 -0.732

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

5 0.664 0.915

Calculations

Covariance between Maths_std and Science_std

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Final Covariance Matrix:

2. Linear Discriminant Analysis (LDA)

Definition

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

LDA is a supervised technique that finds new axes that maximize separation between classes
while minimizing variation inside each class.

Key Idea

• Maximizes between-class scatter.

• Minimizes within-class scatter.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

When to use LDA

• You have labeled data (classification problems).

• Goal is to separate classes clearly (e.g., Spam vs Non-Spam, Disease vs No-Disease).

EXAMPLE:

LDA – Complete Final Real-World Example

Predicting Student Pass/Fail (Unit III / Unit V – TLO 3.3 & 5.4)

Dataset

Student Exam 1 (Maths) Exam 2 (Science) Actual Class

1 65 70 Fail (0)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

2 72 68 Fail (0)

3 60 62 Fail (0)

4 85 88 Pass (1)

5 90 92 Pass (1)

6 78 82 Pass (1)

Goal: Find 1 new axis (LD1) that best separates Pass and Fail students.

Step 1: Class Means & Overall Mean

• Class 0 (Fail) mean: [65.67, 66.67]

• Class 1 (Pass) mean: [84.33, 87.33]
• Overall mean: [75.00, 77.00]

Step 2: Within-Class Scatter Matrix (SW)

SW measures spread inside each class.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

outer product is dot product of deviation and

transpose of deviation

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Step 3: Between-Class Scatter Matrix (SB)

SB measures spread between the two class means.

Step 2: Outer Product for Class 0

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

SB shows how far apart the two class means are. High values indicate good separation between Fail and
Pass groups.

Step 4: Eigenvalues & Eigenvectors

Solve the generalised eigenvalue problem: SW⁻¹ SB w = λ w

• Eigenvalues: 8.046 (LD1), 0 (LD2)

• LD1 eigenvector (best direction after normalisation): [0.257, -0.966]

LD1 captures 100% of the class separation.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Step 5: LD1 Scores for Each Student

Projection Formula

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Final LD1 Scores

Student LD1 Score Actual Class Interpretation

1 -50.91 Fail (0) Higher LD1 → Fail group

2 -47.18 Fail (0) Higher LD1 → Fail group

3 -44.47 Fail (0) Higher LD1 → Fail group

4 -63.16 Pass (1) Lower LD1 → Pass group

5 -65.73 Pass (1) Lower LD1 → Pass group

6 -59.16 Pass (1) Lower LD1 → Pass group

What does LD1 Score Decide?

• Higher (less negative) LD1 score → Student belongs to Fail group

• Lower (more negative) LD1 score → Student belongs to Pass group

Simple Decision Rule (threshold):

• If LD1 score > -47 → Predict Fail

• If LD1 score ≤ -47 → Predict Pass

Result: Reduced 2 exam scores → 1 new score (LD1) with perfect separation between Pass and
Fail.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Quick Revision Table (Exam-Ready)

Aspect PCA LDA

Type Unsupervised Supervised

Uses Labels No Yes

Objective Max variance Max class separation

Number of components ≤ number of original features ≤ (number of classes – 1)

Best for General dimensionality reduction Classification tasks

Pro Tip

• No labels → Use PCA.

• Have labels + classification → Use LDA (usually gives better results than PCA).

What is Mutual Information based Feature Selection?

Mutual Information (MI) is a Filter Method of feature selection that measures how much
information one feature contains about the target variable (or another feature).

Simple Meaning:

It tells us how dependent a feature is on the target.

Higher MI score = feature is more useful for prediction.

Unlike correlation (which only captures linear relationships), MI captures both linear and non-
linear relationships.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Why Use Mutual Information?

• Works with both numerical and categorical data.

• Model-independent (fast filter method).
• Detects non-linear dependencies that correlation misses.
• Helps remove redundant or irrelevant features.

Mathematical Formula (Mutual Information)

Where:

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• p(x,y) = joint probability of X and Y

• p(x) ) and ( p(y) = marginal probabilities

Venn Diagram Explanation (Entropy + Mutual Information)

• Red circle = Entropy of X

• Blue circle = Entropy of Y
• Purple overlap = Mutual Information (shared information)

How Mutual Information Feature Selection Works (Step-by-Step)

1. Calculate MI score between each feature and the target variable.

2. Rank features by MI score (highest first).
3. Select top k features (or those above a threshold).

Example: House Price Prediction

Dataset has features: Area, Location, Age, Owner Name, House ID.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

MI Scores (imaginary values):

• Area → 0.85 (very high)

• Location → 0.72
• Age → 0.45
• Owner Name → 0.05 (very low)
• House ID → 0.00 (zero)

Selected features: Area, Location, Age

Dropped: Owner Name, House ID

Real Plot Example (Wine Quality Dataset – MI scores)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Where does MI fit in Feature Selection Methods?

MI is listed under Filter Methods (along with Correlation, Chi-square, etc.).

Advantages & Disadvantages

Advantages Disadvantages

Captures non-linear relationships Computationally expensive for large datasets

Works with mixed data types Does not consider feature interactions directly

Fast (Filter method) Requires discretization for continuous data sometimes

No model training needed —

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Pro Tip for Exam

• MI is better than correlation when relationship is non-linear.

• Used in scikit-learn as mutual_info_regression or mutual_info_classif.

What is ANOVA in the Context of Feature Selection?

ANOVA (Analysis of Variance) is a statistical method used to compare the means of a numerical
(continuous) feature across two or more groups defined by a categorical target variable (classes in
classification problems).

In feature selection, ANOVA acts as a univariate filter method. It ranks numerical features based
on how well they discriminate between different classes of the target. Features whose means differ
significantly across classes are considered more important.

It is also known as the ANOVA F-test or f_classif (in scikit-learn terminology).

Core Idea Behind ANOVA for Feature Selection

The goal is to test whether the observed differences in feature values across classes are due to real
group differences (signal) or just random variation (noise).

• If a feature has very different means across classes → it helps separate the classes → high
importance.
• If the means are similar across classes → the feature does not help much in distinguishing
classes → low importance.

ANOVA achieves this by partitioning the total variance of the feature into two parts:

1. Between-group variance (explained variance): Variation due to differences between class

means.
2. Within-group variance (unexplained variance): Variation within each class (random noise).

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

The F-Statistic – The Heart of ANOVA

The key output of ANOVA is the F-statistic (or F-score):

Where:

• MSB (Mean Square Between) = Variance explained by class differences

• MSW (Mean Square Within) = Average variance inside each class

Interpretation:

• High F-value → Between-group variance >> Within-group variance → Strong evidence that
class means differ → Feature is highly relevant.
• F-value close to 1 → Between-group variance ≈ Within-group variance → No significant
difference across classes → Feature is less useful or irrelevant.

Along with the F-statistic, ANOVA also provides a p-value. A very small p-value (typically < 0.05)
indicates that the observed difference is statistically significant (unlikely due to chance).

In feature selection, we usually rank features by descending F-score (or ascending p-value) and
select the top-k features.

Mathematical Intuition (One-Way ANOVA)

For a numerical feature X and categorical target with k classes:

• Let μ₁, μ₂, ..., μₖ be the means of feature X in each class.

• Let μ be the overall mean of X.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Between-group sum of squares (SSB) measures how far class means are from the grand mean.

Within-group sum of squares (SSW) measures spread around each class mean.

The F-statistic follows an F-distribution under the null hypothesis (all class means are equal).

When is ANOVA Appropriate for Feature Selection?

• Input features: Continuous / Numerical

• Target variable: Categorical (binary or multi-class classification)
• Best suited when you want a fast, model-agnostic ranking of features before model
training.

For regression problems (continuous target), the equivalent is f_regression (which tests linear
correlation).

Key Assumptions of ANOVA

ANOVA is a parametric test, so it relies on these assumptions:

1. Normality: The numerical feature should be approximately normally distributed within

each class.
2. Homogeneity of variances (Homoscedasticity): The variance of the feature should be
roughly equal across all classes.
3. Independence: Observations should be independent of each other.
4. Linearity: It primarily captures linear relationships between the feature and the target
(through mean differences).

Violations (especially of equal variance or normality in small samples) can make the F-test
unreliable.

Advantages of ANOVA in Feature Selection

• Computationally very efficient — scales well even to thousands of features.

• Interpretable: High F-score directly means good class separation.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• No need to train a full model (pure filter method).

• Works for both binary and multi-class problems.

Limitations and Drawbacks

• Assumption sensitivity: Performs poorly if data is heavily skewed, has many outliers, or
violates normality/homogeneity.
• Only linear relationships: Misses complex non-linear dependencies between feature and
target.
• Univariate: Ignores interactions between features (does not consider redundancy or
synergy among multiple features).
• Sensitive to outliers, which can inflate variances.
• In very high-dimensional settings, multiple testing issues can arise (though rarely corrected
in basic feature selection).

Because of these limitations, ANOVA is often used as a quick first step and is sometimes
combined with other methods (e.g., mutual information for non-linear cases or wrapper methods
for better performance).

Comparison with Related Concepts

• Similar to Fisher’s discriminant ratio: The F-statistic is closely related to the ratio of
between-class to within-class variance used in linear discriminant analysis.
• Vs. Mutual Information: ANOVA captures linear mean differences; mutual information
captures any type of dependency (including non-linear).
• Vs. t-test: ANOVA generalizes the two-sample t-test to more than two groups.

----------------------------------------------------------------------------------------------------------------------------

Recursive Feature Elimination with Cross-Validation (RFECV) is a powerful wrapper-based

feature selection technique that combines Recursive Feature Elimination (RFE) with cross-
validation to automatically select the optimal number of features while reducing the risk of
overfitting.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

What is Recursive Feature Elimination (RFE)?

RFE is a backward elimination (greedy) algorithm:

1. Start with all features.

2. Train a chosen machine learning model (the estimator) on the current set of features.
3. Rank the features based on their importance as determined by the model (e.g., coefficients
in linear models, feature_importances_ in tree-based models).
4. Remove the least important feature(s) (one or more, controlled by the step parameter).
5. Repeat the process with the remaining features until the desired number of features is
reached.

RFE is model-dependent — the quality of selection depends heavily on the estimator used (e.g.,
Logistic Regression, SVM, Random Forest, etc.).

Why Add Cross-Validation? → RFECV

Plain RFE requires you to manually specify how many features to keep (n_features_to_select).
Choosing this number is tricky and can lead to overfitting or underfitting.

RFECV solves this by integrating k-fold cross-validation to automatically determine the best
number of features.

How RFECV Works (Step-by-Step Theory)

1. Outer Process: RFECV considers different possible subset sizes (from all features down to
min_features_to_select, usually 1).
2. For each possible number of features (say, from n down to 1):
a. Perform k-fold cross-validation.
b. In each fold:
i. On the training portion of the fold: Run the full RFE process (train estimator →
rank features → recursively eliminate least important ones) until reaching the
current target subset size.
ii. Evaluate the model's performance (using the chosen scoring metric, e.g.,
accuracy, F1, ROC-AUC) on the validation portion of the fold.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

c. Average the performance scores across all k folds for that subset size.
3. Select the optimal number: Choose the number of features that gives the highest average
cross-validated score.
4. Final Model: Once the optimal number is found, RFECV fits a final RFE on the entire
dataset using that optimal number and produces:
a. support_: Boolean mask of selected features.
b. ranking_: Ranking of all features (1 = selected, higher = eliminated earlier).
c. cv_results_: Detailed scores for each subset size.

This makes RFECV more robust and less prone to selecting a suboptimal number of features
compared to plain RFE.

Key Parameters (Theoretical View)

• estimator: Any model that exposes feature importance (has coef_ or

feature_importances_ attribute).
• step: Number (or fraction) of features to remove per iteration (default=1).
• cv: Number of folds or a cross-validator object (default=5).
• scoring: Metric to optimize during CV (e.g., 'accuracy', 'f1', 'neg_mean_squared_error').
• min_features_to_select: Smallest subset size to consider (default=1).

Advantages of RFECV

• Automatically finds the optimal number of features without manual trial-and-error.

• Uses cross-validation → better estimate of generalization performance → reduces overfitting
in feature selection itself.
• Provides feature ranking (not just selection).
• Works with any estimator that supports feature importance.
• Improves model interpretability and reduces dimensionality effectively.

Limitations and Drawbacks

• Computationally expensive: It runs RFE multiple times (once per fold per subset size). For
large datasets or many features, it can be very slow.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Still greedy and univariate in spirit: Although it uses a model, it eliminates features one-
by-one without considering complex interactions in every step.
• Depends heavily on the estimator: A weak or biased estimator can lead to poor feature
ranking.
• May produce different results across different CV folds or random states because of
variability in feature importance.
• Not ideal for extremely high-dimensional data (thousands of features) without sufficient
computational resources.

RFECV vs Plain RFE vs Other Methods

Aspect RFE RFECV Filter Methods (e.g., ANOVA)

Type Wrapper Wrapper + CV Filter (model-agnostic)

Manual Automatically
Optimal # features Manual or separate tuning
specification selected

Risk of overfitting Higher Lower (due to CV) Generally lower

Computational cost Medium High Low

Captures
Partially (via model) Partially (via model) No (univariate)
interactions

RFECV is generally more reliable than plain RFE but slower than univariate filter methods like
ANOVA F-test or Mutual Information.

When to Use RFECV

• You have a moderate number of features (tens to a few hundred).

• You want the feature selection to be tuned to your specific model and performance metric.
• You care about generalization and want to avoid manually choosing the number of features.
• You are using tree-based models, linear models, or SVMs that provide reliable importance
scores.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

RFECV is a classic wrapper method that balances performance and robustness through cross-
validation, making it one of the most widely used techniques for automatic feature selection in
practice.

-----------------------------------------------------------------------------------------------------------------------------

Feature Importance from Tree-Based Models (Beyond

Single Decision Trees)
Core Concept

Tree-based models (ensemble methods) provide a built-in mechanism to measure how much
each feature contributes to the model's predictions. This is one of the most popular and effective
ways to perform embedded feature selection.

Unlike filter methods (e.g., ANOVA), tree-based importance is model-dependent — it reflects how
useful the features were during actual model training.

1. How Feature Importance is Calculated in Tree-Based Models

The most common method is Mean Decrease in Impurity (MDI), also called Gini Importance or
Information Gain Importance.

Mechanism:

• Every time a feature is used to split a node in a tree, the impurity (Gini, Entropy, or variance
for regression) decreases.
• The weighted decrease in impurity is attributed to that feature.
• This decrease is averaged across all splits where the feature was used, and across all trees
in the ensemble.
• Features that cause larger, more frequent impurity reductions get higher importance
scores.

The importance is usually normalized so that all feature importances sum to 1 (or 100%).

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Formula (simplified for one tree):

In ensembles (Random Forest, XGBoost, etc.), this is averaged over all trees.

2. Popular Tree-Based Models & Their Feature Importance

Here’s a clear comparison:

Feature Importance
Model Key Characteristics Best For Limitations
Method

Averaged across Biased toward

Random Mean Decrease Impurity Classification
many trees, more high-cardinality
Forest (MDI) & Regression
stable features

High-
Even more
Extra Trees MDI (similar to RF) dimensional Similar bias
randomization, faster
data

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Gradient
Based on split gain High Can be unstable
Boosting MDI (gain)
during boosting performance in small data
(GBM)

Multiple types
'gain', 'weight', 'cover', Speed + Default 'gain' is
XGBoost available
'total_gain' Performance most common
(importance_type)

'split' (number of times Very fast, leaf-wise Large-scale 'split' can favor
LightGBM
used), 'gain' growth data high-cardinality

Datasets
Built-in importance Handles categorical Slightly different
CatBoost with
(PredictionValuesChange) features natively interpretation
categoricals

Most Recommended in Practice:

• Random Forest → Most stable and reliable for feature importance.

• XGBoost / LightGBM → Best when you need both high performance and importance.

3. Types of Feature Importance in Boosting Models (XGBoost/LightGBM)

1. Gain (most commonly used): Total improvement in loss/impurity from splits using this
feature.
2. Weight / Split: Number of times the feature was used to split.
3. Cover: Number of samples affected by splits on this feature.
4. Total Gain: Sum of gain across all trees.

Gain is generally preferred because it measures actual contribution to model quality, not just
frequency of use.

4. Advantages of Tree-Based Feature Importance

• Captures non-linear relationships and interactions (unlike ANOVA or correlation).

• Embedded method: Importance comes naturally during model training.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Fast after model is trained.

• Handles mixed data types (numerical + categorical in some models like CatBoost).
• Provides a global view of feature relevance.
• Works for both classification and regression.

5. Major Limitations & Biases

• Bias towards high-cardinality features: Features with many unique values (e.g., IDs,
continuous features with high precision) tend to get inflated importance.
• Not reliable for correlated features: When two features are highly correlated, importance
can be arbitrarily split between them.
• Can be misleading on small datasets or with noisy data.
• Does not consider feature removal effect directly (unlike RFE).
• Permutation importance is often recommended as a more robust alternative (see below).

-----------------------------------------------------------------------------------------------------------------------------
XGBoost Feature Importance
XGBoost (eXtreme Gradient Boosting) is one of the most powerful tree-based ensemble models. It
builds trees sequentially (boosting), where each new tree corrects the errors of the previous ones.
This sequential nature, combined with regularization and optimized splitting, makes its feature
importance scores particularly insightful.

How Feature Importance Works in XGBoost

XGBoost calculates feature importance by analyzing how much each feature contributes during
the tree-building process across all boosting rounds (trees).

The core mechanism is similar to other tree-based models (impurity reduction or split gain), but
XGBoost provides multiple distinct ways to quantify importance. These are accessible via
get_score(importance_type=...) in the Booster API or feature_importances_ in the scikit-
learn wrapper (default is usually 'gain').

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

5 Types of Feature Importance in XGBoost

Here is a clear theoretical breakdown:

1. Gain (Most Recommended & Default in many cases)

a. Measures the average improvement in the loss function (or reduction in
impurity/error) brought by splits using that feature.
b. It quantifies the actual contribution of the feature to making the model more
accurate.
c. Higher gain = the feature, when used for splitting, leads to significantly better
separations and lower overall loss.
d. Best for interpretation: It reflects real predictive power rather than just frequency
of use.
e. Interpretation: Features with high gain are the ones that most strongly drive the
model's decisions.
2. Weight (also called Frequency or Split count)
a. Simply counts how many times the feature was used to make a split across all
trees.
b. It is a frequency-based measure.
c. Limitation: Biased toward features with high cardinality (many unique values) or
features that appear in many shallow splits. It does not consider the quality/impact
of those splits.
3. Cover
a. Measures the average number of training samples (observations) affected by the
splits that use this feature.
b. It reflects how broadly the feature influences the data points across the ensemble.
c. Useful for understanding the "reach" or coverage of a feature in the dataset.
4. Total_Gain
a. The sum (not average) of the gain from all splits that used the feature across all
trees.
b. Emphasizes features that contribute a lot in absolute terms, even if used fewer
times.
5. Total_Cover
a. The sum of the coverage from all relevant splits.

Key Insight:

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Gain / Total_Gain → Focus on quality of contribution (how much the model improves).
• Weight → Focus on frequency of usage.
• Cover → Focus on breadth of influence on samples.

In practice, Gain is widely preferred because it better captures the true discriminative power of a
feature, especially in classification and regression tasks.

Advantages of XGBoost Feature Importance

• Captures non-linear relationships and feature interactions naturally (thanks to tree

structure).
• Computationally efficient — available immediately after training.
• Multiple perspectives (gain vs weight vs cover) allow richer analysis.
• Works well for both classification and regression.
• Scalable to high-dimensional data.

Limitations & Known Biases

• Bias toward high-cardinality features: Features with many distinct values (e.g.,
continuous variables or high-cardinality categoricals) tend to get higher weight/cover scores
because they allow more possible split points.
• Correlated features: Importance can be arbitrarily distributed among highly correlated
features.
• Depends on hyperparameters: Changing max_depth, min_child_weight, number of
trees, or subsample can alter importance rankings.
• Not fully model-agnostic: It explains the trained XGBoost model, not necessarily the
underlying data relationships.
• Can be unstable on small or noisy datasets.

Because of these issues, many practitioners cross-validate importance rankings or combine with
permutation importance or SHAP values for more robust interpretation.

XGBoost vs Random Forest Feature Importance

• Similar Foundation: Both use impurity reduction / split gain averaged across trees.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Key Difference:
o Random Forest: Bagging (parallel trees, averaging predictions) → more stable
importance.
o XGBoost: Boosting (sequential trees, focusing on residuals) + regularization →
importance can be more sensitive to the error-correcting process.
• XGBoost often gives sharper distinction on truly important features due to its focus on hard
examples.
• Random Forest importance tends to be smoother and less prone to extreme shifts.

When to Use XGBoost for Feature Selection / Importance

• As an embedded method: Train XGBoost → extract importance → select top features →

retrain (or use in pipeline).
• Often combined with RFECV (using XGBClassifier/XGBRegressor as the estimator).
• Good for initial ranking before applying wrapper methods.
• Excellent when your dataset has mixed numerical/categorical features (especially with
native categorical support in recent versions).

Summary (Key Takeaways)

• XGBoost provides rich, multi-dimensional feature importance: Gain (quality), Weight

(frequency), Cover (breadth), and their total variants.
• Gain is generally the most reliable single metric for understanding which features truly drive
performance.
• Always interpret importance relative to your specific model and dataset.
• For critical applications, do not rely solely on built-in importance — consider permutation
importance or SHAP for confirmation.
• XGBoost importance is powerful because it reflects the actual splitting decisions made
during gradient boosting, making it more aligned with the model's learning process than
simple filter methods like ANOVA.

This fits perfectly in the sequence you asked:

• ANOVA → univariate statistical filter (linear mean differences)

• RFECV → wrapper with CV for optimal subset

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Tree-based importance (Random Forest → XGBoost) → embedded, non-linear, interaction-

aware

-----------------------------------------------------------------------------------------------------------------------------
LightGBM Feature Importance
LightGBM (Light Gradient Boosting Machine) is a highly efficient gradient boosting framework
developed by Microsoft. It is designed for speed and scalability on large datasets, using leaf-wise
tree growth (instead of level-wise like XGBoost), histogram-based splitting, Gradient-based One-
Side Sampling (GOSS), and Exclusive Feature Bundling (EFB).

Despite these optimizations, its feature importance mechanism remains rooted in the same tree-
based impurity reduction principles as other boosting algorithms.

Types of Feature Importance in LightGBM

LightGBM provides two primary types of feature importance (controlled by the importance_type
parameter):

1. Split (Default)
a. Counts the number of times a feature is used to make a split across all trees in the
ensemble.
b. It is a frequency-based measure.
c. Higher value = the feature was selected more often for splitting.
d. Pros: Simple, fast to compute, gives a good sense of how "active" a feature is.
e. Cons: Biased toward features with high cardinality (many possible split points) or
features that appear in shallow splits. It does not measure the quality of the splits.
2. Gain
a. Measures the total improvement (reduction in loss or impurity) achieved by splits
that use this feature across all trees.
b. It sums up the gain (decrease in the objective function) from every split made on
that feature.
c. Higher gain = the feature contributes more to improving the model's accuracy/loss
reduction.
d. Pros: More informative and better reflects the actual predictive contribution of the
feature.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

e. Cons: Values can be very large and skewed; harder to interpret in absolute terms.

Key Recommendation:

Use Gain for most interpretability and feature selection tasks because it focuses on the quality and
impact of splits rather than mere frequency.

Split is useful for a quick overview of feature usage.

In the scikit-learn API (LGBMClassifier / LGBMRegressor), the default importance_type is

'split'.

You can change it to 'gain' when calling .feature_importances_ or plot_importance().

How LightGBM Calculates Feature Importance

• During training, every split evaluates potential features using histogram approximations for
speed.
• For each valid split, LightGBM computes the gain — the reduction in the loss function (e.g.,
log loss for classification, MSE for regression).
• Split importance simply increments a counter each time the feature is chosen.
• Gain importance accumulates the actual gain value from those splits.
• The final importance is aggregated over all boosting iterations (trees).

This process is closely tied to LightGBM’s leaf-wise growth strategy: trees grow by always splitting
the leaf with the largest loss reduction, which can lead to deeper, more complex trees and
potentially sharper importance distinctions.

LightGBM vs XGBoost Feature Importance

Aspect LightGBM XGBoost

Gain, Weight (frequency), Cover,

Main Types Split (frequency), Gain (total)
Total Gain

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Default Split Gain (in many interfaces)

Growth Strategy Leaf-wise → can favor features that allow big Level-wise → more balanced
Impact loss reductions splits

Speed &
Extremely fast even on large data Slightly slower
Scalability

Similar bias, but Cover adds

Bias High-cardinality features (split)
sample reach

Interpretation Gain preferred for quality Gain also preferred

Similarities: Both rely on split gain for the "quality" measure.

Differences: LightGBM’s leaf-wise growth and histogram approximations can sometimes produce
more aggressive splits, making Gain importance even more critical to examine (as frequency alone
may mislead).

Advantages of LightGBM Feature Importance

• Very fast to extract after training (thanks to histogram-based approach).

• Naturally captures non-linear relationships and higher-order interactions.
• Works well with large-scale data and high-dimensional datasets.
• Native support for categorical features (via optimal splitting) often leads to more
meaningful importance scores for categoricals.
• Efficient for iterative feature selection pipelines.

Limitations & Biases

• Split importance is biased toward features with more possible split points (high cardinality
or continuous features).
• Gain can produce extremely large or skewed values, especially with more trees.
• Correlated features: Importance gets split arbitrarily between them.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Sensitive to hyperparameters (e.g., num_leaves, max_depth, min_split_gain,

min_child_samples).
• Like all tree-based importances, it explains the trained model rather than pure data
relationships.
• On small or noisy datasets, rankings can be unstable.

For robust analysis, many practitioners combine LightGBM importance with permutation
importance or SHAP values.

Role in Feature Selection

LightGBM importance is an excellent embedded method:

• Train the model → extract Gain or Split scores → select top-k features or apply a threshold.
• Often used inside RFECV (with LGBMClassifier as estimator).
• Good first step before more expensive wrapper methods.
• Helps reduce dimensionality quickly on large feature sets.

--------------------------------------------------------------------------------------------------------------

Random Forest Feature Importance

Random Forest (RF) is a bagging-based ensemble of decision trees. It provides built-in feature
importance scores directly after training. This makes it one of the most popular embedded feature
selection methods.

1. Built-in Feature Importance: Gini Importance (Mean Decrease in

Impurity - MDI)

This is the default and true built-in feature importance in Random Forest.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

How It Works (Theoretical Explanation)

• Each decision tree in the forest uses Gini impurity (for classification) or variance reduction
(for regression) as the splitting criterion.
• When a feature is chosen for a split at a node, it reduces the impurity (or variance) of that
node.
• The decrease in impurity caused by the split is attributed to that feature.
• This decrease is weighted by the number of samples reaching that node.
• For the entire forest:
o The impurity decrease for a feature is averaged across all trees.
o The result is normalized so that the sum of all feature importances equals 1.

Mathematical Intuition:

In Random Forest, this is averaged over all trees → Mean Decrease in Impurity (MDI), commonly
called Gini Importance.

In scikit-learn: rf.feature_importances_ returns this Gini/MDI score.

Advantages:

• Extremely fast (computed during training, no extra cost).

• Captures non-linear relationships and interactions to some extent.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Easy to interpret globally (higher score = more important on average).

2. SHAP Values with Random Forest

Important Clarification:

• Random Forest does NOT provide SHAP values as a built-in feature.

• SHAP (SHapley Additive exPlanations) is a separate, model-agnostic interpretation
method based on cooperative game theory.
• However, SHAP works very well with Random Forest because there is an efficient
TreeExplainer that exploits the tree structure for fast exact computation (much faster than
Kernel SHAP).

What are SHAP Values?

SHAP values assign a value to each feature for every individual prediction, representing how
much that feature pushes the prediction away from the base (average) prediction.

• They satisfy local accuracy, missingness, and consistency properties (from Shapley
values).
• Global feature importance from SHAP is usually the mean absolute SHAP value across
all samples.

Key Differences from Gini Importance:

Aspect Gini Importance (MDI) SHAP Values

Post-hoc explanation
Type Built-in to Random Forest
(TreeExplainer)

Computation During training After training

Scope Global only Local (per prediction) + Global

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Strong bias toward high-cardinality

Bias Much less biased, more consistent
features

Reliability on unseen Computed on training data → can Can be computed on

data overfit validation/test data

Captures interactions Partial Better (additive per instance)

Computational cost Free Higher (but efficient for trees)

Why SHAP is Often Preferred:

• Gini/MDI importance can be misleading due to bias toward features with many unique
values (high cardinality).
• It is calculated on training data, so it may inflate importance of features that help overfit.
• SHAP provides more faithful explanations aligned with actual contribution to predictions.

Many practitioners use both:

• Gini for quick screening.

• SHAP (via [Link]) for reliable interpretation and feature selection.

===============================================================================
===============================================================================
===============================================================================

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Supervised Learning
1. What is Supervised Learning?

Supervised Learning is a type of Machine Learning where the algorithm learns from
labeled data.

In simple words:

→ We give the model both input (features) and correct output (label/target) during
training.

→ The model learns the relationship between input and output.

→ After training, it can predict the output for new, unseen data.

Key Point:

The word “Supervised” means the learning process is guided/supervised by the correct
answers (labels) provided in the training data.

2. Real-Life Examples of Supervised Learning

Application Input (Features) Output (Label) Type

House Price Prediction Area, location, no. of rooms, age Price (in ₹) Regression

Spam or Not
Email Spam Detection Email text, sender, subject Classification
Spam

Student Pass/Fail Study hours, attendance, previous

Pass or Fail Classification
Prediction marks

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Cat, Dog, Car,

Image Recognition Pixel values of image Classification
etc.

Has Disease or
Medical Diagnosis Blood pressure, sugar level, age Classification
Not

Previous prices, volume, news Tomorrow’s

Stock Price Prediction Regression
sentiment price

3. Types of Supervised Learning

There are two main types:

A. Classification

• When the output is a category or class (discrete value).

• Examples: Spam/Not Spam, Pass/Fail, Cat/Dog, Yes/No.

B. Regression

• When the output is a continuous numerical value.

• Examples: House price, temperature, salary, stock price.

4. Supervised Learning Process (Simple Flow)

1. Collect labeled data

2. Clean and preprocess data
3. Split data into Training set (80%) and Testing set (20%)
4. Train the model using training data
5. Test the model on testing data
6. Evaluate performance
7. Deploy the model for new predictions

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

1. Decision Tree

Definition:

A Decision Tree is a tree-like structure where each internal node represents a test on a
feature; each branch represents the outcome of the test, and each leaf node represents
the final prediction.

How it Works:

• It recursively splits the data into subsets based on the most informative feature.
• Splitting criteria: Gini Impurity or Information Gain (Entropy).
• Continues until pure nodes or stopping conditions are met.

Example:

Predict if a person will buy a laptop:

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Is Age < 30?

o Yes → Predict “Will Buy”
o No → Is Income > 50,000?
▪ Yes → Predict “Will Buy”
▪ No → Predict “Will Not Buy”

Advantages: Easy to understand, visual, handles mixed data types.

Disadvantages: Prone to overfitting.

2. K-Nearest Neighbors (KNN)

Definition:

KNN is a lazy, instance-based learning algorithm that classifies a new data point based on
the majority class of its K nearest neighbors.

How it Works:

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

1. Choose value of K (e.g., 3 or 5).

2. Calculate distance (Euclidean) to all training points.
3. Select K closest points.
4. Assign the class with majority vote.

Example:

New patient has symptoms (fever=1, cough=0).

Look at 5 most similar patients → 4 had “Flu”, 1 had “Cold” → Predict Flu.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Advantages: Simple, no training phase.

Disadvantages: Slow for large data, very sensitive to scaling and irrelevant features.

3. Support Vector Machine (SVM)

Definition

SVM is a supervised classification algorithm that finds the hyperplane (decision boundary) that best
separates the classes with the maximum margin.

Simple Meaning

It draws the widest possible “street” between two classes so that the boundary is as far as possible from
the nearest data points (called support vectors).

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

How SVM Works

• Maximizes the margin (gap) between the hyperplane and support vectors.
• For non-linear data → uses Kernel Trick (maps data to higher dimension).

Common Kernels

• Linear
• Polynomial
• RBF (Radial Basis Function) – most popular

Visual Example

Real-Life Example

Classify emails as Spam / Not Spam.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

SVM draws a boundary that keeps maximum gap between spam and non-spam points.

Advantages

• Works very well in high-dimensional data.

• Effective with non-linear data using kernels.
• Robust to overfitting.

Disadvantages

• Slow on very large datasets.

• Sensitive to choice of kernel and parameters.
• Does not give probability (only class).

KNN vs SVM – Quick Comparison

Aspect KNN SVM

Type Lazy / Instance-based Eager / Model-based

Training Phase None Yes (finds hyperplane)

Speed (Prediction) Slow (calculates distances) Fast after training

Best for Small datasets, simple problems High-dimensional, non-linear data

Sensitive to Scaling Very High Medium

Handles Non-linearity No (needs many neighbors) Yes (Kernel Trick)

Output Class only Class (can be extended for probability)

Pro Tip for Exam

• Choose KNN when dataset is small and interpretability is needed.

• Choose SVM when data is high-dimensional or non-linear.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

2. Regression Algorithms

A. Linear Regression

Definition: Predicts a continuous value by fitting a straight line to the data.

Equation:

Y = mx + c

Where,

• Y → Output (dependent variable)

• x → Input (feature)
• m= → Slope (y2-y1/ x2-x1)
o How much Y changes when x changes
• c = intercept

How it Works: Minimizes the sum of squared errors (best fit line).

Example: Predict house price based on area ([Link]). As area increases, price increases
linearly.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Advantages: Simple and interpretable.

Disadvantages: Assumes linear relationship.

B. Logistic Regression

Definition: Used for binary classification (output 0 or 1). It is called “regression” but used
for classification.

How it Works:

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Calculates linear combination, then applies Sigmoid function to get probability (0

to 1).

• Sigmoid:

• If p > 0.5 → Class 1, else Class 0.

Calculation:

What is z in Logistic Regression?

In Machine Learning, z is the linear combination of input features and weights.

Definition

z is the raw score calculated before applying the sigmoid function.

Meaning of Terms

Term Meaning

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

(x) Input features

(w) Weights
(b) Bias
(z) Linear output (raw score)

Role of z

1. First, compute z (linear output)

2. Then apply sigmoid function:

3. Convert z → probability (0 to 1)

📊 Example
Suppose:

• (x_1 = 2), (x_2 = 3)

• (w_1 = 0.5), (w_2 = 1)
• (b = 0.2)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

This value is then passed to sigmoid

Understanding
• z = weighted sum of inputs
• It is the input to sigmoid function
• It decides final probability

Example: Predict whether a student will Pass or Fail based on study hours.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Advantages: Gives probability output.

Disadvantages: Assumes linear decision boundary.

What is Ridge Regression?

Ridge Regression is a type of Linear Regression with L2 regularization (penalty on the square of
coefficients).

It is used to prevent overfitting and handle multicollinearity (when features are highly correlated).

NOTE:

Overfitting is a situation in machine learning where a model learns the training data too well, including
noise and unnecessary details, and performs poorly on new (unseen) data.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Multicollinearity

Multicollinearity occurs when two or more independent variables (features) in a dataset are highly
correlated with each other.

• Features give similar information

• One feature can be predicted from another

Feature 1 (Height in cm) Feature 2 (Height in feet)

170 5.57
180 5.90

Both features represent the same information → high correlation

Problem

1. Model becomes confused about which feature to use

2. Coefficients become unstable
3. Reduces model interpretability
4. Affects performance of linear models

Simple Meaning

Normal Linear Regression can make very large coefficients → unstable model.

Ridge adds a small penalty on large coefficients so the model becomes simpler and more stable.

Why Ridge Regression?

• Reduces overfitting when there are many features.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Shrinks coefficients towards zero (but never exactly to zero).

• Works well when features are correlated.

Mathematical Formulation

• First term → Ordinary Least Squares (OLS) error

• Second term → L2 Penalty (λ controls strength)
• λ (lambda) = regularization parameter (higher λ → stronger shrinkage)

Effect of λ (as λ increases, coefficients shrink)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

How Ridge Regression Works

1. Start with ordinary linear regression.

2. Add L2 penalty to the cost function.
3. Optimize (using gradient descent) → coefficients become smaller.
4. Larger λ → more shrinkage → simpler model.

Coefficient Path Plot (how coefficients change with λ)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Real-Life Example

House Price Prediction (features: Area, Location, Age, Rooms, etc.)

• Without Ridge → some coefficients become very large (overfitting).

• With Ridge (λ = 1) → all coefficients shrink slightly → model generalizes better on new houses.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Advantages & Disadvantages

Advantages Disadvantages

Does not perform feature selection (coefficients never become

Reduces overfitting
exactly zero)

Handles multicollinearity Needs tuning of λ (hyperparameter)

Stable even with many correlated

Slightly biased estimates
features

Computationally efficient —

Ridge vs Simple Linear Regression

Aspect Simple Linear Regression Ridge Regression

Regularization None L2 (λ ∑ β²)

Coefficient Size Can be very large Shrunk towards zero

Overfitting High risk Lower risk

Feature Selection No No (use Lasso for that)

Best when Few features, low correlation Many features / multicollinearity

Pro Tip for Exam

• Ridge = L2 penalty (square of coefficients).

• Use when you want to keep all features but reduce their impact.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

3. Model Performance Evaluation

Confusion Matrix (for Classification)

Key Metrics:

• Accuracy = (TP + TN) / Total

• Precision = TP / (TP + FP)
• Recall = TP / (TP + FN)

Quick Revision Table

Algorithm Type Output Key Concept Example

Decision Tree Classification Category Gini/Entropy splits Buy laptop (Yes/No)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Majority vote of K
KNN Classification Category Flower classification
neighbors

House price
Linear Regression Regression Continuous Best fit straight line
prediction

Logistic
Classification Probability Sigmoid function Pass/Fail prediction
Regression

5. Model Performance Evaluation (TLO 4.3)

What is Model Performance Evaluation?

After training a model (especially in Supervised Learning), we must check how well it is performing on
unseen test data.

This is done using a Confusion Matrix + key metrics: Accuracy, Precision, and Recall.

These metrics are mainly used for Classification problems (e.g., Spam/Not-Spam, Pass/Fail, Disease/No-
Disease).

1. Confusion Matrix

A table that shows the actual vs predicted classes.

Predicted: Positive Predicted: Negative

Actual: Positive True Positive (TP) False Negative (FN)

Actual: Negative False Positive (FP) True Negative (TN)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Real-Life Example (Email Spam Detection)

• Total emails = 10,000

• Confusion Matrix:

Predicted: Spam Predicted: Non-Spam

Actual: Spam TP = 600 FN = 300

Actual: Non-Spam FP = 100 TN = 9,000

2. Key Metrics (Formulas)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

3. Example with Numbers (Spam Detection)

Quick Revision Table

Metric Formula What it Answers When to use (Priority)

Accuracy (TP+TN)/Total Overall correctness Balanced classes

Reliability of positive When FP is costly (e.g., spam

Precision TP/(TP+FP)
predictions filter)

How many positives When FN is costly (e.g., cancer

Recall TP/(TP+FN)
were caught detection)

Pro Tip:

• Use Precision when you want to reduce false alarms.

• Use Recall when you cannot afford to miss any positive case.
• For a single score → F1-score = 2 × (Precision × Recall) / (Precision + Recall) (not in syllabus but
very common).

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

=====================================================================================
=====================================================================================
=====================================================================================

Unsupervised Learning
1. What is Unsupervised Learning?

Unsupervised Learning is a type of Machine Learning where the algorithm learns from
unlabelled data.

• There is no correct output labels provided during training.

• The model tries to find hidden patterns, structures, or groupings in the data by
itself.

Key Difference from Supervised Learning:

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Supervised Learning → Data has labels (e.g., Pass/Fail, Price, Spam/Not Spam)
• Unsupervised Learning → Data has no labels

Example of Unlabelled Data (Very Important)

Dataset: Customer information in a shopping mall (No labels given)

Customer ID Age Annual Income (₹) Spending Score (1–100)

1 25 45,000 65

2 32 78,000 82

3 45 32,000 25

4 28 65,000 78

5 50 1,20,000 15

... ... ... ...

→ No column says “Customer Type = Premium / Regular / Budget”.

This is unlabelled data.

The unsupervised algorithm will automatically discover groups like:

• Young high-spenders
• Middle-aged low-spenders
• Old premium customers

This is the power of Unsupervised Learning — it finds hidden patterns without any
guidance.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

2. Main Tasks in Unsupervised Learning

1. Clustering → Grouping similar data points .

2. Dimensionality Reduction → Reducing number of features.

3. Clustering Techniques

What is Clustering?

Clustering is the task of dividing unlabelled data into groups (clusters) such that data
points in the same cluster are more similar to each other than to those in other clusters.

Applications: Customer segmentation, anomaly detection, image compression,

document grouping.

A. K-Means Clustering

Steps:

1. Choose number of clusters K.

2. Randomly place K centroids.
3. Assign each point to the nearest centroid.
4. Recalculate centroids as mean of assigned points.
5. Repeat until centroids stabilize.

Example using unlabelled data (from above table):

We set K=3 → K-Means automatically creates 3 clusters:

• Cluster 1: Young customers with high spending (Age ~25–30, Spending >70)
• Cluster 2: Middle-aged moderate spenders
• Cluster 3: Older customers with low spending

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Advantages: Fast, simple, scalable.

Disadvantages: Need to choose K, sensitive to outliers.

B. Hierarchical Clustering

How it Works (Agglomerative – Bottom-up):

• Start with each point as its own cluster.

• Repeatedly merge the two closest clusters.
• Final result shown as a Dendrogram (tree diagram).

Types of Hierarchical Clustering (Pure Theory)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Hierarchical clustering is an unsupervised method that builds a hierarchy of clusters,

represented as a dendrogram (tree-like diagram). It does not require pre-specifying the
number of clusters. You can cut the dendrogram at any level to obtain the desired number
of clusters.

There are two primary types of hierarchical clustering based on the direction of hierarchy
construction:

1. Agglomerative Hierarchical Clustering (Bottom-Up Approach)

This is the most common and widely used type.

• Starting Point: Each data point begins as its own singleton cluster (N clusters for N
data points).
• Process: Iteratively merge the two most similar (closest) clusters into a larger
cluster.
• Ending Point: Continue merging until all points form a single cluster (or until a
stopping criterion is met).
• Direction: Bottom → Up (small clusters → larger clusters).

Advantages:

• Simple to implement.
• Computationally more efficient for small to medium datasets.
• Produces a natural hierarchy.

Disadvantages:

• Once two clusters are merged, they cannot be separated later (greedy nature — no
backtracking).
• Can be sensitive to noise/outliers depending on the linkage method.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

2. Divisive Hierarchical Clustering (Top-Down Approach)

This is the less common type (rarely used in practice).

• Starting Point: All data points start in one single large cluster.
• Process: Recursively split the largest (or most heterogeneous) cluster into smaller
sub-clusters.
• Ending Point: Continue splitting until each data point becomes its own singleton
cluster.
• Direction: Top → Down (large cluster → small clusters).

Advantages:

• Can sometimes produce more balanced hierarchies in certain cases.

Disadvantages:

• Computationally much more expensive (requires deciding how and where to split
at each step, often involving combinatorial choices).
• More complex to implement.
• Less popular due to higher time complexity.

Comparison Table:

Aspect Agglomerative (Bottom-Up) Divisive (Top-Down)

Starting Point Each point = own cluster All points in one cluster

Process Merge closest clusters Split heterogeneous clusters

Direction Bottom to Top Top to Down

Common Usage Very common Rare

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Computational Cost Lower (especially for small/medium data) Higher

Reversibility Irreversible merges Irreversible splits

In practice, agglomerative clustering is almost always preferred due to its efficiency and
ease.

Key Characteristics of Hierarchical Clustering

• Produces a dendrogram for visual interpretation.

• No need to specify number of clusters in advance (cut dendrogram later).
• Works with any distance metric (Euclidean, Manhattan, Cosine, etc.).
• Non-parametric and does not assume cluster shapes (except indirectly via
linkage).

Limitations (Overall)

• Scalability: O(N²) time and space complexity — not suitable for very large datasets
(millions of points).
• Irreversibility: Once merged/split, decisions cannot be undone.
• Sensitive to noise and outliers (especially single linkage).
• Choice of linkage and distance metric significantly impacts results.

Example using unlabelled data:

The dendrogram may show that at height = 10, there are 3 natural groups of customers —
same as K-Means but with hierarchy.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Comparison: K-Means vs Hierarchical Clustering

Feature K-Means Hierarchical Clustering

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Need to specify K Yes No

Speed Fast Slow

Output Flat clusters Dendrogram (hierarchy)

Best for Large datasets Small-medium datasets

Sensitive to outliers High Lower

5.3 Dimensionality Reduction: Importance of Dimensionality Reduction

What is Dimensionality Reduction?

Dimensionality Reduction is the process of reducing the number of input features (dimensions) while
preserving as much useful information as possible.

Why is it Important? (Key Reasons)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

1. Curse of Dimensionality
a. In high dimensions, data becomes sparse → distance between points becomes
meaningless.
b. Models overfit easily and performance drops.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

2. Computational Efficiency
a. Fewer features = faster training and lower memory usage.
3. Overfitting Prevention
a. Removes noise and redundant features → better generalization on new data.
4. Improved Visualization
a. High-dimensional data (50+ features) cannot be plotted → reduce to 2D/3D for easy
understanding.
5. Better Model Performance
a. Removes irrelevant features → higher accuracy and faster convergence.

Real-Life Example

House price dataset with 100 features → after dimensionality reduction → only 10–15 important
components → model trains 10x faster with same or better accuracy.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

5.4 PCA (Principal Component Analysis)

Definition

PCA is an unsupervised dimensionality reduction technique that transforms original features into a new
set of uncorrelated features (Principal Components) while maximizing variance (information retained).

Fundamental Principles

• New components are orthogonal (uncorrelated).

• First principal component captures maximum variance.
• Each next component captures the next highest variance.

Eigenvectors & Eigenvalues (Mathematical Foundation)

• Eigenvectors → directions of maximum variance (new axes).

• Eigenvalues → amount of variance explained by each eigenvector.
Higher eigenvalue = more important component.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Let’s break down Eigenvalues and Eigenvectors in a clear, student-friendly way (very important for PCA
in your syllabus).

What are Eigenvalues and Eigenvectors?

Simple Idea:

When a matrix (transformation) is applied to a vector:

• Most vectors change direction + length

• But some special vectors:
o Do NOT change direction
o Only stretch or shrink

These special vectors are called Eigenvectors, and the amount of stretching/shrinking is called the
Eigenvalue.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

🔹 Mathematical Definition

Intuition

Think of it like:

• You rotate/stretch a rubber sheet

• Some arrows on it:
o Change direction → not eigenvectors
o Stay in same direction → eigenvectors

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Step-by-Step Example

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Eigenvalue (λ) Eigenvector (v)

2 [1, 0]
3 [0, 1]

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Key Points

• Eigenvector → direction remains same

• Eigenvalue → scaling factor
• Found using:
o |A − λI| = 0
• Used in:
o PCA
o Dimensionality Reduction
o Face Recognition, Image Processing

Why Important in PCA?

• Eigenvectors → principal directions (components)

• Eigenvalues → importance (variance) of each direction

Bigger eigenvalue = more important feature direction

Steps in PCA

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

1. Standardize the data (mean = 0, variance = 1).

2. Compute Covariance Matrix.
3. Calculate Eigenvectors and Eigenvalues.
4. Sort eigenvectors by eigenvalues (descending order).
5. Select top k eigenvectors → Project data:

(where ( W ) = matrix of top ( k ) eigenvectors)

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Explained Variance & Choosing Optimal Dimensionality

How to Choose Optimal k?

• Scree Plot or Cumulative Explained Variance.

• Usually keep components that explain ≥ 90–95% total variance.

Advantages & Disadvantages of PCA

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

Advantages Disadvantages

Reduces overfitting Loses interpretability (new components are combinations)

Faster computation & less memory Assumes linear relationships

Removes multicollinearity Sensitive to scaling

Easy visualization in 2D/3D Not suitable for classification (unsupervised)

Applications of PCA

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Image compression & face recognition

• Gene expression analysis (bioinformatics)
• Stock market analysis
• Customer segmentation
• Noise removal in data
• Feature extraction before feeding to other models

Pro Tip for Exam

• PCA is unsupervised → does not use class labels.

• Always standardize data before PCA.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204
 by Kunal Sir

• Choose k using cumulative explained variance ≥ 95%.

Stop, Near, 1st Floor, Above Rupam Sweets/ Priyanka Collections Building Vikas Mitra Mandal Chowk
Road, Karve Nagar, Pune, Maharashtra 411052 , Mobile No.- 8888022204