MACHINE LEARNING

UNIT 1: Foundations of Machine Learning

• Introduction to ML:
1.1 What is Machine Learning (ML)?
Machine Learning (ML) is a branch of Artificial Intelligence in which
computers learn patterns from data and make decisions or predictions
without being explicitly programmed for every task.
Example
If we give a machine many emails marked spam and not spam, it learns
the pattern and can classify new emails automatically.
1.2 What problems require ML?
Traditional programming works when rules are clear and definable. ML
steps in when patterns are too complex, too numerous, or too dynamic to
code by hand.
Problems that require ML:
• Prediction problems → predicting future values from past data
Example: house price prediction, weather forecasting
• Classification problems → assigning data into categories
Example: spam email detection, disease diagnosis
• Pattern recognition problems → finding hidden patterns
Example: face recognition, handwriting recognition
• Recommendation problems → suggesting items based on user
behavior
Example: Netflix movie recommendations, Amazon product
suggestions
• Decision-making in uncertain environments
Example: fraud detection in banking, stock trend analysis

1.3 How is ML different from traditional programming?

In traditional programming:
• Programmer writes fixed rules
• Input + Rules → Output
Flow:
Data + Program → Output
Example:
If marks > 40 → Pass
Else → Fail
This works only when rules are clearly defined.
In ML: Machine Learning first learns from past examples, then uses
that learning to predict future outcomes.
• Data and output examples are given to the machine
• Machine learns rules automatically
Flow:
Data + Output → Model (learned rules)
Then:
New Data + Model → Prediction
Example:
Student performance prediction based on attendance, study hours,
previous marks.
Traditional Programming Machine Learning

Rules written manually Rules learned automatically

Fixed logic Learns from data

Best for simple tasks Best for complex tasks

Does not improve automatically Improves with more data

1.4 Role of ML in Modern Applications

1.5 Types of Machine Learning

1) Supervised Learning
Supervised Learning is a type of machine learning in which the
model is trained using labeled data, where both input and correct
output are provided.
The machine learns the relationship between input and output and
predicts results for new data.
Example:
Predicting student result using study hours.

2) Unsupervised Learning
Unsupervised Learning is a type of machine learning in which the
model works with unlabeled data and discovers hidden patterns or
groups in the data by itself.
No correct output is given.
Example:
Grouping customers based on purchase behavior.
3) Reinforcement Learning
Reinforcement Learning is a type of machine learning in which an
agent learns by interacting with the environment and receiving
rewards or penalties for its actions.
Learning happens through trial and error.
Example:
A robot learning the correct path.
 • Supervised Learning: Learning from labeled data.
• Unsupervised Learning: Learning hidden patterns from unlabeled data.
• Reinforcement Learning: Learning through rewards and penalties.

1.6 Bayes’ Theorem

Bayes’ Theorem is used to find the probability of an event based on
prior knowledge of related conditions.
Formula:

Where:
• P(A|B) = probability of A given B

• P(B|A) = probability of B given A

• P(A) = prior probability of A

• P(B) = probability of B
1.7 ML Development Lifecycle
The Machine Learning Development Lifecycle is the step-by-step
process used to build, train, test, and deploy a machine learning
model.

Problem → Data → Preprocessing → Features → Model → Training →

Testing → Deployment → Monitoring.

1.8 ML Tools and Libraries with Difference

Tool Short Definition Main Difference

Scikit-learn Python library for basic machine learning algorithms Best for traditional ML models

TensorFlow Framework for deep learning developed by Google Best for large-scale deep learning

PyTorch Deep learning library developed by Meta Platforms More flexible and popular in research

Google Colab Online platform to run Python code No installation, runs in browser
UNIT 2: Supervised Learning – Regression Models and Feature
Engineering.
2.1 Regression models are used when the target/output is a continuous
numeric value such as salary, price, temperature, or marks.

1. Linear Regression — the foundation of all regression. Fit a straight

line (or hyperplane) through data to predict a continuous output.
Formula : slope : y= mx+c
2. Polynomial Regression — when the relationship curves. You add
powers of x (x², x³…) as new features, but the model is still linear in its
coefficients.

3. Ridge & Lasso Regression — both add a penalty term to the loss
function to prevent overfitting. They differ in how they penalise large
coefficients. Here βi=weight (wi).

Ridge — when you believe most features are relevant but their effects are small
Lasso — when you suspect many features are irrelevant and want automatic selection
4. Bayesian Regression
Bayesian Regression estimates output while considering uncertainty in
data.
When to use:
When prediction confidence is important.
Example:
Risk prediction in healthcare.

Model Formula Penalty Best for Output

Linear Single
Linear y = β₀ + β₁x None
relationships value

y = β₀ + β₁x₁ + Single
Multiple None Many features
… value

y = β₀ + β₁x + Curved Single

Polynomial None
β₂x² + … patterns value

Correlated Single
Ridge (L2) Loss + λΣwᵢ² Shrinks all w
features value

Feature Single
Lasso (L1) Loss + λΣ|wᵢ| Zeros out w
selection value

Prior +
Prior acts as Small data,
Bayesian Likelihood → Distribution
regulariser uncertainty
Posterior
2.2 Feature Engineering in Supervised Learning
Feature Engineering means selecting and transforming input features so
the model learns better and gives accurate predictions.

1) Importance of Feature Selection

Feature Selection is the process of choosing only important input
variables for model training.
Why use it?
• improves accuracy

• reduces training time

• removes irrelevant data

• helps prevent overfitting

When to use it?

When dataset has many columns but only some affect output.
Example:
For salary prediction:
• useful features → experience, education

• less useful → ID number

Model performs better with important features only.

2) Feature Scaling
Feature scaling makes all features come to similar range.
Needed because different feature values may have different scales.
Example:
• Age = 25

• Salary = 50000

Large values dominate small values if scaling is not done.

Normalization Standardization
Range 0 to 1 Mean = 0, SD = 1
Used when fixed range needed Used when data distribution matters
3) Encoding Techniques:
Encoding converts categorical data into numerical form.

1) One-Hot Encoding
One-Hot Encoding creates separate binary columns for each
category.
Example:
Color = Red, Blue, Green
Color Red Blue Green
Red 1 0 0
Blue 0 1 0
Used when categories have no order.

2) Label Encoding
Label Encoding assigns a unique number to each category.
Example:
Color Label
Red 0
Blue 1
Green 2
Simple and memory efficient.
Model may assume order exists.

3) Ordinal Encoding
Ordinal Encoding is used when categories have natural order.
Example:
Size Code
Small 1
Medium 2
Large 3
Used when order matters.

5) Frequency Encoding
Frequency Encoding replaces each category with how often it
appears in the dataset.
Example:
Suppose data:
City

Mumbai , Pune , Mumbai , Delhi , Mumbai , Pune

Count frequency:
• Mumbai = 3

• Pune = 2

• Delhi = 1
Encoded Table:
City Frequency
Mumbai 3
Pune 2
Delhi 1
Useful when dataset has many categories.

6) Binary Encoding
Binary Encoding first converts categories into numbers, then into
binary form.
Example:
Categories:
Color Label Binary
Red 1 001
Blue 2 010
Green 3 011
Reduces number of columns compared to one-hot encoding.

Summary :
Encoding Method
One-Hot Separate columns
Label Single number
Frequency Count of occurrence
Binary Binary digits
Embeddings Dense vectors
 • L1 Normalization
L1 Normalization scales values so that the sum of absolute values
becomes 1.
Formula
𝑿𝒊
𝑿′𝒊 =
∑ ∣ 𝑿𝒊 ∣
Where:
• Xᵢ = original value

• Σ|Xᵢ| = sum of absolute values of all elements

Example
Given vector:
[2, 3, 5]
Step 1: Sum absolute values
|2| + |3| + |5| = 10
Step 2: Divide each value by 10
• 2/10 = 0.2

• 3/10 = 0.3

• 5/10 = 0.5

Result:
[0.2, 0.3, 0.5]
L1 normalized vector = [0.2, 0.3, 0.5]
2.3 Bias-Variance Tradeoff
Bias-Variance Tradeoff explains how to balance underfitting and
overfitting in machine learning.

1) Bias
Bias is the error caused when a model is too simple and cannot capture
the actual pattern of data.
High bias leads to underfitting.
Example:
Using a straight line for highly curved data.
2) Variance
Variance is the error caused when a model learns too much from training
data, including noise.
High variance leads to overfitting.
Example:
Model performs well on training data but poorly on test data.
3) Underfitting
Underfitting happens when the model is too simple and gives poor
performance on both training and test data.
Cause:
• high bias
 • insufficient learning
4) Overfitting
Overfitting happens when the model learns training data too exactly and
fails on new data.
Cause:
• high variance

• too complex model

2.3.1 Regularization Techniques

Regularization reduces overfitting by adding penalty to model
coefficients.
Formulas:
• L1 Regularization
𝐿 = Loss + 𝜆∑ ∣ 𝑤 ∣
L1 adds absolute penalty to weights.

• L2 Regularization
𝐿 = Loss + 𝜆∑𝑤 2
L2 adds square penalty to weights.

• Loss Formula
For one prediction:
Loss = (𝑦 − 𝑦̂)2
Where:
• y = actual value

• ŷ = predicted value

Summary:
L1 → add absolute weights
L2 → add squared weights
Loss = Actual − Predicted error
2.4 Model Evaluation for Regression
These metrics tell us how well a regression model predicts data.
1)MAE (Mean Absolute Error)
Definition
MAE measures the average absolute difference between actual and
predicted values.
It tells average prediction error without squaring.
Formula
∑ ∣ 𝑦 − 𝑦̂ ∣
𝑀𝐴𝐸 =
𝑛

Where:
• y = actual value

• ŷ = predicted value

• n = number of observations

2) MSE (Mean Squared Error)

Definition
MSE measures the average squared difference between actual and
predicted values.
It tells how much error the model makes.
Formula
∑(𝑦 − 𝑦̂)2
𝑀𝑆𝐸 =
𝑛

Where:
• y = actual value

• ŷ = predicted value

• n = number of observations
3) RMSE (Root Mean Square Error)
Definition
RMSE measures average prediction error.
Formula
∑(𝑦 − 𝑦̂)2
𝑅𝑀𝑆𝐸 = √
𝑛

Where:
• y = actual value

• ŷ = predicted value

• n = number of observations

4) R-squared (Coefficient of Determination)

Definition
Shows how much variance is explained by model.
Value range: RSS measures prediction error.
• 0 → poor model Formula
𝑅𝑆𝑆 = ∑(𝑦 − 𝑦̂)2
• 1 → perfect model
Where:
Formula • y = actual value
𝑅𝑆𝑆 • ŷ = predicted value
𝑅2 = 1 −
𝑇𝑆𝑆 TSS (Total Sum of Squares)
Definition
TSS measures total variation from mean.
Where: Formula
• RSS = residual sum of squares 𝑇𝑆𝑆 = ∑(𝑦 − 𝑦ˉ)2
• TSS = total sum of squares Where:
• ȳ = mean of actual values

5) Adjusted R-squared
Definition
Improved version of R² that considers number of features.
Used when multiple variables exist.
Formula
𝑛−1
𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑 𝑅 2 = 1 − (1 − 𝑅 2 )
𝑛−𝑝−1

Where:
• n = observations

• p = features
UNIT 3 : Supervised Learning – Classification Models and
Ensemble Learning.
3.1 Classification models:
In classification, output is categorical (Yes/No, Pass/Fail,
Spam/Not Spam).
1) Logistic Regression
Definition
Used for binary classification when output has two classes.
Examples:
• 0 = No

• 1 = Yes

Unlike linear regression, it gives probability between 0 and 1.

Formula:
1
𝑃(𝑦 = 1) =
1 + 𝑒 −𝑧

Where:
𝑧 = 𝑏0 + 𝑏1 𝑥
1) Decision Tree
Definition
A Decision Tree is a tree-like model that makes decisions by splitting
the data based on features.
It’s a supervised learning algorithm used for classification and
regression.
• Each node represents a feature (condition).

• Each branch represents a decision (yes/no).

• Each leaf represents an outcome (class label or value).

Example – Classification
Suppose we want to predict if a student passes or fails based on marks:
• If marks > 40 → Pass

• Else → Fail

Decision Tree:
Marks > 40?
/ \
Yes No
Pass Fail

How it Works
1. Start at the root node.
2. Split data based on best feature (using information gain or Gini
index).
3. Repeat splitting until all nodes are pure or stopping condition
reached.
Where pᵢ = fraction of samples in class i
4. Assign labels to leaf nodes.
Metrics to Choose Best Split • Perfect split → Gini = 0

• Gini Index (for classification) • Worst split → Gini → 0.5 (for 2

𝐺𝑖𝑛𝑖 = 1 − ∑𝑝𝑖2 classes)

• Entropy / Information Gain
Entropy(S)= −∑pi log2 pi + qi log2 qi
𝐼𝐺 = 𝐸𝑛𝑡𝑟𝑜𝑝𝑦(𝑝𝑎𝑟𝑒𝑛𝑡) − ∑𝐸𝑛𝑡𝑟𝑜𝑝𝑦(𝑐ℎ𝑖𝑙𝑑𝑟𝑒𝑛)

• MSE (for regression)

2) Random Forest
Definition
Random Forest is an ensemble of multiple decision trees.
It combines predictions from many trees to improve accuracy.
• For classification → majority vote of trees

• For regression → average of tree outputs

3.2 Ensemble Learning – Combining Multiple Models
Ensemble Learning is a method in machine learning where multiple
models (often called weak learners) are combined to form a stronger,
more accurate model. The idea is that combining models reduces
errors and variance, leading to better generalization.
Why use ensemble learning?
• Single models may overfit or underfit.

• Combining predictions can reduce bias and variance.

• Often improves accuracy and robustness of predictions.

Common ensemble methods:

1. Bagging (Bootstrap Aggregating)
2. Boosting
3. Stacking (less common in basics)
1. Bagging (Bootstrap Aggregating)

Definition:
Bagging is an ensemble method that builds multiple independent models
on different random subsets of the training data and combines their
predictions to improve accuracy. The main goal is to reduce variance
(i.e., make the model less sensitive to noise in the training data).

2. Boosting
Definition:
Boosting is an ensemble method that builds models sequentially, where
each new model focuses on correcting the mistakes of previous models.
The main goal is to reduce bias and variance, creating a strong model
from weak learners.

Feature Bagging Boosting

Training Independent models Sequential models
Goal Reduce variance Reduce bias & variance
Hard-to-predict/misclassified
Focus Random samples
samples
High variance model Weak learners (e.g., stump
Base Learner
(e.g., tree) tree)
Combining
Majority vote / Average Weighted combination
Predictions
Parallelization Yes No, sequential
Example Random Forest AdaBoost, Gradient Boosting
3.3 Hyperparameter Tuning in Supervised Learning
Hyperparameters are settings for a machine learning model that are set
before training and not learned from the data. Proper tuning can
significantly improve model performance.
Examples:
• Decision Tree: max_depth, min_samples_split

• Random Forest: n_estimators, max_features

• SVM: C, gamma, kernel

• KNN: k (number of neighbors)

The three tuning strategies — each with a different trade-off between

coverage and speed:
3.4 Model Evaluation for Classification.
UNIT 4 : Unsupervised Learning – Clustering
4.0 Unsupervised Learning
Definition:
Unsupervised Learning is a type of machine learning where the model
learns patterns from unlabeled data (no target variable).
4. 1. Clustering
Definition:
Clustering is the task of grouping similar data points together so that:
• Within a cluster: data points are similar.

• Between clusters: data points are different.

Use Cases:
• Customer segmentation in marketing.

• Grouping genes in bioinformatics.

• Organizing documents or images.

Popular Clustering Algorithms:

1. K-Means
2. DBSCAN
3. Hierarchical Clustering

A. K-Means Clustering
Concept:
• Partitions data into K clusters.

• Each cluster has a centroid (mean of points).

• Assigns points to the nearest centroid.

Steps:
1. Choose K (number of clusters).
2. Initialize K centroids randomly.
3. Assign each point to the nearest centroid.
4. Recalculate centroids as the mean of assigned points.
5. Repeat steps 3-4 until centroids don’t change (convergence).
C. Hierarchical Clustering
Concept:
• Builds a tree of clusters (dendrogram).

• No need to specify the number of clusters initially.

Types:
1. Agglomerative (Bottom-Up)
o Each point starts as its own cluster.

o Merge closest clusters step by step.

2. Divisive (Top-Down)
o Start with one cluster containing all points.

o Split recursively until individual points.

Distance Metrics:
• Euclidean distance:
𝑑 = √(𝑥2 − 𝑥1 )2 + (𝑦2 − 𝑦1 )2
• Manhattan
For 2D points:
𝐷Manhattan =∣ 𝑥2 − 𝑥1 ∣ +∣ 𝑦2 − 𝑦1 ∣
• Cosine, etc.
Linkage Methods:
• Single linkage (nearest) – MINIMUM VALUE

• Complete linkage (farthest)- MAXIMUM VALUE

• Average linkage-AVERAGE
B. DBSCAN (Density-Based Spatial Clustering of Applications with
Noise)
Concept:
• Groups dense regions of points as clusters.

• Points in low-density regions are considered noise (outliers).

Parameters:
1. eps → radius to search for neighbors
2. minPts → minimum points to form a dense region
Point types in DBSCAN:
1. Core point: ≥ minPts within eps radius
2. Border point: < minPts in its neighborhood but reachable from a
core point
3. Noise: not a core point and not reachable
UNIT 5 : Unsupervised Learning – Deep Learning
5.1. Deep Learning
Deep Learning is a subset of Machine Learning that uses Neural
Networks with multiple layers to learn complex patterns from data.
Neuron: A neuron is the smallest decision-making unit in Deep
Learning.

Main Parts:
1. Dendrites
• Receive signals from other neurons

2. Cell Body (Soma)

• Processes signals

3. Axon
• Sends output signal

4. Synapse
• Connection between neurons
5.2 Artificial Neural Network (ANN) (fully connected)
An Artificial Neural Network (ANN) is a computational model inspired
by the human brain, consisting of interconnected artificial neurons that
process data and learn patterns.
Structure of ANN
ANN is organized into layers:
• Input Layer → Receives input data

• Hidden Layer(s) → Performs computations and feature extraction

• Output Layer → Produces final result

5.3 Perceptron
• Simplest neural network (single neuron)

• Used for binary classification

• Formula:

𝒚 = 𝒇(𝒘 ⋅ 𝒙 + 𝒃)

• Limitation: Cannot solve non-linear problems (e.g., XOR)

5.4 MLP — Multi-Layer Perceptron

A Multi-Layer Perceptron (MLP) is a type of artificial neural network
made up of multiple layers of neurons arranged as an input layer, one or
more hidden layers, and an output layer. Each neuron in these layers is
connected with weighted links, and the network processes data by
passing it forward from input to output. MLP uses activation functions
such as ReLU or sigmoid, which allow it to learn and model complex,
non-linear relationships in data. Because of this ability, MLP is widely
used for tasks like classification and prediction.

5.6 Backpropagation is the learning algorithm used to train an MLP or

any neural network. In this process, the network first performs a forward
pass to generate a prediction. The predicted output is then compared
with the actual output to calculate the error using a loss function. This
error is propagated backward through the network, and the weights are
updated using gradient descent to minimize the error. By repeating this
process multiple times, the network gradually improves its accuracy.

• Activation function : An activation function is a mathematical

function used in a neuron that decides whether the neuron should
activate (fire) or not and what output it should produce.

In simple terms, after a neuron calculates the weighted sum of

inputs, the activation function converts that value into the final
output.
• How it Works
First, the neuron computes:
𝑧 = 𝑤1 𝑥1 + 𝑤2 𝑥2 + ⋯ + 𝑏
Then activation function is applied:
𝑦 = 𝑓(𝑧)
So, activation function transforms input into output.

• Common Activation Functions

1. ReLU (Rectified Linear Unit)
𝑓(𝑥) = max⁡(0, 𝑥)
• Outputs 0 if input is negative, otherwise same value

• Most widely used in deep learning

2. Sigmoid
1
𝑓(𝑥) =
1 + 𝑒 −𝑥
• Output between 0 and 1
• Used for probability-based outputs
5.7 CNNs (Convolutional Neural Networks)
CNNs are specialized for image and spatial data. They automatically extract features
like edges, textures, and shapes.
• Key Components:
o Convolution Layer: Applies filters/kernels to detect features.
o Pooling Layer: Reduces dimensionality while retaining important features
(Max Pooling, Avg Pooling).
o Fully Connected Layer: Combines features for final prediction.

• Use Case: Object detection, image classification, face recognition.

5.8 RNNs (Recurrent Neural Networks)
RNNs are specialized neural networks for sequential data, meaning they
consider the order of inputs. Unlike traditional networks, RNNs maintain
a hidden state that stores information from previous steps, enabling them
to "remember" past data.
• Variants:

o LSTM (Long Short-Term Memory): Handles long-term

dependencies, prevents the “vanishing gradient” problem.

o GRU (Gated Recurrent Unit): A simpler, faster alternative

to LSTM with similar performance.

• Applications: Text generation, sentiment analysis, speech

recognition, time series prediction.

5.9 Autoencoders and Variational Autoencoders (VAEs)

Autoencoders (AE):
Neural networks trained to reconstruct the input. They have two parts:
1. Encoder: Compresses input into a smaller, latent representation.
2. Decoder: Rebuilds the input from this compressed representation.
• Purpose: Feature learning, dimensionality reduction, anomaly

detection.
• Anomaly Detection: Poor reconstruction of unusual inputs signals

anomalies.
Variational Autoencoders (VAEs):
A probabilistic version of AE that learns a distribution in the latent
space. New data can be generated by sampling from this distribution.
• Applications: Image generation, denoising, creating synthetic

data.

5.10 Generative Models

Generative models learn the underlying distribution of data to
generate new, realistic samples.
GANs (Generative Adversarial Networks):
• Consist of two networks:

1. Generator: Creates fake data.

2. Discriminator: Detects whether data is real or fake.
• They compete in a minimax game, improving each other

iteratively.
• Applications: Image synthesis, art generation, super-resolution,

data augmentation.
 Data Type
Component Purpose Key Feature Use Cases
/ Input
Tabular,
Layers of Classification,
Neural General- structured
neurons, learns regression,
Networks (NNs purpose data,
weights via predictive
/ MLPs) learning simple
backpropagation modeling
images
CNNs Convolution + Image
Feature
(Convolutional pooling layers, Images, recognition,
extraction from
Neural spatial videos object detection,
images
Networks) hierarchies medical imaging
Sequences
RNNs Text generation,
Maintains hidden (text,
(Recurrent Sequential data sentiment
state to capture speech,
Neural modeling analysis, speech
temporal info time
Networks) recognition
series)
Feature
Encoder
Dimensionality Images, extraction,
Autoencoders compresses,
reduction & signals, anomaly
(AEs) decoder
reconstruction tabular detection,
reconstructs
denoising
Learns latent Image synthesis,
Variational
Probabilistic distribution, Images, text generation,
Autoencoders
data generation generates new text, audio data
(VAEs)
samples augmentation
GANs Two networks Image synthesis,
(Generative Generate (generator & Images, art generation,
Adversarial realistic data discriminator) audio, text super-resolution,
Networks) compete augmentation