CSEN3261 Machine Learning

Course Outcome 2
Apply various learning approaches on real time problems using
Classification

2
 Contents
1. Classification,
2. Linear Regression,
3. Gradient Descent,
4. Performance measures,
5. Learning curves,
6. Training a binary classifier, Logistic regression.
7. Multiclass classification, Multi label classification,
Multi output classification.
8. Polynomial Regression,
9. Regularized linear models,
10. Error analysis,
3
 Reference
Chapters 3
Aurelion Geron, Hands-on Machine Learning with Scikit-
Learn, Keras, and Tensor Flow: Concepts, Tools and Techniques
to build Intelligent Systems, 2/e, O’Reilly Media, 2019.

4
 Linear Regression
Linear regression is one of the simplest and most widely
used supervised learning algorithms for predicting a
continuous numeric output based on one or more input
features.
It assumes a linear relationship between the input features
𝑦 = 𝑤𝑇𝑥 + 𝑏
𝑤: weights (slopes)
𝑏: bias (intercept)
𝑥: input vector
𝑦: predicted output

5
 Linear Regression
Goal: Find the values of w and b that minimize the
difference between the predicted and actual values, typically
using the Mean Squared Error (MSE)

6
 Gradient Descent
Gradient Descent is an optimization algorithm used to
minimize a loss function by iteratively updating model
parameters (like weights in linear regression)
1. The algorithm computes the gradient (slope) of the loss
function (𝐿 𝜃 ) with respect to the parameters 𝜃.
2. It updates parameters in the opposite direction of the
gradient to reduce the error.
𝜃 ← 𝜃 − 𝛼∇𝜃 𝐿 𝜃
𝛼 is learning rate range is [0.1−1]

7
 Algorithm to Update the Weights
Given : Input features: 𝑋∈𝑅𝑛×𝑑 ;Output values: 𝑦∈𝑅𝑛
1. Initialize weights and bias
2. Repeat for T epochs:
For each sample x(i),y(i) :
a. Compute prediction 𝑦ො (𝑖) = 𝑤 𝑇 𝑥 (𝑖) + 𝑏
b. Compute error 𝑒 (𝑖) = 𝑦ො (𝑖) − 𝑦 (𝑖)
1 𝑛 (𝑖) (𝑖) 2
𝐿(𝑤, 𝑏) = σ 𝑦ො − 𝑦
𝑛 𝑖=1
𝜕𝐿 1 𝑛 𝜕𝐿 1
c. Compute gradients = σ𝑖=1 𝑒 (𝑖) 𝑥 (𝑖) ; = σ𝑛𝑖=1 𝑒 (𝑖)
𝜕𝑤 𝑛 𝜕𝑏 𝑛
3. Update the parameters
𝜕𝐿 𝜕𝐿
𝑤 ←𝑤−𝛼 ;𝑏 ← 𝑏 − 𝛼
8 𝜕𝑤 𝜕𝑏
 Example
x(i) y(i) Initial values:
1 2 𝑤=0; b=0;
2 3 α=0.1 (learning rate)
3 4

Step 1: Predictions
y1=w⋅1+b=0 ;y2=w⋅2+b=0;y3=w⋅3+b=0
Step 2: Errors
e1=y1−y1=0−2=−2;e2=0−3=−3;e3=0−4=−4
9
 Example
3. Compute gradients
𝑛
𝜕𝐿 1
= ෍ 𝑒 (𝑖) 𝑥 (𝑖) =
𝜕𝑤 𝑛
𝑖=1
𝑛
𝜕𝐿 1
= ෍ 𝑒 (𝑖) =
𝜕𝑏 𝑛
𝑖=1
4. Update Parameters
𝜕𝐿
𝑤 ←𝑤−𝛼 ;=
𝜕𝑤
𝜕𝐿
𝑏 ←𝑏−𝛼 =
𝜕𝑏
10
 Example
3. Compute gradients
𝑛
𝜕𝐿 1 (𝑖) (𝑖)
1
= ෍ 𝑒 𝑥 = −2 × 1 − 3 × 2 − 4 × 3 = −6.67
𝜕𝑤 𝑛 3
𝑖=1
𝑛
𝜕𝐿 1 (𝑖)
1
= ෍ 𝑒 = −2 − 3 − 4 = −3
𝜕𝑏 𝑛 3
𝑖=1
4. Update Parameters
𝜕𝐿
𝑤 ←𝑤−𝛼 ; = 0 − −0.1 −0.67 = 0.667
𝜕𝑤
𝜕𝐿
𝑏 ←𝑏−𝛼 = 0 − −0.1 −3 = 0.3
𝜕𝑏

11
 Example

12
 Example

13
 Learning curves

14
 Learning curves

15
 Learning curves

16
 Quadratic Regression
Given : Input features: 𝑋∈𝑅𝑛×𝑑 ;Output values: 𝑦∈𝑅𝑛
1. Initialize weights and bias
2. Repeat for T epochs:
For each sample x(i),y(i) :
(𝑖) (𝑖) 2
a. Compute prediction 𝑦ො = 𝑎 𝑥 + 𝑏 𝑥 (𝑖) + 𝑐
b. Compute error 𝑒 (𝑖) = 𝑦ො (𝑖) − 𝑦 (𝑖)
1 𝑛 2
𝐿(𝑤, 𝑏) = σ 𝑦ො (𝑖) − 𝑦 (𝑖)
𝑛 𝑖=1
c. Compute gradients
𝑛 𝑛 𝑛
𝜕𝐿 1 (𝑖) (𝑖) 2 𝜕𝐿 1 (𝑖) (𝑖)
𝜕𝐿 1
= ෍𝑒 𝑥 ; = ෍𝑒 𝑥 , = ෍ 𝑒 (𝑖)
𝜕𝑎 𝑛 𝜕𝑏 𝑛 𝜕𝑐 𝑛
𝑖=1 𝑖=1 𝑖=1
𝜕𝐿
3. Update the parameters𝑎, 𝑏, 𝑐 ← 𝑎, 𝑏, 𝑐 − 𝛼
17 𝜕𝑎,𝑏,𝑐
 Quadratic Regression

18
 Classification

19
 Binary vs Multiclass

20
21
 Multi-output
Predicts multiple outputs simultaneously. In multi-output
classification, the model will give two or more outputs after making
any prediction. In other types of classifications, the model usually
predicts only a single output.

An example of a multi-output classification model is a model that

predicts the type and color of fruit simultaneously. The type of
fruit can be, orange, mango and pineapple (multi-class).
The color can be, red, green, yellow, and orange (multi-class). The
multi-output classification solves this problem and gives two
prediction results.

All multi-label classification is technically multi-output, but not all

multi-output is multilabel.

22
 Car Brand Classification
1. Binary Classification
 Use case: Classify whether a car belongs to a specific brand or
not.
 Example: "Is this car a Tesla?" YES/NO
2. Multi-class Classification
 Use case: Predict exactly one brand from several possible ones.
 Example: "Classify the car as either Ford, Toyota, BMW, or Tesla"
3. Multilabel Classification
 Use case: Predict multiple brand affiliations (rare, but
applicable in case of co-branded or rebadged vehicles).
 Example: A jointly developed car model (e.g., Toyota 86 = Toyota
23
+ Subaru)
 Car Brand Classification
4. Multi-output Classification
 Use case: Predict brand plus additional related targets like
model segment, country of origin, or luxury rating.
 Target: Multiple independent outputs e.g., Brand: Tesla,
Origin: USA, Type: Electric
Scenario Type Example Output
Is it a Tesla? Binary 0 or 1
Which brand is it? Multi-class "Toyota"
Does it belong to
Multilabel ["Toyota", "Subaru"]
multiple brands?
Predict brand + ["Tesla", "USA",
Multi-output
24 origin + type "Electric"]
 Logistic Regression
Logistic regression is a supervised learning algorithm
used for binary and multiclass classification tasks. It
predicts the probability that a given input belongs to a
particular class.

25
 Algorithm
Given : Input features: 𝑋∈𝑅𝑛×𝑑 ;Output values: 𝑦∈𝑅𝑛
1. Initialize weights and bias
2. Repeat for T epochs:
For each sample x(i),y(i) :
(𝑖) (𝑖)
a. Compute prediction 𝑧 (𝑖) = 𝑤1 𝑥1 + 𝑤2 𝑥2 + 𝑏;
(𝑖) (𝑖)
1
𝑦ො = 𝜎 𝑧 = (𝑖)
1 + 𝑒 −𝑧
b. Compute Loss function (binary cross entropy
𝑛
1
𝐿 = ෍ 𝑦 (𝑖) log 𝑦ො (𝑖) + 1 − 𝑦 (𝑖) log 1 − 𝑦ො (𝑖)
𝑛
𝑖=1
c. Compute gradients by applying chain rule.
𝜕𝐿 𝜕𝐿 𝜕𝑦ො (𝑖) 𝜕𝑧 (𝑖) 𝜕𝐿 𝜕𝐿 𝜕𝑦ො (𝑖) 𝜕𝑧 (𝑖)
= (𝑖) . (𝑖) . , = (𝑖) . (𝑖) .
26 𝜕𝑤𝑗 𝜕𝑦ො 𝜕𝑧 𝜕𝑤𝑗 𝜕𝑏 𝜕𝑦ො 𝜕𝑧 𝜕𝑏
 Algorithm
𝑛 𝑛
𝜕𝐿 1 (𝑖) 𝜕𝐿 1
= ෍ 𝑦ො (𝑖) − 𝑦 𝑖
𝑥𝑗 ; = ෍ 𝑦ො (𝑖) − 𝑦 𝑖
𝜕𝑤𝑗 𝑛 𝜕𝑏 𝑛
𝑖=1 𝑖=1
3. Update the parameters
𝜕𝐿
𝑤𝑗 , 𝑏 ← 𝑤𝑗 , 𝑏 − 𝛼
𝜕𝑤𝑗 , 𝑏
Derivative of loss w.r.t prediction:
𝜕𝐿 𝑦𝑖 1−𝑦 𝑖 𝑦ො 𝑖 − 𝑦 𝑖
(𝑖)
=− 𝑖 + 𝑖
= 𝑖
𝜕𝑦ො 𝑦ො 1 − 𝑦ො 𝑦ො 1 − 𝑦ො 𝑖
Derivative of sigmoid:
𝜕𝑦ො (𝑖) −1 −𝑧 𝑖
1 1 𝑖 𝑖
= 2𝑒 −1 = 1− = 𝑦ො 1 − 𝑦ො
𝜕𝑧 (𝑖) 1 + 𝑒 −𝑧
𝑖 1+𝑒 −𝑧 𝑖
1+𝑒 −𝑧 (𝑖)

Derivative of z:
𝜕𝑧 (𝑖) (𝑖)
= 𝑥𝑗
27
𝜕𝑤𝑗
 Example
x1 x2 y
1 2 0
2 1 0
3 4 1
4 3 1

28
 𝑤1≈2.165
𝑤2≈2.165
𝑏≈−10.43

29
30
31
 Interpretation
 Final decision boundary is
2.16 x1 +2.16x2 − 10.43 = 0
This boundary clearly separates Class 0 and Class 1 points in
the training set.
 Visual Interpretation
Points above the line:
w1x1+w2x2+b>0 ⟹ y^>0.5 ⟹ Class 1
 Points below the line:
w1x1+w2x2+b<0 ⟹ y^<0.5 ⟹ Class 0

32
 Thank You All Very Much

33