भारतीय सूचना ौ यो गक सं थान नागपु र

Indian Institute of Information Technology Nagpur

An Institution of National Importance By An Act of Parliament

Regression Analysis

Presented by:
Dr. Amol P. Bhopale
Assistant Professor
Department of Computer Science and Engineering
IIIT Nagpur
Given a Dataset
Training Eg. or Input Features Output
Instance
X1 X2 X3 . . Xn Y

I1 a1 a2 a3 . . an Y1

I2 b1 b2 b3 . . bn Y2 Classification
or Regression
I3 c1 c2 c3 . . cn Y3 ??
.
.
.
Im P1 P2 P3 . . Pn Ym

Test Input Z1 Z2 Z3 . . Zn ??
Model has to
predict it.
09-02-2026 Indian Institute of Information Technology, Nagpur 2
Supervised Learning-
For each input x, the desired output y is given. Here y is
the label.

Pre-classified training example-

Given: x, y pairs.
for any unseen value of x, determine the best label y.

Classification - When y is discrete

Eg: Masked or not
Red or Black
Cat, Dog, Horse

Regression - When y is continuous.

Eg:09-02-2026
carpet area, location - predict the price of the house.
Indian Institute of Information Technology, Nagpur 3
Classification
• Classification is a process of categorizing a given set of data
into classes.
• It can be performed on both structured or unstructured
data.
• The process starts with predicting the class of given data
points. The classes are often referred to as target, label or
categories.

09-02-2026 Indian Institute of Information Technology, Nagpur 4

Classification Example:-
You are given with the collection of emails, determine the
spam or non-spam email from it.
Here the task is to learn
the function which can
classify and predict if the
email is spam or
non-spam.
The line F2 is better than
Classification
F1.
Algorithms
• Decision Tree
• K-NN
• SVM
09-02-2026 Indian Institute of Information Technology, ANN
• Nagpur 5
Regression
• A technique for determining the statistical relationship
between two or more variables where a change in a
dependent variable is associated with, and depends on, a
change in one or more independent variables.

• A regression problem is used when the output variable is a

real or continuous value, such as "Salary" or "weight".

09-02-2026 Indian Institute of Information Technology, Nagpur 6

Regression Example –
• Sales of a product can be predicted by using the
relationship between sales volume and amount of
advertising.
• The performance of a student can be predicted by using
the relationship between hours of study and score in the
exam.
• The size of a child’s vocabulary can be predicted by using
the relationship between the vocabulary size, the child’s age
Regression
and the parents’ educational input.
Techniques
• Linear Regression
• Multiple Regression
09-02-2026 Indian Institute of Information Technology, Nagpur 7
Regression Example –
Estimate the price of the house from the given data.

09-02-2026 Indian Institute of Information Technology, Nagpur 8

Regression Example –
Estimate the price of the house from the given data.

09-02-2026 Indian Institute of Information Technology, Nagpur 9

Regression Example –
Estimate the price of the house from the given data.

What would be the

price of the medium
09-02-2026
size house
Indian Institute of Information Technology, Nagpur 10
Regression Example –
Estimate the price of the house from the given data.

09-02-2026 Indian Institute of Information Technology, Nagpur 11

Regression Example –
Estimate the price of the house from the given data.

09-02-2026 Indian Institute of Information Technology, Nagpur 12

Regression Example –
Estimate the price of the house from the given data.

09-02-2026 Indian Institute of Information Technology, Nagpur 13

Linear Regression: Pictorially
▪ Linear regression is like fitting a line or (hyper)plane to a set of points
What if a line/plane
doesn’t model the
input-output relationship
Original (single) Two features
very well, e.g., if their Can fit a plane
feature
relationship is better Nonlinear curve (linear)
modeled by a nonlinear needed
curve or curved surface?

Do linear
models become
useless in such
cases?
(Feature (Feature
2) 1)

▪ The line/plane must also predict outputs the unseen (test) inputs well
09-02-2026 Indian Institute of Information Technology, Nagpur 14
Simplest Possible Linear Regression Model
⚫ This is the base model for all
statistical machine learning
⚫ x is a one feature data
variable
⚫ y is the value we are trying to
predict
⚫ The regression model is

Two parameters to estimate –

the slope of the line w1 and
the y-intercept w0
⚫ ε is the unexplained,
random, or error
component.
09-02-2026 Indian Institute of Information Technology, Nagpur 15
Dependent and Independent Variable
• Independent variables are considered as an input to a
system and may take on different values freely.
• Dependent variables are those values that change as a
consequence of changes in other values in the system.
• Independent variable is also called as predictor or
explanatory variable and is denoted by X
• Dependent variable is also called as response variable and
is denoted by Y.

16
Linear Regression
• The Simplest mathematical relationship between two
variables x and y is a linear relationship
• In a cause and an effect relationship, the independent
variable is cause, and the dependent variable is the
effect.
• Least squares linear regression is a method for
predicting the value of a dependent variable Y, based
on the value of an independent variable X.

17
More generally
•

Unlike models like KNN and DT, here

we have an explicit problem-specific
09-02-2026 objective (loss function)
Indian Institute thatTechnology,
of Information we wish to
Nagpur 18
optimize for
Linear Regression using Least Square Method

19
Linear Regression using Least Square Method

20
Linear Regression using Least Square Method

21
Linear Regression using Least Square Method

22
Linear Regression using Least Square Method

Regression Line Equation is

y = mx + b

23
Linear Regression using Least Square Method

24
Linear Regression using Least Square Method

25
Linear Regression using Least Square Method

26
Linear Regression using Least Square Method

27
Linear Regression using Least Square Method

28
Linear Regression using Least Square Method

Find the value of b from is y = mx + b using

mean
b = 4 values
– 0.6*3 =
2.2
29
Linear Regression with Squared Loss
•

09-02-2026 Indian Institute of Information Technology, Nagpur 30

 Choice of loss function
Alternative loss functions usually depends on the
nature of the data. Also,
some loss functions result in
▪ Many possible loss functions for regression problems easier optimization problem
than others
Squared Absolute
loss Loss Loss
Very commonly loss
used for regression. Grows more slowly than
Leads to an squared loss. Thus better
easy-to-solve suited when data has
optimization some outliers (inputs on
problem which model makes large
errors)

Loss
Huber loss Loss

Note: Can also use

squared loss
instead of absolute
loss

09-02-2026 Indian Institute of Information Technology, Nagpur 31

Linear Regression Performance Evaluation

32
 Error Variable

The inclusion of the random error term allows (x,y)

to fall either above the true regression line (When E
> 0) or below the line (When E < 0) 33
 Performance Evaluation
• The performance of a regression model can be
understood by knowing the error rate of the
predictions made by the model.
• One can also measure the performance by knowing
how well your regression line fit the dataset.
• A good regression model is one where the difference
between the actual or observed values and predicted
values for the selected model is small and unbiased for
train, validation and test data sets.

09-02-2026 Indian Institute of Information Technology, Nagpur 34

 Performance Evaluation

09-02-2026 Indian Institute of Information Technology, Nagpur 35

 Mean Absolute Error(MAE)
This is the simplest of all the metrics. It is measured by taking
the average of the absolute difference between actual values
and the predictions.
• Less sensitive to outliers, making it suitable for financial data
• Provides a stable and interpretable error measure
• Directly reflects pricing deviation in the same scale as the target
variable

09-02-2026 Indian Institute of Information Technology, Nagpur 36

 Root Mean Square Error (RMSE)
• The Root Mean Square Error is measured by taking the
square root of the average of the squared difference
between the prediction and the actual value.
• It represents the sample standard deviation of the
differences between predicted values and observed
values(also called residuals).
• Penalizes larger errors more heavily
• Sensitive to large mispricing errors, useful for detecting
model instability

09-02-2026 Indian Institute of Information Technology, Nagpur 37

 MAE vs. RMSE
• Compared to MAE, RMSE gives a higher total error and
the gap increases as the errors become larger. It penalizes
a few large errors more than a lot of small errors. If you
want your model to avoid large errors, use RMSE over
MAE.
• Root Mean Square Error (RMSE) indicates the average
error in units of y, the predicted feature, but penalizes
larger errors more severely than MAE. A value of 0
indicates a perfect fit.
• You should also be aware that as the sample size increases,
the accumulation of slightly higher RMSEs than MAEs
means that the gap between these two measures also
09-02-2026 Indian Institute of Information Technology, Nagpur 38
Here,
– SSR = Sum Square of Residuals(the squared difference
between the predicted and the average value)
– SST = Sum Square of Total(the squared difference between the
actual
09-02-2026 and average value)
Indian Institute of Information Technology, Nagpur 39
• Range: 0 to 1 (can be negative in poor models)
• Higher R² → better model fit

Limitations:
• Does not measure prediction error
• Always increases with more features
• Can be misleading for overfitted models

40
41
42
43
44
• Modified form of R-Squared whose value increases if new
predictors tend to improve models performance and
decreases if new predictors does not improve
performance as expected.
• The Adjusted R-squared would penalize you for adding
features which are not useful for predicting the target.
• It takes into account the number of independent variables
used for predicting the target variable.
where,
N = number of records in the data set.
p = number of independent variables.
09-02-2026 Indian Institute of Information Technology, Nagpur 45
Gradient Descent in LR

46
Gradient Descent in LR

The linear relationship

between 2 variables can
be represented as a
straight line y = mx + b
“y” is target
“x” is feature
m & b are model
parameters

47
Gradient Descent in LR

48
Gradient Descent in LR
Depending on the values of m and b, multiple
possible lines can be possible.

We need to find out the value of parameters b and m

corresponding to which straight line fit best to the data.
49
Gradient Descent in LR

50
The Cost Function of Linear Regression

51
How to minimize the Cost Function?
• We have established the fact that all the straight
lines are just different combination of model
parameters b & m.
• Cost Function is the function of parameters b & m.
• Therefore, by changing the values of b & m we can
change the cost function.
• We will keep changing the values of b & m till we
find a combination where cost function is
minimized.
• To find the best combination we use Gradient
Descent Algorithm.

52
 Gradient Descent in LR
Gradient Descent Algorithm
Gradient Descent (GD) is an optimization algorithm used to minimize a cost (loss) function by iteratively
updating model parameters in the direction of the steepest decrease of the loss.

Core Idea
• Compute the gradient (partial derivatives) of the loss function
• Move parameters opposite to the gradient direction
• Repeat until convergence

53
Gradient Descent in LR

54
Gradient Descent in LR

55
Gradient Descent in LR

56
Gradient Descent in LR

57
Gradient Descent in LR

58
Gradient Descent in LR

59
 Gradient Descent in LR
Algorithm Steps
[Link] parameters randomly
[Link] predictions
[Link] loss
[Link] gradients
[Link] parameters
[Link] until:
a. loss converges, or
b. max iterations reached
60
Gradient Descent in LR

• These iteration of Gradient Descent algorithm can

run multiple times depending on nature of function
and 'α' (learning rate) and of course where we start
from, 'a1' in this case.

• Same methodology can be used to minimize cost

function ( J) which is a function of model
parameters b & m by changing them through
iterations of Gradient Descent Algorithm.

61
Gradient Descent in LR

Steps
1. Calculate slope at the current value of parameter b &
m separately
2. Take step α and update new parameters.
3. Calculate the cost function J with new (b & m)
values.

Repeat it multiple times.

62
 Types of Regression Analysis

09-02-2026 Indian Institute of Information Technology, Nagpur 63

 Multiple Regression

09-02-2026 Indian Institute of Information Technology, Nagpur 64

Logistic Regression

09-02-2026 Indian Institute of Information Technology, Nagpur 65

Polynomial Regression

09-02-2026 Indian Institute of Information Technology, Nagpur 66

Ridge and Lasso Regression

09-02-2026 Indian Institute of Information Technology, Nagpur 67

Multivariate Regression

09-02-2026 Indian Institute of Information Technology, Nagpur 68

Multiple Linear Regression

69
Multiple Linear Regression
• Multiple linear regression is an extension of simple
linear regression, which models the relationship
between a dependent variable and multiple
independent variables.
• In multiple linear regression, there are multiple
independent variables.

70
Multple Linear Regression

71
Multiple Linear Regression
• The general form of a multiple linear regression
model with k independent variables is given by:

Here,

72
How to use Multiple Regression Analysis?
The processes involved the selection of features,
engineering the features, feature normalization,
selection loss functions, hypothesis analysis, and
creating a regression model.

1. Selection of features:
It is the most important step in multiple regression.
Also known as variable selection, this process involves
selecting viable variables to build efficient models.

73
Feature Elimination

74
How to use Multiple Regression Analysis?
2. Feature Normalization: This involves feature scaling to
maintain streamlined distribution and data ratios. This
helps in better data analysis. The value of all the features
can be changed according to the requirement.

3. Selecting Loss function and hypothesis: The loss

function is used for predicting errors. The loss function
comes into play when the hypothesis prediction changes
from the actual figures. Here, the hypothesis represents the
value predicted from the feature or variable.

4. Fixing hypothesis parameter: The parameter of the

hypothesis is fixed or set in such a way that it minimizes
the loss function and enhances better prediction.
75
How to use Multiple Regression Analysis?
5. Reducing the loss function: The loss function is
minimized by generating an algorithm specifically for
loss minimization on the dataset which in turn
facilitates the alteration of hypothesis parameters.
Gradient descent is the most commonly used algorithm
for loss minimization.

6. Analyzing the hypothesis function: The function of the

hypothesis needs to be analyzed as it is crucial for
predicting the values. After the function is analyzed, it is
then tested on test data.

76
Assumptions in the Multiple Regression Model
• The dependent and the independent variables have
a linear relationship.
• The independent variables do not have a strong
correlation among themselves.
• The observations of yi are chosen randomly and
individually from the population.

77
Advantages of Multiple Regression

• It allows you to study the effect of several predictors

simultaneously, which is more realistic for most
real-world problems.
• By including relevant variables, the model often
explains more variance in the dependent variable,
leading to better predictions.
• The correlation between dependent and
independent variables helps in predicting the
outcome.
• Each regression coefficient shows the unique
contribution of an independent variable while
keeping others constant. 78
Disadvantages of Multiple Regression
• The complexity of multiple techniques requires
complex mathematical calculations.
• It is not easy to interpret the output of the multiple
regression model since there are inconsistencies in
the loss and error outputs.
• Multiple regression models cannot be applied to
smaller datasets; they are designed for producing
accurate outputs when it comes to larger datasets.

79
Multivariate Linear Regression

80
 MvLR Model: Scalar Form

The multivariate (multiple) linear regression model

has the form
p
yik =0kb + b Σx + jkeij ik
j=1

for i ∈ {1, . . . , n} and k ∈ {1, . . . , m} where

Nathaniel E. Helwig (U of Minnesota)

yik ∈ R is the k -th real-valued response for the i-th

observation
b0k ∈ R is the regression intercept for k -th response
bjk ∈ R is the j-th predictor’s regression slope for k -th
response
xij ∈ R is the j-th predictor for the i-th observation
(ei1, . . . , eim) iid N(0m, Σ) is a multivariate Gaussian error
vector
~
 MvLR Model: Nomenclature

The model is multivariate because we have m > 1 response

variables. The model is multiple because we have p > 1

predictors.
Nathaniel E. Helwig (U of Minnesota)

If p = 1, we have a multivariate simple linear regression

model

The model is linear because yik is a linear function of the

parameters (bjk are the parameters for j ∈ {1, . . . , p + 1} and k ∈
 MvLR Model: Assumptions

The fundamental assumptions of the MLR

model
1 are:
Relationship between X and Y is linear (given other
j k
2 predictors)
3 x
(eij,and
. . . ,yeik )are
iid observed random variables (known
N(0 , Σ) is an unobserved random
vector ~ m
constants) r
bki1= (bim , b , . . . , b ) for k ∈ {1, . . . , m} are unknown
Nathaniel E. Helwig (U of Minnesota)
4
0k 1k pk
5 constants

Note: bjk is expected increase in Yk for 1-unit increase in Xj

with all other predictor variables held constant
Regression Types - Summary
Type of Description Use Cases Strengths Limitations
Regression
Linear Models the Predicting Simple to Assumes
Regression relationship continuous implement; linearity;
between a variables like interpretable sensitive to
dependent variable house prices, coefficients; outliers; limited
and one/more sales, or salaries. works well with flexibility for
independent linear complex
variables as a relationships. relationships.
straight line.
Logistic Models the Binary Effective for Not suitable for
Regression probability of a classification binary multi-class
binary outcome tasks like spam outcomes; problems
(e.g., yes/no) based detection or probabilistic without
on independent disease interpretation; extension.
variables. diagnosis. relatively simple
09-02-2026 to implement.
Indian Institute of Information Technology, Nagpur 84
Regression Types - Summary
Type of Description Use Cases Strengths Limitations
Regression
Ridge Linear High-dimension Reduces Requires
Regression regression with al data, e.g., overfitting; hyperparameter
L2 regularization genetics or text balances tuning; not
to reduce classification. simplicity and suitable for
overfitting by complexity; sparse data.
penalizing large robust to
coefficients. multicollinearity.
Lasso Linear Feature selection Performs May eliminate
Regression regression with in automatic relevant
L1 high-dimensiona feature selection; variables;
regularization, l datasets like reduces sensitive to high
which performs genomics or overfitting; correlations
variable selection finance. interpretable. between
by shrinking predictors.
09-02-2026 coefficients. Indian Institute of Information Technology, Nagpur 85
LR Exercise 1
Q. Study the relationship between the monthly sales
and the advertising costs surveyed for different
stores as given below. Find the equation of the
straight line that fits the data best. Determine the
R-squared value.
Store Sales (units) Advertising Cost
1 368000 1700
2 340000 1500
3 665000 2800
4 954000 5000
5 331000 1300
6 556000 2200
7 376000 1300 86
LR Exercise 2
Car age in Price in
Q. Examine the relationship years Lakhs
between the age and price for 4 6.3
used cars sold in the last year
4 5.8
by a car dealership company.
5 5.7
Find the equation of the
5 4.5
straight line that fits the data
best. Determine the R-squared 7 4.5
value. 7 4.2
8 4.1
9 3.1
10 2.1
11 2.5
12 2.2
87