.

IN
S. J. P. N. TRUST’S
HIRASUGAR INSTITUTE OF TECHNOLOGY, NIDASOSHI

C
Accredited at 'A+' Grade by NAAC
Programmes Accredited by NBA: CSE & ECE

N
Department of Computer Science & Engineering

SY
U
VT

Module 3:
Chapter 5: Regression Analysis
Dr. S. V. Manjaragi
Asso. Prof. , Dept. of Computer Science & Engg.,
Hirasugar Institute of Technology, Nidasoshi
6/2/2025
 .IN
Introduction to Regression

C
• Regression analysis is the premier method of supervised

N
learning.

SY
• This is one of the most popular and oldest supervised learning
technique.

U
• Given a training dataset D containing N training points (xi, yi),
VT
where i=1...N, regression analysis is used to model the
relationship between one or more independent variables xi;
and a dependent variable yi.
• The relationship between the dependent and independent
variables can be represented as a function as follows: y=f(x).
• Here, the feature variable x is also known as an explanatory
variable, exploratory variable, a predictor variable, an
independent variable, a covariate, or a domain point.
• y is a dependent variable.
• Dependent variables are also called as labels, target variables,
or response variables.
 .IN
Introduction to Regression

C
• Regression analysis determines the change in

N
response variables when one exploration variable is

SY
varied while keeping all other parameters constant.
• This is used to determine the relationship each of the
U
VT
exploratory variables exhibits. Thus, regression
analysis is used for prediction and forecasting.
• Regression is used to predict continuous variables or
quantitative variables such as price and revenue.
 .IN
Introduction to Regression

C
• Thus, the primary concern of regression analysis is to find

N
answer to questions such as:

SY
1. What is the relationship between the variables?
2. What is the strength of the relationships?

U
3. What is the nature of the relationship such as linear or non-
VT
linear?
4. What is the relevance of the attributes?
5. What is the contribution of each attribute?
• There are many applications of regression analysis. Some
of the applications of regressions include predicting:
1. Sales of a goods or services
2. Value of bonds in portfolio management
3. Premium on insurance companies
4. Yield of crops in agriculture
5. Prices of real estate
 .IN
C
N
• The quality of the regression analysis is

SY
determined by the factors such as correlation
U
and causation(Interconnection/relationship.
VT
 .IN
Regression and Correlation
• Correlation among two variables can be done effectively

C
using a Scatter plot, which is a plot between explanatory

N
variables and response variables.

SY
• It is a 2D graph showing the relationship between two
variables. The x-axis of the scatter plot is independent, or
U
input or predictor variables and y-axis of the scatter plot is
output or dependent or predicted variables.
VT
• The scatter plot is useful in exploring data.
• The Pearson correlation coefficient is the most common
test for determining correlation if there is an association
between two variables. The correlation coefficient is
denoted by r.
– In positive correlation, one variable change is associated with
the change in another variable.
– In negative correlation, the relationship between the variables
is reciprocal.
– In random correlation, no relationship exists between variables.
 .IN
Regression and Correlation

C
N
SY
U
VT
 .IN
Regression and Causation

C
N
• Causation is about causal relationship among
variables, say x and y.

SY
• Causation means knowing whether x causes y to
U
happen or vice versa. x causes y is often denoted as x
implies y. Correlation and Regression relationships are
VT
not same as causation relationship.
• For example, the correlation between economical
background and marks scored does not imply that
economic background causes high marks.
• Similarly, the relationship between higher sales of cool
drinks due to a rise in temperature is not a causal
relation. Even though high temperature is the cause of
cool drinks sales, it depends on other factors too.
 .IN
Linearity and Non-linearity Relationships

C
N
• The linearity relationship between the variables

SY
means the relationship between the dependent
and independent variables can be visualized as a
U
straight line.
VT
• The line of the form, y = ax + b
• can be fitted to the data points that indicate the
relationship between x and y.
• By linearity, it is meant that as one variable
increases, the corresponding variable also
increases in a linear manner.
 .IN
Linearity and Non-linearity Relationships

C
N
SY
U
VT
 .IN
Types of Regression Methods

C
N
SY
U
VT
 .IN
Types of Regression Methods
• Linear Regression It is a type of regression where a line is fitted upon

C
given data for finding the linear relationship between one independent

N
variable and one dependent variable to describe relationships.
• Multiple Regression It is a type of regression where a line, is fitted for

SY
finding the linear relationship between two or more independent
variables and one dependent variable to describe relationships among
variables.
U
• Polynomial Regression It is a type of non-linear regression method of
VT
describing relation ships among variables where N th degree
polynomial is used to model the relationship between one
independent variable and one dependent variable. Polynomial multiple
regression is used to model two or more independent variables and one
dependant variable.
• Logistic Regression It is used for predicting categorical variables that
involve one or more independent variables and one dependent
variable. This is also known as a binary classifier.
• Lasso and Ridge Regression Methods These are special variants of
regression method where regularization methods are used to limit the
number and size of coefficients of the independent variables.
 .IN
Limitations of Regression Method

C
1. Outliers — Outliers are abnormal data. It can bias the

N
outcome of the regression model, as outliers push the

SY
regression line towards it.
2. Number of cases — The ratio of independent and
U
dependent variables should be at least 20 : 1.
VT
3. For every explanatory variable, there should be at least 20
samples. At least five samples are required in extreme
cases.
4. Missing data — Missing data in training data can make
the model unfit for the sampled data.
5. Multicollinearity — If exploratory variables are highly
correlated (0.9 and above), the regression is vulnerable to
bias. Singularity leads to perfect correlation of 1.
 LINEAR REGRESSION

.IN
• the linear regression model can be created by fitting a line

C
among the scattered data points. The line is of the form given

N
in Eq. (5.2).

SY
• Here, ao is the intercept which represents the bias and a1
U
represents the slope of the line. These are called regression
VT
coefficients. e is the error in prediction.
• The assumptions of linear regression are listed as follows:
– The observations (y) are random and are mutually independent.
– The difference between the predicted and true values is called
an error.
– The distribution of the error term is independent of the joint
distribution of explanatory variables.
– The unknown parameters of the regression models are
constants.
 LINEAR REGRESSION

.IN
• The idea of linear regression is based on Ordinary Least Square (OLS)
approach.

C
• In this method, the data points are modelled using a straight line.

N
SY
U
VT

• In Figure 5.4, three data points and their errors (e1, e2, e3) are shown.
• The vertical distance between each point and the line predicted by the
approximate line equation y = a0 + a1x is called an error.
• These individual errors are added to compute the total error of the
predicted line. This is called sum of residuals.
• The squares of the individual errors can also be computed and added to
give a sum of squared error.
• The line with the lowest sum of squared error is called line of best fit.
 LINEAR REGRESSION

.IN
• Mathematically, based on Eq. (5.2), the line equations for

C
points (x1, x2, ..., xn) are:

N
SY
U
VT

• In general, the error is given as

.IN
• A regression line is the line of best fit for which the sum of the

C
squares of residuals is minimum.

N
• The minimization can be done as minimization of individual

SY
errors by finding the parameters ao and a1, such that:

U
VT

• Therefore, linear regression is modelled as a minimization

.IN
C
• Therefore, linear regression is modelled as a minimization

N
function as follows:

SY
U
VT