SIMPLE LINEAR REGRESSION

[Link], SCOPE, VIT CHENNAI

REGRESSOR TERMINOLOGY

[Link], SCOPE, VIT CHENNAI

We want a line that predicts 𝑦from 𝑥 (Mathematically, the linear
relationship is approximately modelled as)

𝑦ො = 𝛽0 + 𝛽1 𝑥
This line should be the best line - meaning the prediction errors
should be as small as possible.
[Link], SCOPE, VIT CHENNAI
 LINEAR REGRESSION
Input X – Independent variable: Input or predictor variable is the
variable that helps predict the value of the output variable. It is
commonly referred to as X.
Output Y – Dependent variable: The output or target variable is
the variable that we want to predict. It is commonly referred to
as y.

y = mx + c

[Link], SCOPE, VIT CHENNAI

Simple Linear Regression
• Simple linear regression is a very straightforward simple linear
approach for predicting a quantitative response y on the basis of a
single predictor variable x
• It assumes that there is approximately a linear relationship between x
and y

We want a line that predicts 𝑦from 𝑥 (Mathematically, the linear

relationship is approximately modelled as)

𝑦ො = 𝛽0 + 𝛽1 𝑥
This line should be the best line - meaning the prediction errors
should be as small as possible.
55
Simple Linear Regression
• For example, X may represent TV
advertising and Y may represent
sales.
• Then we can regress sales onto TV
by fitting the mode

66
Simple Linear Regression

• 0 - Intercept that fixes offset

to the line
• 1 - Slope that specifies the
factor by which X has an
impact on Y
• 0 and 1 – Model coefficients
or parameters

77
LINEAR REGRESSION

[Link], SCOPE, VIT CHENNAI

CLASSIFICATION OF REGRESSION

[Link], SCOPE, VIT CHENNAI

LOSS FUNCTION – FITTING THE MODEL
(TRAINING THE MODEL)
RSS – Residual Sum of Squares

[Link], SCOPE, VIT CHENNAI

 SIMPLE LINEAR REGRESSION
Modeling between the
dependent variable y and
one independent variable X.
When there is only one
independent variable X in the
linear regression model, the
model is generally termed as
simple linear regression
model.
When there are more than
one independent variables X1,
X2…. in the model, then the
linear model is termed as the
multiple linear regression
model.

[Link], SCOPE, VIT CHENNAI

[Link], SCOPE, VIT CHENNAI
Non-Linear Regression

[Link], SCOPE, VIT CHENNAI

DERIVATION – SIMPLE LINEAR REGRESSION
We want a line that predicts 𝑦from 𝑥:
𝑦ො = 𝛽0 + 𝛽1 𝑥
This line should be the best line — Prediction errors should be as small as
possible.
STEP 1: Define the error (residual)
For each data point 𝑥𝑖 𝑦𝑖 : error = 𝑦𝑖 − 𝛽0 + 𝛽1 𝑥𝑖
The errors should be small. But errors can be + or −, so we square them:

This is called the sum of squared errors.

Choose 𝛽0 and 𝛽1 so that 𝑆is as small as possible.
This is the basic idea of least squares.
[Link], SCOPE, VIT CHENNAI
DERIVATION – SIMPLE LINEAR
REGRESSION
STEP 2: Take derivatives to find minimum
Just like in calculus, minimum occurs when derivative = 0.
Derivative with respect to β0:
Σ (yi – β0 – β1 xi) = 0
Rearranging gives:
β0 = ȳ – β1 x̄
This means:
The regression line always passes through (x̄, ȳ).

[Link], SCOPE, VIT CHENNAI

DERIVATION – SIMPLE LINEAR
REGRESSION
STEP 3: Derivative with respect to β1
Σ xi (yi – β0 – β1 xi) = 0
Substitute β0 = ȳ – β1 x̄ into this equation.
After simplifying (students don’t need the messy algebra), we obtain:
Σ (xi – x̄)(yi – ȳ)
β1 = --------------------
Σ (xi – x̄)²
This is the formula for the slope.

Slope = how x and y move together / how much x varies

[Link], SCOPE, VIT CHENNAI

Simple Linear Regression
Estimating the coefficients 0 and 1

(SSE) Sum of squares regression (SSR)

is different (from residual)
Residual means error (actual - predicted)
SSE =
Residual sum of square = SSE = sum of squares error
17
17
Simple Linear Regression
Estimating the coefficients 0 and 1

18
18
 SIMPLE LINEAR REGRESSION
In the equation, you can see that two entities can have changeable
values (regression coefficients / variables), 𝛽0 which is the point at
which the line intercepts the x-axis, and 𝛽1 , which is how steep the line
will be, or slope.
At first, if these two regression coefficients are not properly
optimized, you get a line that might not properly fit the model.
As you optimize the values of the model, for some variables, you will
get the perfect fit.
The perfect fit will be a straight line running through most of the data
points while ignoring the noise and outliers.

Two phases in ML:

Training phase → Fit
Testing phase → Predict

[Link], SCOPE, VIT CHENNAI

 SIMPLE LINEAR REGRESSION
In the equation, you can see that two entities can have changeable
values (regression coefficients / variables), 𝛽0 which is the point at
which the line intercepts the x-axis, and 𝛽1 , which is how steep the line
will be, or slope.
At first, if these two regression coefficients are not properly
optimized, you get a line that might not properly fit the model.
As you optimize the values of the model, for some variables, you will
get the perfect fit.
The perfect fit will be a straight line running through most of the data
points while ignoring the noise and outliers.

Two phases in ML:

Training phase → Fit
Testing phase → Predict

[Link], SCOPE, VIT CHENNAI

 COST FUNCTION
For the Linear regression model, the cost function will be the minimum
of the Root Mean Squared Error of the model, obtained by
subtracting the predicted values from actual values. The cost function
will be the minimum of these error values.
ℎ𝜃 𝑥𝑖 = 𝑦ො𝑖
Independent
variable
𝑥𝑖
Model predicted target
𝑦𝑖
Original target y
if h = Model/Hypothesis
Theta is learnable
parameter
[Link], SCOPE, VIT CHENNAI
Independent
variable 𝑥𝑖
Model predicted target
𝑦𝑖
Original target y

[Link], SCOPE, VIT CHENNAI

Simple Linear Regression
• For the Advertising data, the
least squares fit for the
regression of sales onto TV is
shown
• The fit is found by minimizing
the sum of squared errors
• Each grey line segment
represents an error, and the
fit makes a compromise by
averaging their squares

23
23
Model Estimation and Evaluation
Question-1:
Consider the following five training examples
X = [2 3 4 5 6]
Y = [12.8978 17.7586 23.3192 28.3129 32.1351]
We want to learn a function f(x) of the form f(x) = ax + b which is
parameterized by (a, b). Find the best linear fit

24
24
Simple Linear Regression
Solution:

X Y (X-Xmean) (Y-Ymean) (X-Xmean)(Y-Ymean) (X-Xmean)2

2 12.8978 -2 -9.9869 19.9738 4
3 17.7586 -1 -5.1261 5.1261 1
4 23.3192 0 0.4345 0.0000 0
5 28.3129 1 5.4282 5.4282 1
6 32.1351 2 9.2504 18.5008 4 The best linear fit is
Sum 20 114.4236 0 0.0000 49.0289 10
Y = 3.2732 + 4.9029X
Mean 4 22.88472
Substituting in
the formula

25
25
Simple Linear Regression
Matrix Method
Numerical Example

26
Matrix approach for solving Regression numericals

Basic matrix representation to find the regression

coefficient

Augment for w0
regression
coefficient

27
27
Matrix approach for solving Regression numericals

28
28
Matrix approach for solving Regression numericals

29
29
[Link], SCOPE, VIT CHENNAI
Matrix approach for solving Regression numericals

31
31
[Link], SCOPE, VIT CHENNAI
SIMPLE LINEAR REGRESSION (3 CASES)

[Link], SCOPE, VIT CHENNAI

[Link], SCOPE, VIT CHENNAI
Correlation
➢ The correlation coefficient r is given by

➢ This will always be a number between -1 and 1 (inclusive).

35
35
QUIZ
List out real time examples of all three cases
Create different dataset for X and Y for all three cases

[Link], SCOPE, VIT CHENNAI

CASE 1: CORRELATION & SLOPE < 0

[Link], SCOPE, VIT CHENNAI

CASE 2: CORRELATION & SLOPE = 0

[Link], SCOPE, VIT CHENNAI

CASE 3: CORRELATION & SLOPE > 0

[Link], SCOPE, VIT CHENNAI

IDENTIFY

[Link], SCOPE, VIT CHENNAI

Simple Linear Regression
Ordinary Method
Numerical Example

41
Simple Linear Regression
Question-2:
Consider the following five training examples
X = [2 3 4 5 6]
Y = [12.8978 17.7586 23.3192 28.3129 32.1351]
(a)Find the best linear fit
(b)Determine the minimum SSE
(c) Draw the residual plot for the best linear fit and comment on the
suitability of the linear model to this training data.

42
42
Simple Linear Regression
Solution:
(a) To find the best fit, calculate the model coefficients using the formula

43
43
Simple Linear Regression
Solution:

X Y (X-Xmean) (Y-Ymean) (X-Xmean)(Y-Ymean) (X-Xmean)2

2 12.8978 -2 -9.9869 19.9738 4
3 17.7586 -1 -5.1261 5.1261 1
4 23.3192 0 0.4345 0.0000 0
5 28.3129 1 5.4282 5.4282 1
6 32.1351 2 9.2504 18.5008 4 The best linear fit is
Sum 20 114.4236 0 0.0000 49.0289 10
Y = 3.2732 + 4.9029 X
Mean 4 22.88472
Substituting in
the formula

44
44
Simple Linear Regression
Solution:

Best Linear Fit

45
45
Simple Linear Regression
Solution:
(b) To determine RSS

X Y (X-Xmean) (Y-Ymean) (X-Xmean)(Y-Ymean) (X-Xmean)2 Ypredicted (Y-YPredicted)2

2 12.8978 -2 -9.9869 19.9738 4 13.0789 0.0328
3 17.7586 -1 -5.1261 5.1261 1 17.9818 0.0498
4 23.3192 0 0.4345 0.0000 0 22.8847 0.1888
5 28.3129 1 5.4282 5.4282 1 27.7876 0.2759
6 32.1351 2 9.2504 18.5008 4 32.6905 0.3085
Sum 20 114.4236 0 0.0000 49.0289 10 SSE 0.8558
Mean 4 22.88472

Y predicted is calculated using the best linear fit

Y = 3.2732 + 4.9029 X 46
46

SSE𝒎𝒊𝒏
SSE𝒎𝒊𝒏 = 0.8558
Simple Linear Regression
Solution: Residual Plot
(c) Residual plot for the best linear fit
Residual
X Y Ypredicted (Y-YPredicted)
2 12.8978 13.0789 -0.1811
3 17.7586 17.9818 -0.2232
4 23.3192 22.8847 0.4345
5 28.3129 27.7876 0.5253
6 32.1351 32.6905 -0.5554

The random pattern in it is an indication that a linear model is suitable for this data
47
47
Simple Linear Regression
Question:
Consider the following five training examples
X = [2 3 4 5 6]
Y = [12.8978 17.7586 23.3192 28.3129 32.1351]
We want to learn a function f(x) of the form f(x) = ax + b which is
parameterized by (a, b). Using squared error as the loss function, which of
the following parameters would you use to model this function.
(a) (4 3)
(b) (5 3)
(c) (5 1)
(d) (1 5) 48
48
Simple Linear Regression
Solution:
1. Calculate Ypredicted for the given X using the given (a, b) values
2. For each (a, b) value, calculate the SSE
3. The best set of parameters is the one that gives minimum SSE

To calculate RSS, use the following formula

SSE

where
49
49
Simple Linear Regression
Solution:
There are four options given.
1. Lets start with first option, for a = 4 and b = 3
2. Calculate Ypredicted for the given X using the given (a, b) values
f(x) = Ypredicted = aX + b
Y1predicted = aX1 + b = 4(2)+3 =11
3. Calculate ( Y1 - Y1predicted)2
( Y1 - Y1predicted)2 = (12.8978 – 11)2 = 3.6016

X Y Ypredicted (Y-YPredicted)2
2 12.8978 11 3.6016
3 17.7586 15 7.6099
4 23.3192 19 18.6555
5 28.3129 23 28.2269
6 32.1351 27 26.3693 50
50

SSE = 84.4632
Simple Linear Regression
Solution:
There are four options given.
Lets start with first option, for a = 4 and b = 3

X Y Ypredicted (Y-YPredicted)2
2 12.8978 11 3.6016
3 17.7586 15 7.6099
4 23.3192 19 18.6555
5 28.3129 23 28.2269
6 32.1351 27 26.3693
SSE = 84.4632

Formula 51
51

SSE
sum of squares error
Simple Linear Regression
Solution:
There are four options given.
Lets start with first option, for a = 4 and b = 3

X Y Ypredicted (Y-YPredicted)2
2 12.8978 11 3.6016
3 17.7586 15 7.6099
4 23.3192 19 18.6555
5 28.3129 23 28.2269
6 32.1351 27 26.3693
SSE = 84.4632

Formula 52
52

SSE

sum of squares error

Simple Linear Regression
Solution: Formula
The give second option,
SSE
For a = 5 and b = 3
sum of squares error
X Y Ypredicted (Y-YPredicted)2
2 12.8978 13 0.0104
3 17.7586 18 0.0583
4 23.3192 23 0.1019
5 28.3129 28 0.0979
6 32.1351 33 0.7481
SSE= 1.0166

53
53
Simple Linear Regression
Solution: Formula
The given third option, SSE
For a = 5 and b = 1
sum of squares error

X Y Ypredicted (Y-YPredicted)2
2 12.8978 11 3.6016
3 17.7586 16 3.0927
4 23.3192 21 5.3787
5 28.3129 26 5.3495
6 32.1351 31 1.2885
SSE= 18.7110

54
54
Simple Linear Regression
Solution: Formula
The given fourth option,
SSE
For a = 1 and b = 5
sum of squares error
X Y Ypredicted (Y-YPredicted
)2
2 12.8978 7 34.7840
3 17.7586 8 95.2303
4 23.3192 9 205.0395
5 28.3129 10 335.3623
6 32.1351 11 446.6925
SSE= 1117.1086

55
55
 SSE
(a) (4 3) 84.4632
(b) (5 3) 1.0166
(c) (5 1) 18.7110
(d) (1 5) 1117.1086

Answer: The parameter (5,3) which gives least SSE (1.016). Hence
(5,3) second option is optimally used to model this function
[Link], SCOPE, VIT CHENNAI
 Correlation
Numerical Example

57
Correlation
• A correlation is a relationship between two variables.
• Is there a relationship between the number of employee training
hours and the number of jobs produced?
• Is there a relationship between the number of hours a student spends
studying for a Mathematics test and the student’s score on that test?
• Let x to be the independent variable and y to be the dependent
variable. Data is represented by a collection of ordered pairs (x, y)
• Mathematically, the strength and direction of a linear relationship
between two variables is represented by the correlation coefficient.

58
58
Correlation
➢ The correlation coefficient r is given by

➢ This will always be a number between -1 and 1 (inclusive).

59
59
Correlation
Question:
➢ The time x in years that an employee spent at a company and the
employee’s hourly pay, y, for 5 employees are listed in the table below.
Calculate and interpret the correlation coefficient r

60
60
Correlation

61
61
Correlation

62
62
Correlation

63
63
Correlation

1
2
3
4
5

64
64
Correlation

65
65
Correlation

• Interpret this result: There is a strong positive correlation

between the number of years and employee has worked and the
employee’s salary, since r is very close to 1

66
66
[Link], SCOPE, VIT CHENNAI
 R2 metric / score
Numerical Example

68
[Link], SCOPE, VIT CHENNAI
Model Estimation and Evaluation

SST = SSR + SSE.

SST → Total sum of squared deviations 70
70

SSR → Sum of squares due to regression

SSE → Sum of squared residual errors
Simple Linear Regression

Original target deviation

Predicted target deviation

Error/Difference between
Original and Predicted
target

71
71
Simple Linear Regression

72
72
Model Estimation and Evaluation

73
73
Regression Modelling

74
Model Estimation and Evaluation
• We assume that the true relationship between X and Y takes the form
Y = f(X) +  for some unknown function f, where  is a mean-zero
random error term
• If f is to be approximated by a linear function, then we can write this
relationship as

✓ β0 is the intercept, that is the expected value of Y when X = 0

✓ β1 is the slope—the average increase in Y associated with a one-unit
increase in X
✓  the error term is independent of X

75
75
Model Estimation and Evaluation

• This model defines the population

regression line which is the best
linear approximation to the true
relationship between X and Y

• Population mean = μ which is

unknown

76
76
Model Estimation and Evaluation

• This model defines the least

squares line estimated from least
square coefficients
• In real applications, set of
observations is used to compute the
least squares line
• Sample mean =  ෝ
• The sample mean and the
population mean are different, but
in general the sample mean will
provide a good estimate of the
population mean
77
77
Model Estimation and Evaluation
• Figure shows a simulated data set
• The red line represents the true
relationship, f(X)=2+3X, which is
known as the population
regression line
• The blue line is the least squares
line; it is the least squares estimate
for f(X) based on the observed data,
shown in black

78
78
Model Estimation and Evaluation
• Figure shows a simulated data set
• The population regression line is
shown in red, and the least squares
line in dark blue
• In light blue, ten least squares lines
are shown, each computed on the
basis of a separate random set of
observations from f(X)=2+3X + 
• Each least squares line is different,
but on average, the least squares
lines are quite close to the
population regression line
79
79
Model Estimation and Evaluation

• The standard error tells us the average amount that this estimate ෝ differs
from the actual value of . The standard error equation tells us how this
deviation shrinks with n – the more observations we have, the smaller the
standard error of 
ෝ 80
80
Model Estimation and Evaluation

Standard error

SSE sum of squares error

Residual standard error
81
81
Model Estimation and Evaluation
• Standard errors can be used to compute confidence intervals
• A 95% confidence interval is defined as a range of values such that
with 95% interval probability, the range will contain the true
unknown value of the parameter
• The range is defined in terms of lower and upper limits computed
from the sample of data

82
82
Model Estimation and Evaluation
• For linear regression, the 95% confidence interval for β0
approximately takes the form
Standard error

• That is, there is approximately a 95 % chance that the interval

will contain the true value of β0

83
83
Model Estimation and Evaluation
• Similarly, a confidence interval for β1 approximately takes the form
Standard error

will contain the true value of β1

• The word ‘approximately’ is included mainly because

✓ The errors are assumed to be Gaussian and
✓ The factor ‘2’ in front of term will vary slightly depending on
the number of observations ‘n’ in the linear regression

84
84
Model Estimation and Evaluation

85
85
Model Estimation and Evaluation
• The RSE provides an absolute measure of lack of fit of the model to the
data. A small RSE indicates that the model fits the data well whereas a
large RSE indicates that the model doesn’t fit the data well. But since it
is measured in the units of Y, it is not always clear what constitutes a
good RSE
• The R2 statistic provides an alternative measure of fit. It takes the
form of a proportion of variance, expressed as
SSE

• Note that R2 statistic is independent of the scale of Y, and it always

takes a value between 0 and 1 86
86
Model Estimation and Evaluation

TSS - SSE

• An R2 statistic that is close to 1 indicates that a large proportion of the

variability in the response has been explained by the regression. A
number close to 0 indicates that the regression did not explain much
of the variability in the response; this might occur because the linear
model is wrong, or the inherent error σ2 is high, or both.
• The R2 statistic is also a measure of the linear relationship between X
and Y and it is closely related to correlation between X and Y 87
87
Model Estimation and Evaluation
Question-3:
Consider the following five training examples
X = [2 3 4 5 6]
Y = [12.8978 17.7586 23.3192 28.3129 32.1351]
We want to learn a function f(x) of the form f(x) = ax + b which is
parameterized by (a, b).
(a)Find the best linear fit
(b)Evaluate the standard errors associated with
(c) Determine the 95% confidence interval for a and b
(d)Compute R2 statistic
88
88
Simple Linear Regression
Solution:

X Y (X-Xmean) (Y-Ymean) (X-Xmean)(Y-Ymean) (X-Xmean)2

2 12.8978 -2 -9.9869 19.9738 4
3 17.7586 -1 -5.1261 5.1261 1
4 23.3192 0 0.4345 0.0000 0
5 28.3129 1 5.4282 5.4282 1
6 32.1351 2 9.2504 18.5008 4 The best linear fit is
Sum 20 114.4236 0 0.0000 49.0289 10
Y = 3.2732 + 4.9029X
Mean 4 22.88472
Substituting in
the formula

89
89
Simple Linear Regression
Solution:

X Y (X-Xmean) (Y-Ymean) (X-Xmean)(Y-Ymean) (X-Xmean)2 Ypredicted (Y-YPredicted)2

2 12.8978 -2 -9.9869 19.9738 4 13.0789 0.0328
3 17.7586 -1 -5.1261 5.1261 1 17.9818 0.0498
4 23.3192 0 0.4345 0.0000 0 22.8847 0.1888
5 28.3129 1 5.4282 5.4282 1 27.7876 0.2759
6 32.1351 2 9.2504 18.5008 4 32.6905 0.3085
Sum 20 114.4236 0 0.0000 49.0289 10 SSE 0.8558
Mean 4 22.88472

Y predicted is calculated using the best linear fit SSE𝒎𝒊𝒏 = 0.8558

Y = 4.9029 + 3.2732 X 90
90
Model Estimation and Evaluation

SSE  = SSE

Substituting SS E= 0.8558 and n = 5, then RSE = 0.5341.

Standard error for a is

SE(a) = 0.1689

Standard error for b is

SE(b) = 0.7186

91
91
Model Estimation and Evaluation

95% confidence interval for standard error for a is

[a – 2 SE(a) , a + 2 SE(a)] = [4.5651, 5.2407]

95% confidence interval for standard error for b is

[b – 2 SE(b) , b + 2 SE(b)] = [1.8400, 4.7063]

92
92
Model Estimation and Evaluation

X Y (Y-Ymean)2 To find R2 value, first find TSS

2 12.8978 99.73857
= 241.2391
3 17.7586 26.27711
SSE
4 23.3192 0.188773 = 0.9965
5 28.3129 29.46514
6 32.1351 85.56953
Sum 20 114.4236 241.2391
Mean 4 22.88472

93
93
PRACTICE PROBLEMS
[Link], SCOPE, VIT CHENNAI
Also calculate the R2 score metric for the linear regression model

[Link], SCOPE, VIT CHENNAI

 EXERCISE
A psychologists claims that the length
of deep sleep in children diminishes
linearly with increasing daily TV time.
He observes TV time and deep sleep
time per day for 9 children and wants
to calculate the linear regression line
and regression coefficients. Also
calculate the R2score.

[Link], SCOPE, VIT CHENNAI

[Link], SCOPE, VIT CHENNAI
4. K-Fold Cross Validation

[Link], SCOPE, VIT CHENNAI

K-FOLD CROSS VALIDATION

1. Cross-validation is applied to the training set, since selecting the shrinkage

parameter is part of the training process.
2. The test set is there to judge the performance of the selected model.

[Link], SCOPE, VIT CHENNAI

K-fold cross validation

[Link], SCOPE, VIT CHENNAI

[Link], SCOPE, VIT CHENNAI
[Link], SCOPE, VIT CHENNAI
[Link], SCOPE, VIT CHENNAI
[Link], SCOPE, VIT CHENNAI
[Link], SCOPE, VIT CHENNAI
REGULARIZATION STRATEGIES / SHRINKAGE
METHODS
REGULARIZATION

“In the context of machine learning, most regularization

strategies are based on regularizing estimators.
Regularization of an estimator works by trading
increased bias for reduced variance.
An effective regularizer is one that makes a profitable
trade, reducing variance significantly while not overly
increasing the bias.”
REGULARIZATION / SHRINKAGE METHOD

The bias error is an error from wrong assumptions in

the learning algorithm. High bias can cause an algorithm
to miss the relevant relations between features and
target outputs. This is called underfitting.
The variance is an error from sensitivity to small
fluctuations in the training set. High variance may result
in modeling the random noise in the training data. This
is called overfitting.
The bias-variance tradeoff is a term to describe the
fact that we can reduce the variance by increasing the
bias. Good regularization techniques strive to
simultaneously minimize the two sources of error.
Hence, achieving better generalization.
DISCLAIMER
Blue Dots → Training samples
Green Dots → Testing samples

[Link], SCOPE, VIT CHENNAI

[Link], ASP, SCOPE, VIT
[Link], ASP, SCOPE, VIT
[Link], SCOPE, VIT CHENNAI
[Link], SCOPE, VIT CHENNAI
REGRESSION MODEL → HOW IT UPDATED
THE REGRESSION/WEIGHT COEFFICIENTS?

[Link], SCOPE, VIT CHENNAI

REGRESSION MODEL VS SHRINKAGE
Loss function of Regression model:
Objective of regression model is to continue training process until the SSE
is reduced to a certain extent.

Loss function =

Modified Loss function of Shrinkage Regression model:

The expression that is added to SSE (Sum of squared errors) is called
the shrinkage quantity. The lambda(Alpha) symbol is the tuning
parameter. This modified loss function can now estimate the coefficients.
The tuning parameter decides how much we should penalize our model.

Modified Loss function =

TESTING ON ML ALGORITHMS
Underfitting
Overfitting
Bias (Error obtained during training phase)
Variance (Error obtained during testing phase)
TESTING ON ML ALGORITHMS
Underfitting
Overfitting
Bias
Variance

Low Variance
High Variance
[Link], SCOPE, VIT CHENNAI
Training Testing
Training Testing Error
Error Testing Error Training
Error Error Error

[Link], SCOPE, VIT CHENNAI

UNDERFITTING VS OVERFITTING
 BIAS AND VARIANCE
What is Bias?
Bias is the difference between the average prediction of our model and
the correct value which we are trying to predict. A model with high bias
pays very little attention to the training data and oversimplifies the model.
Simple definition: “Resulted Error from Training Data!”

What is a Variance?
Variance is the variability of model prediction for a given data point or a
value that tells us the spread of our data. A model with high variance pays
a lot of attention to training data and does not generalize on the data
which it hasn’t seen before.
Simple definition: “Resulted Error from Test Data!”
 BIAS VARIANCE TRADEOFF
▪If you reduce bias, variance
usually increases
Reduce Bias (Underfitting)
▪If you reduce variance, bias •Use more complex models
usually increases •Add more features
▪Goal: Find a balance where •Reduce regularization
total prediction error is Reduce Variance (Overfitting)
minimum. •Use simpler models
•Get more training data
•Apply regularization (L1/L2, dropout)
•Use cross-validation

Refer to :
[Link]
[Link], SCOPE, VIT CHENNAI
BAIS VS VARIANCE
“Most Generalized Model”.
[Link], SCOPE, VIT CHENNAI
 BIAS AND VARIANCE

MSE = X[With bias] MSE = 0[No bias] – Overfitting model

Testing phase

MSE = X[Less Variance] MSE = High [Link],

[ Variance is large]
SCOPE, VIT CHENNAI
[Link], SCOPE, VIT CHENNAI
 TYPES OF REGULARIZATION λ → regularization strength
𝛽𝑗 → model coefficients

1. Lasso Regularization

Lasso forces some coefficients to become exactly zero.

• When dataset has many irrelevant features

• When feature selection is needed
• When interpretability is important
[Link], SCOPE, VIT CHENNAI
TYPES OF REGULARIZATION λ → regularization strength
𝛽𝑗 → model coefficients

2. Ridge Regularization

Ridge regression shrinks coefficients, but never makes them exactly zero.

• When all features are important

• When features are correlated
• When multicollinearity exists [Link], SCOPE, VIT CHENNAI
MULTICOLLINEARITY

[Link], SCOPE, VIT CHENNAI

WHAT IS MULTICOLLINEARITY?
Multicollinearity means:
Two or more input features give almost the same information
They are highly correlated
Imagine predicting house price using:
Feature 1: Area ([Link])
Feature 2: Number of rooms
These two are highly related.

[Link], SCOPE, VIT CHENNAI

VARIATION INFLATION FACTOR

[Link], SCOPE, VIT CHENNAI

VARIATION INFLATION FACTOR
Variance Inflation Factor (VIF) quantifies how much the variance of a regression
coefficient is inflated due to multicollinearity among predictors.

VIF Value Meaning

1 No correlation
Moderate correlation
1–5
(acceptable)

5 – 10 High correlation (problematic)

> 10 Severe multicollinearity

[Link], SCOPE, VIT CHENNAI

Ridge Regression Works By Increasing The Bias To Improve Variance.

This Works By Changing The Slope Of The Line.

The Model Performance Might Be Little Poor On The Training Set But It Will Perform
Consistently Well On Both The Training And Testing Dataset.

Slope Has Been Reduced With Ridge Regression Penalty(Α)

And Therefore The Model Becomes Less Sensitive To Change In The Independent Variable

Least Square Regression

Min(Sum Of Squared Residuals)

Ridge Regression

Min(Sum Of Squared Residuals +𝞪*Slope2 )

 CHARACTERISTICS OF THE
PENALIZED TERM (𝞪 ALPHA / LAMDA)
As Alpha Increases, The Slope Of The Regression Line Is Reduced And
Becomes More Horizontal.

As Alpha Increases, The Model Becomes Less Sensitive To The Variation Of

The Independent Variable (Here It Is “Years Of Experience” In Our Example).
LASSO / L1 NORM PARAMETER
REGULARIZATION

[Link], ASP, SCOPE, VIT

RIDGE / L2 NORM PARAMETER
REGULARIZATION

[Link], ASP, SCOPE, VIT

RIDGE REGRESSION

[Link], SCOPE, VIT CHENNAI

USE OF LAMBDA Case 1: 𝜆 = 1

VARIABLE

Case 3: 𝜆 = 3
Case 2: 𝜆 = 2

[Link], SCOPE, VIT CHENNAI

RIDGE REGRESSION
Without ridge regression
Error in testing phase is high
 RIDGE REGRESSION
Ridge regression provides better
long term predictions…. How?

With ridge regression

With adding 𝜆 𝑤𝑒 Introduce a small bias Error in testing phase is
thereby reduce the variance somewhat low
RIDGE REGRESSION

[Link], SCOPE, VIT CHENNAI

 How to determine Lamda value?
LASSO REGRESSION Lamda value can be determined by
cross validation

Ridge regression and Lasso

regression looks very similar

[Link], SCOPE, VIT CHENNAI

LASSO REGRESSION

[Link], SCOPE, VIT CHENNAI

SUMMARY

[Link], SCOPE, VIT CHENNAI

LINEAR REGRESSION (X1, X2 →B0,B1,B2)

[Link], SCOPE, VIT CHENNAI

LASSO REGRESSION (X1, X2 →B0,B1,B2)
Coefficient → shrinkage effect

[Link], SCOPE, VIT CHENNAI

 RIDGE REGRESSION (X1, X2 →B0,B1,B2)
Coefficient → No shrinkage effect

[Link], SCOPE, VIT CHENNAI

ELASTIC NET
Elastic Net = Ridge + Lasso, used when features are highly
correlated and feature selection is needed.

[Link], SCOPE, VIT CHENNAI

[Link], SCOPE, VIT CHENNAI