Module 3

Outline
• Linear regression
• Polynomial Regression
• Stepwise Regression
• Ridge regression
• Lasso regression
• ElasticNet Regression
Linear Regression
• Supervised learning has two types:
• Classification: It predicts the class of the dataset based on the independent
input variable. Class is the categorical or discrete values. like the image of an
animal is a cat or dog?
• Regression: It predicts the continuous output variables based on the
independent input variable. like the prediction of house prices based on
different parameters like house age, distance from the main road, location,
area, etc.
Linear Regression
• a supervised machine-learning algorithm that learns
• from the labelled datasets and maps the data points to the most optimized
linear functions,
• which can be used for prediction on new datasets.
• When there is only one independent feature, it is known as Simple
Linear Regression, and when there are more than one feature, it is
known as Multiple Linear Regression.
Importance
• Simple
• Easy to implement
• Transparent
• Foundational concept for more complex algorithms
Simple Linear Regression
Multiple Linear Regression
Linear Regression
• The goal of the algorithm is to find the best Fit Line equation that can
predict the values based on the independent variables.
Statistical Learning – Advertising Data

 The Advertising data set consists of the

sales of a product in 200 different cities,
along with advertising budgets for three
different media: TV, radio, and
newspaper

9
 The Advertising data set consists of the
sales of a product in 200 different cities,
along with advertising budgets for three
different media: TV, Radio, and
newspaper

10
 The Advertising data set consists of the sales
of a product in 200 different cities, along
with advertising budgets for three different
media: TV, Radio, and newspaper
 Goal is to develop an accurate model that can
be used to predict sales on the basis of the
three media budgets

11
• Input Variables: Advertising budgets
• Input Variables are denoted by X
• X1 – TV budget
• X2 – Radio budget
• X3 – Newspaper budget
• Input variables are called by different
names like
• Predictors
• Independent variables
• Features
• Variables

12
12
• Output Variable: Sales
• Output Variables are denoted by Y
• Output variables are called by
different names like
• Responses,
• Dependent variables

13
 There is some relationship between Y and X = (X1,
X2,...,Xp)
 General form of relationship is
 Y = f(X) + 
 where
 f is some fixed but unknown function of
X1,...,Xp
  is a random error term, which is
independent of X and has mean zero

14
 • The black lines represent the error associated with
each observation.
• Here some errors are positive (if an observation lies
above the blue curve) and some are negative (if an
observation lies below the curve)
• Overall, these errors have approximately mean
zero

Observed values of
income and years of
education for 30
individuals
15
 Statistical Learning refers to a set of approaches for
estimating f in the equation
• Y = f(X) + 
 Reasons to estimate ‘f’,:
 Prediction
 Inference

16
• Linear Regression is a statistical procedure that determines the equation for the straight line that best fits a
specific set of data.
• In particular, it is a useful tool for predicting a quantitative response.

17
On the basis of given advertising data,
• Marketing plan for next year can be made
• To develop the marketing plan, some information is
required.
• Is there a relationship between advertising
budget and sales?
• Is the relationship linear?
• Predicting sales with a high level of accuracy
requires a strong relationship.

18
18
 The important questions are
 Which media contribute more to sales?
 Do all three contribute to sales, or do just one or
two.
 The individual effects of each medium on the
money spent
 For every dollar spent on advertising in TV or
Radio or Newspaper, by what amount will sales
increase?
 How accurately can we predict this amount of
increase?

Linear regression can be used to answer each of these questions

19
19
Linear Regression - Types
• Types:
 Based on the number of independent variables,
there are two types of linear regression
 Simple Linear Regression
 Multiple Linear Regression
• Mathematically, the linear relationship is approximately
modeled as
• y = 0 + 1 x

20
Simple Linear Regression
Estimating the coefficients 0 and 1

21
21
 Performance Metrics
– Mean Absolute Error MAE
– Mean Square Error MSE
– Root MSE RMSE
– Residual Sum of Squares RSS
– Residual Standard Error RSE
– Total Sum of Squares TSS
– Sum of Squares Error SSE
– R-Squared
– Adjusted R Squared
Mean Absolute Error(MAE)

We aim to get a minimum MAE because this is a loss.

Mean Absolute Percentage Error (MAPE)

What is a good value of MAPE?

The closer the MAPE value is to zero, the better the predictions.
A MAPE less than 5% is considered as an indication that the forecast is acceptably accurate.
Mean Squared Error(MSE)

The lower the value the better and 0 means the model is perfect.
Root Mean Squared Error(RMSE)

This produces a value between 0 and 1, where values closer to 0 represent better fitting models. Based on a rule of
thumb, it can be said that RMSE values between 0.2 and 0.5 shows that the model can relatively predict the data
accurately.
RSS and TSS

33
33
RSS, RSE and TSS

To calculate RSS, use the following formula

where

34
34
 RSquared
• The RSE provides an absolute measure of lack of fit of the model to the
data. A small RSE indicates that the model fits the data well whereas a
large RSE indicates that the model doesn’t fit the data well. But since it
is measured in the units of Y, it is not always clear what constitutes a
good RSE
• The R2 statistic provides an alternative measure of fit. It takes the
form of a proportion of variance, expressed as

• Note that R2 statistic is independent of the scale of Y, and it always

takes a value between 0 and 1 35
35
• For example, if 𝑅2 =0.8, then 80% of variance in the data is
explained by the model.
 R-squared and Adjusted R-squared
• A high value of R2 is a good indication.
• However, as the value of R2 tends to increase when more
predictors are added in the model, such as in multiple linear
regression model, you should mainly consider the adjusted R-
squared, which is a penalized R2 for a higher number of
predictors.
– An (adjusted) R2 that is close to 1 indicates that a large proportion of the
variability in the outcome has been explained by the regression model.
– A number near 0 indicates that the regression model did not explain
much of the variability in the outcome.
Rsquared Adjusted
It measures the proportion of variation explained by only those independent variables that really help
in explaining the dependent variable. It penalizes you for adding independent variable that do not help
in predicting the dependent variable.

Adjusted R-Squared can be calculated mathematically in terms of sum of squares. The only difference
between R-square and Adjusted R-square equation is degree of freedom.

38
38
Question-4:
Consider the following five training examples
X = [2 3 4 5 6]
Y = [12.8978 17.7586 23.3192 28.3129 32.1351]
We want to learn a function f(x) of the form f(x) = ax + b which is
parameterized by (a, b).
(a) Find the best linear fit
(b) Find the residuals.
(c) Compute R2 and Adjusted R2

39
39
 Solution:
(X- (X-Xmean)(Y-
X Y Xmean) (Y-Ymean) Ymean) (X-Xmean)2
2 12.8978 -2 -9.9869 19.9738 4
3 17.7586 -1 -5.1261 5.1261 1
4 23.3192 0 0.4345 0.0000 0
5 28.3129 1 5.4282 5.4282 1
6 32.1351 2 9.2504 18.5008 4 The best linear fit is
Sum 20 114.4236 0 0.0000 49.0289 10
Y = 3.2732 + 4.9029X
Mean 4 22.88472
Substituting in
the formula

40
40
 Solution:

(X- (Y- (X-Xmean)(Y-

X Y Xmean) Ymean) Ymean) (X-Xmean)2 Ypredicted (Y-YPredicted)2
2 12.8978 -2 -9.9869 19.9738 4 13.0789 0.0328
3 17.7586 -1 -5.1261 5.1261 1 17.9818 0.0498
4 23.3192 0 0.4345 0.0000 0 22.8847 0.1888
5 28.3129 1 5.4282 5.4282 1 27.7876 0.2759
6 32.1351 2 9.2504 18.5008 4 32.6905 0.3085
Sum 20 114.4236 0 0.0000 49.0289 10 RSS 0.8558
Mean 4 22.88472

Y predicted is calculated using the best linear fit RSS𝒎𝒊𝒏 = 0.8558

Y = 4.9029 + 3.2732 X 41
41
Substituting RSS = 0.8558 and n = 5, then RSE = 0.5341.

42
42
 X Y (Y-Ymean)2

2 12.8978 99.73857 To find R2 value, first find TSS

3 17.7586 26.27711
= 241.2391
4 23.3192 0.188773
5 28.3129 29.46514 = 0.9965
6 32.1351 85.56953
Sum 20 114.4236 241.2391
Mean 4 22.88472

R2adjusted = 1-(1-0.9965)*4/3= 0.9953

43
43
Methods to find best fit line in Linear Regression

1. Statistical
2. Gradient Descent
3. Randomly drawing straight line
Gradient Descent for Linear Regression

 A linear regression model can be trained using the optimization algorithm gradient
descent by iteratively modifying the model’s parameters to reduce the mean squared error
(MSE) of the model on a training dataset.
 To update θ1 and θ2 values in order to reduce the Cost function (minimizing RMSE value)
and achieve the best-fit line the model uses Gradient Descent.
 The idea is to start with random θ1 and θ2 values and then iteratively update the values,
reaching minimum cost.
 A gradient is nothing but a derivative that defines the effects on outputs of the function with a
little bit of variation in inputs.
Advantages of Linear Regression
• Linear regression is a relatively simple algorithm, making it easy to understand and implement. The
coefficients of the linear regression model can be interpreted as the change in the dependent variable for a one-
unit change in the independent variable, providing insights into the relationships between variables.
• Linear regression is computationally efficient and can handle large datasets effectively. It can be trained
quickly on large datasets, making it suitable for real-time applications.
• Linear regression is relatively robust to outliers compared to other machine learning algorithms. Outliers may
have a smaller impact on the overall model performance.
• Linear regression often serves as a good baseline model for comparison with more complex machine learning
algorithms.
• Linear regression is a well-established algorithm with a rich history and is widely available in various machine
learning libraries and software packages.
Disadvantages of Linear Regression
• Linear regression assumes a linear relationship between the dependent and independent variables. If the
relationship is not linear, the model may not perform well.
• Linear regression is sensitive to multicollinearity, which occurs when there is a high correlation
between independent variables. Multicollinearity can inflate the variance of the coefficients and lead to
unstable model predictions.
• Linear regression assumes that the features are already in a suitable form for the model. Feature
engineering may be required to transform features into a format that can be effectively used by the
model.
• Linear regression is susceptible to both overfitting and underfitting. Overfitting occurs when the
model learns the training data too well and fails to generalize to unseen data. Underfitting occurs when
the model is too simple to capture the underlying relationships in the data.
• Linear regression provides limited explanatory power for complex relationships between variables.
More advanced machine learning techniques may be necessary for deeper insights.
Multicollinearity

Multicollinearity occurs when two or more independent variables have a high correlation with one another in a regression
model, which makes it difficult to determine the individual effect of each independent variable on the dependent variable.
The presence of multicollinearity can lead to unstable and unreliable coefficient estimates, making it challenging to
interpret the results and draw meaningful conclusions from the model.
Correlation
If the correlation between two variables is +/- 1.0, then the variables are said to be perfectly collinear.

If there is no relationship at all between two variables, then the correlation coefficient will certainly be 0.

When the correlation coefficient is positive, an increase in one variable also increases the other.

When the correlation coefficient is negative, the changes in the two variables are in opposite directions.
Correlation Computation

Correlation between the two variables is obtained from normalizing the covariance by dividing it with
the product of the standard deviations of the two variables.
Correlation Computation
Correlation…
Correlation value ranges from -1 to +1.

The closer it is to +1 or -1, the more closely the two variables are related.

If there is no relationship at all between two variables, then the correlation coefficient will certainly be 0.

When the correlation coefficient is positive, an increase in one variable also increases the other.

When the correlation coefficient is negative, the changes in the two variables are in opposite directions.
Types of correlation
Types of correlation
Positive correlation
What is a correlation matrix

A correlation matrix is essentially a table depicting the correlation coefficients for various variables. The
rows and columns contain the value of the variables, and each cell shows the correlation coefficient.
Correlation Computation
Question:
 The time x in years that an employee spent at a company and the
employee’s hourly pay, y, for 5 employees are listed in the table below.
Calculate and interpret the correlation coefficient r

59
59
 The correlation coefficient r is given by

 This will always be a number between -1 and 1 (inclusive).

60
60
61
61
62
62
• Interpret this result: There is a strong positive correlation
between the number of years and employee has worked and the
employee’s salary, since r is very close to 1

63
63
Correlation…
Correlation value ranges from -1 to +1.

The closer it is to +1 or -1, the more closely the two variables are related.

If there is no relationship at all between two variables, then the correlation coefficient will certainly be 0.

When the correlation coefficient is positive, an increase in one variable also increases the other.

When the correlation coefficient is negative, the changes in the two variables are in opposite directions.
Types of correlation
Types of correlation
Positive correlation
What is a correlation matrix

A correlation matrix is essentially a table depicting the correlation coefficients for various variables. The
rows and columns contain the value of the variables, and each cell shows the correlation coefficient.
Overfitting and Underfitting

Overfitting occurs when our machine learning model tries to cover all the data points or more than the required data
points present in the given dataset. Because of this, the model starts caching noise and inaccurate values present in the
dataset, and all these factors reduce the efficiency and accuracy of the model. The overfitted model has low
bias and high variance.

Note:
•Bias: Bias is the difference between the predicted values and the actual values.
•Variance: If the machine learning model performs well with the training dataset, but does not perform well with the
test dataset, then variance occurs.

Underfitting represents the inability of the model to learn the training data effectively result in poor performance both
on the training and testing data. In simple terms, an underfit model is inaccurate, especially when applied to new,
unseen examples.
Multilinear Regression
 One of the most common types of predictive analysis is multiple linear regression.
 This type of analysis allows you to understand the relationship between a continuous dependent
variable and two or more independent variables.
 The independent variables can be either continuous (like age and height) or categorical (like
gender and occupation).
 It's important to note that if your dependent variable is categorical, you should dummy code it
before running the analysis.

Here : Y = b0 + b1 * x1 + b2 * x2 + b3 * x3 + …… bn * xn
Y = Dependent variable and x1, x2, x3, …… xn = multiple independent variables
Assumptions of Multiple Linear Regression

•A linear relationship should exist between the target and predictor variables.

•MLR assumes little or no multicollinearity (correlation between the independent variable) in data.
Example of Multiple Linear Regression

Consider a dataset containing information about houses.

•Number of Bedrooms: The count of bedrooms in the house.

•Square Footage: The total area of the house in square feet.
•Age of the House: The number of years since the house was built.
•Price: The target variable represents the price of the house in dollars.

To build a multiple linear regression model to predict the house price based on the first three features.
Multiple Linear Regression Steps

Data Preparation

•Cleaning the Data: Remove or impute missing values to ensure the dataset is complete.
•Encoding Categorical Variables: If any categorical variables are present, convert them into numerical form. In
this example, our dataset is purely numerical.
•Feature Scaling: Standardise or normalise the features if they vary significantly in scale. This step ensures that
all features contribute equally to the model.
Multiple Linear Regression Steps

Model Training

Splitting the Dataset: Divide the dataset into training and testing sets. A split of 80% training and 20% testing
is typically used.

Fitting the Model: Use the training data to fit the multiple linear regression model.

Evaluation of the Model

After training the model, evaluate its performance using the test data
Predicting: Use the model to make predictions on the test set

Metrics: Assess the model’s performance using metrics such as Mean Absolute Error (MAE), Mean Squared Error
(MSE), and R-squared score.
 Multiple Regression
• Multiple regression extends simple regression to include several independent variables
(called predictors).
• Multiple regression is required when a single-predictor model is inadequate to describe
the true relationship between the dependent variable Y (the response variable) and its
potential predictors (X1, X2, X3, . . .).
• The interpretation of multiple regression is similar to simple regression because simple
regression is a special case of multiple regression.

76
Regression Terminology
• The response variable (Y) is assumed to be related to the
k predictors (X1, X2, . . . , Xk ) by a linear equation called
the population regression model:
y = β 0 + β 1 x1 + β 2 x2 + . . . + β k xk + ε
• A random error ε represents everything that is not part of
the model. The unknown regression coefficients β0, β1, β2,
. . . , βk are parameters and are denoted by Greek letters

77
Regression Terminology
• The sample estimates of the regression coefficients are
denoted by Roman letters b0, b1, b2, . . . , bk . The
predicted value of the response variable is denoted 𝑦 and
is calculated by inserting the values of the predictors into
the estimated regression equation:
𝑦 = 𝑏0 + 𝑏1 𝑥1 + 𝑏2 𝑥2 + … . +𝑏𝑘 𝑥𝑘 (predicted value of Y)

78
Data Format
• To obtain a fitted regression, we need n observed values
of the response variable Y and its proposed predictors X1,
X2, . . . , Xk . A multivariate data set is a single column of
Y-values and k columns of X-values.

79
Example

80
[Link]

81
82
83
Sample Data
• Illustration: Home Prices
• Table shows sales of 30 new homes in an upscale development. Although the
selling price of a home (the response variable) may depend on many factors, we
will examine three potential explanatory variables.
• Definition of Variable Short Name
• Y = selling price of a home (thousands of dollars) Price
• X1 = home size (square feet) SqFt
• X2 = lot size (thousand square feet) LotSize
• X3 = number of bathrooms Baths

84
Sample Data

85
Sample Data Home
1
X1
Sqft
2192
X2
LotSize
16.4
X3
Baths
2.5
Y
Price
505.5
2 3429 24.7 3.5 784.1
3 2842 17.7 3.5 649
4 2987 20.3 3.5 689.8
5 3029 22.2 3 709.8
6 2616 20.8 2.5 590.2
7 2978 17.3 3 643.3
8 3595 22.4 3.5 789.7
9 2838 27.4 3 683
10 2591 19.2 2 544.3
11 3633 26.9 4 822.8
12 2822 23.1 3 637.7
13 2994 20.4 3 618.7
14 2696 22.7 3.5 619.3
15 2134 13.4 2.5 490.5
16 3076 19.8 3 675.1
17 3259 20.8 3.5 710.4
18 3162 19.4 4 674.7
19 2885 23.2 3 663.6
20 2550 20.2 3 606.6
21 3380 19.6 4.5 758.9
22 3131 22.5 3.5 723.3
23 2754 19.2 2.5 621.8
24 2710 21.6 3 622.4
25 2616 20.8 2.5 631.3
26 2608 17.3 3.5 574
27 3572 29 4 863.8
28 2924 21.8 2.5 652.7
29 3614 25.5 3.5 844.2
30 2600 24.1 3.5 629.9

86
Estimation of Regression

• Intercept = -28.85
• Slope Sqft = 0.171
• Slope LotSize = 6.78
• Slope Baths = 15.53

87
Predictions from a Fitted Regression

• We can use the fitted regression model to make

predictions for various assumed predictor values. For
example, what would be the expected selling price of a
2,800-square-foot home with 21⁄2 baths on a lot with
18,500 square feet?

88
Predictions from a Fitted Regression

• We can use the fitted regression model to make

predictions for various assumed predictor values. For
example, what would be the expected selling price of a
2,800-square-foot home with 21⁄2 baths on a lot with
18,500 square feet?

89
 Model accuracy

The overall quality of the linear regression fit can be assessed using the following
quantities, displayed in the model summary:
• The Residual Standard Error (RSE).
• The R-squared (R2) (coefficient of determination

90
Polynomial Regression
• Polynomial Regression is a regression algorithm that models the
relationship between a dependent(y) and independent variable(x) as
nth degree polynomial.
• The Polynomial Regression equation is given below:
y= b0+b1x1+ b2x12+ b2x13+...... bnx1n

• It is also called the special case of Multiple Linear Regression in ML.

• It is a linear model with some modification in order to increase the
accuracy.
• The dataset used in Polynomial regression for training is of non-linear
nature.
• It makes use of a linear regression model to fit the complicated and
non-linear functions and datasets.
• Hence, "In Polynomial regression, the original features are
converted into Polynomial features of required degree (2,3,..,n) and
then modeled using a linear model."
Need for Polynomial Regression:

• If we apply a linear model on a linear dataset, then it provides us a

good result as we have seen in Simple Linear Regression, but if we
apply the same model without any modification on a non-linear
dataset, then it will produce a drastic output. Due to which loss
function will increase, the error rate will be high, and accuracy will be
decreased.
• So for such cases, where data points are arranged in a non-linear
fashion, we need the Polynomial Regression model. We can
understand it in a better way using the below comparison diagram of
the linear dataset and non-linear dataset.
• We can understand it in a better way using the below comparison
diagram of the linear dataset and non-linear dataset.
Steps for Polynomial Regression:

• Data Pre-processing
• Build a Linear Regression model and fit it to the dataset
• Build a Polynomial Regression model and fit it to the dataset
• Visualize the result for Linear Regression and Polynomial Regression
model.
• Predicting the output.
Numerical Example
• Given data:
• x = [1, 2, 3, 4, 5]
• y = [3, 6, 11, 18, 27]
x = [1, 2, 3, 4, 5]
y = [3, 6, 11, 18, 27]

• Linear Regression
Equation: y=−5.00+6.00x
x = [1, 2, 3, 4, 5]
y = [3, 6, 11, 18, 27]

• Linear Regression
Equation: y=−5.00+6.00x
x = [1, 2, 3, 4, 5]
y = [3, 6, 11, 18, 27]

• Linear Regression
Equation: y=−5.00+6.00x
Mean Squared Error (MSE): 2.8
R² Score: 0.96
x = [1, 2, 3, 4, 5]
y = [3, 6, 11, 18, 27]

• Linear Regression
Equation: y=−5.00+6.00x
Mean Squared Error (MSE): 2.8
R² Score: 0.96
Linear vs Polynomial Regression
References
• [Link]
• [Link]
• [Link]
intelligence/articles/what-is-linear-regression/
• [Link]
with-scikit-learn-part-4-501068cf021