10/3/2025

Unit 02
Supervised Learning

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) |1

Simple Linear Regression (SLR)

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) |2

1
 10/3/2025

Linear regression
Linear Regression:
 a statistical approach to modeling the relationship
between a dependent variable and a given set of
independent variables

 There is a positive linear

correlation(relationship) when the variable
on the x-axis increases as the variable on
the y-axis increases

 There is a negative linear correlation when

one variable increases as the other variable
decreases
R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) |3

Linear regression
Linear Regression:
 There is a non-linear
correlation when there is a
relationship between variables
but the relationship is
not linear (straight)

 There is no correlation when

there is no pattern that can be
detected between the variables

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) |4

2
 10/3/2025

Simple Linear regression

 a statistical method for establishing the relationship between single
dependent variable and a single independent variable using a straight
line(e.g. Predicting a student’s marks based on hours studied)
 Dependent variable must be a continuous/real value
 Independent variable can be measured on continuous or categorical
values
 Objectives:
1 Model
1. M d l the
th relationship
l ti hi between
b t th two
the t variables
i bl – e.g. Income
I &
Expenditure, Experience & Salary etc.
2. Forecasting new observations - Weather forecasting according to
temperature
R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) |5

Simple Linear Regression

Equation of SLR:
y = a0+a1x + є
Where,
Where
 y: Predicted Variable
 a0 : ‘y’ Intercept
a1
 a1: Coefficient of regression
 x: Independent variable a0
 Є : Error

 Goal: find optimum values of a0 and a1 (Best Fit Line)

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) |6

3
 10/3/2025

Simple Linear regression

Best Fit Line
 A regression line that
implies as minimum error
in actual and predicted
value as possible
 Points exactly on
regression line(Red dots) –
predicted values
 Points around the
regression line(Black dots)
– actual values
 Several lines may exists
R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) |7

Simple Linear regression

Best Fit Line
 A regression line that
implies as minimum error
in actual and predicted
value as possible
 Points exactly on
regression line – predicted
values
 Points around the
regression line – actual
values
 Several lines may exists
R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) |8

4
 10/3/2025

Simple Linear Regression

Equation for SLR:

y = a0 + a1X
Where,
 y: Predicted value
 a0: Y- intersect
 a1 : Slope of independent variable
 X : Independent variable

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) |9

Simple Linear regression

Assumptions of SLR:
1. Linearity:
 The independent and dependent variables have a linear relationship
 Changes in the dependent variable follow those in the independent
variable(s) in a linear fashion
 If the relationship is not linear, then linear regression will not be an
accurate model

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 10

5
 10/3/2025

Simple Linear regression

Assumptions of SLR:
2. Independence:
 The observations/actual
/ values in the dataset are independent
p of each
other
 i. e. Value of the dependent variable for one observation does not depend
on the value of the dependent variable for another observation
 If the observations are not independent, then linear regression will not be
an accurate model
Height
H i ht iin ft Weight
W i ht iin K
Kg BMI
55 is independent of 60 and 75 (X) (y)
5 55 9.3
5.5 60 21.3
9.3 is independent of 60 and 75
6 75 22.4
R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 11

Simple Linear regression

Assumptions of SLR:
3. Homoscedasticity (Constant Variance) :
 Across all levels of the independent
variable(s), the variance of the errors
is constant

 This indicates that the amount of the

independent variable(s) has no impact
on the variance of the errors

 If the variance of the errore is not

constant, then linear regression will
not be an accurate model
R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 12

6
 10/3/2025

Simple Linear regression

Assumptions of SLR:
4. Normality :
 The residuals (Actual value –
Predicted value) should be normally
distributed
 If all the residuals (errors) are plotted
in a histogram, they should roughly
form a bell-shaped p curve ((normal
distribution)
 If the residuals are not normally
distributed, then linear regression
will not be an accurate model
R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 13

Simple Linear regression

Example 1 - (Least Square Method ):
 Given a dataset, find the equation (using least
square method) for prediction of student's final
exam grade based on mid term grade in the
course Machine Learning.
 Predict the final exam grade of student who
obtained 70 grade in the mid term exam.

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 14

7
 10/3/2025

Simple Linear Regression

Equation for a0 and a1(Least Square Method):

Equation of SLR: y = a0 + a1X a0 = y

y̅ - a1X

Where,
 Xi: ‘i‘
i th input variable
 yi: ‘i‘ th onput variable
 x̅ : Mean of all X values
 y̅ : Mean of all y values
R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 15

Simple Linear Regression

Example (Least Square Method ):
 Here, Mid Term is independent variable and Final
Exam is dependent variable
 Equation of Simple Linear regression:
Final Exam = a0 + a1(Mid Term)

 Least Square Method:

a1

a0 = y̅ - a1x͞

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 16

8
 10/3/2025

Simple Linear Regression

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 17

Simple Linear Regression

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 18

9
 10/3/2025

Simple Linear Regression

-0.17 10 -1. 7 0.03

-22.17 11 243.87 491.51
8.83 3 26.49 77.97
1.83 4 7.32 3.35
21.83 16 349.28 476.55
13.83 1 13.83 91.27
-13.17 -25 329.25 173.45
10.83 5 54.15 117.29
-7.17 3 -21.51 51.41
-39.17 -22 861.74 1534.29
15.83 0 0 250.59
8.83 16 141.28 77.97

2004 3445.67

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 19

Simple Linear Regression

Example (Least Square Method ):
 Hence, SLR equation is:

Fi l Exam
Final E = 32.02
32 02 + 0.5816
0 5816 x (Mid Term)
T )

 When Mid Term is 70

Final Exam = 32.02 + 0.5816 x 70
= 72 74
72.74

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 20

10
 10/3/2025

Simple Linear Regression (SLR) Code

Example

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 21

Simple Linear Regression

Evaluation metric: Mean Squared Error (MSE )
 Mean Squared Error is a loss function used in regression
g squared
 It measures the average q difference between the ppredicted values and
the actual values

 Squaring makes all errors positive and penalizes large mistakes more than
small ones
 For example, being wrong by 10 (error² = 100) is much worse than being
wrong by 2 (error² = 4)
R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 22

11
 10/3/2025

Simple Linear Regression

Evaluation metric: Mean Squared Error (MSE )

Interpretation of MSE:
 MSE = 0 → Perfect predictions (no error).
 Small MSE → Predictions are close to actual values
(good model).
 Large MSE → Predictions are far from actual values
(bad model)

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 23

Simple Linear Regression

Mean Squared Error (MSE) Final Exam ( y) Predicted (ŷ)
Example 1: 84 73.903067
 From Final Exam Marks and 63 61.107865
Predicted marks calculate the Mean 77 79.137467
Squared Error. 78 75.066267
90 86.698268
75 82.045468
49 66.342266
79 80.300668
77 69.831866
52 51.220664
74 83.208668
90 79.137467

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 24

12
 10/3/2025

Simple Linear Regression

Mean Squared Error (MSE) Final Exam ( y) Predicted (ŷ)
Example 1: 84 73.903067
63 61.107865
77 79.137467
78 75.066267
90 86.698268
75 82.045468
49 66.342266
79 80.300668
77 69.831866
52 51.220664
74 83.208668
90 79.137467

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 25

Simple Linear Regression

Mean Squared Error Final ( y) Predicted (ŷ) Error e=(y−ŷ) Squared error (e2)
(MSE) Example 1: 84 73.903067
63 61.107865
77 79.137467
78 75.066267
90 86.698268
75 82.045468
49 66.342266
79 80.300668
77 69.831866
52 51.220664
74 83.208668
90 79.137467

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 26

13
 10/3/2025

Simple Linear Regression

Mean Squared Error Final ( y) Predicted (ŷ) Error e=(y−ŷ) Squared error (e2)
(MSE) Example 1: 84 73.903067 10.096933
63 61.107865 1.892135
77 79.137467 -2.137467
78 75.066267 2.933733
90 86.698268 3.301732
75 82.045468 -7.045468
49 66.342266 -17.342266
79 80.300668 -1.300668
77 69.831866 7.168134
52 51.220664 0.779336
74 83.208668 -9.208668
90 79.137467 10.862533

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 27

Simple Linear Regression

Mean Squared Error Final ( y) Predicted (ŷ) Error e=(y−ŷ) Squared error (e2)
(MSE) Example 1: 84 73.903067 10.096933 101.948056
63 61.107865 1.892135 3.580175
77 79.137467 -2.137467 4.568765
78 75.066267 2.933733 8.606789
90 86.698268 3.301732 10.901434
75 82.045468 -7.045468 49.638619
49 66.342266 -17.342266 300.75419
79 80.300668 -1.300668 1.691737
77 69.831866 7.168134 5
51.382145
5
52 51.220664 0.779336 0.607365
74 83.208668 -9.208668 84.799566
90 79.137467 10.862533 117.994623

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 28

14
 10/3/2025

Simple Linear Regression

Mean Squared Error Final ( y) Predicted (ŷ) Error e=(y−ŷ) Squared error (e2)
(MSE) Example 1: 84 72.251717 11.748283 138.022153
63 61.478379 1.521621 2.315330
77 76.658992 0.341008 0.116286
78 73.231112 4.768888 22.742293
90 83.025056 6.974944 48.649844
75 79.107478 -4.107478 16.871376
49 65.885653 -16.885653 285.125277
79 77.638386 1.361614 1.853993
77 68 823837
68.823837 8 176163
8.176163 66 849641
66.849641
52 53.153526 -1.153526 1.330622
Interpretation: 74 80.086872 -26.086872 680.524891
Since, MSE is high this model 90 76.658992 13.341008 177.982494
is a bad model
Σ 736.473465
R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 29

Simple Linear Regression

Mean Squared Error (MSE) Code

Example

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 30

15
 10/3/2025

Simple Linear Regression

Example2(Least Square Method ) Exercise: Hours_Studied Exam_Score
 Given a dataset, find the equation (using least (X) (y)
square method) for prediction of student's final 1 35
Exam_Score based on Hours_Studied. 2 42

 Predict the Exam_Score of student who has 3 50

studied for 12 hours daily. 4 55

5 62
6 70
7 72
8 78
9 85
10 90
12 ?

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 31

Simple Linear Regression

Hours_Studied Exam_Score
Example 2 (Least Square Method ): (X) (y)
 Here, Hours_Studied is independent variable and 1 35
Exam_Score is dependent variable 2 42
 Equation of Simple Linear regression: 3 50

Exam_Score = a0 + a1(Hours_Studied ) 4 55
5 62
6 70
 Least Square Method: 7 72
8 78
9 85
10 90
12 ?

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 32

16
 10/3/2025

Simple Linear Regression

Hours_Stud Exam_Score
ied(X) (Y)
1 35 -4.5 -28.9 130.05 20.25
2 42 -3.5 -21.9 76.65 12.25
3 50 -2.5 -13.9 34.75 6.25
4 55 -1.5 -8.9 13.35 2.25
5 62 -0.5 -1.9 0.95 0.25
6 70 0.5 6.1 3.05 0.25 a0 = 63.9 – (6.03)(5.5)
= 30.7
7 72 1.5 8.1 12.15 2.25
8 78 2.5 14.1 35.25 6.25 y = 30.7 + 6.0(Hours_studied)

9 85 3.5 21.1 73.85 12.25 When Hours_studied = 12

10 90 4.5 26.1 117.45 20.25
y = 30.7 + 6.0 (12)
Σ = 497.5 Σ = 82.5
y = 103
R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 33

Simple Linear Regression

Mean Squared Error (MSE) Hours_Studied Exam_Score Predicted_Final
Example 2: 1 35 36.763636
2 42 42.793939
3 50 48.824242
4 55 54.854545
5 62 60.884848
6 70 66.915152
7 72 72.945455
8 78 78.975758
9 85 85.006061
10 90 91.036364

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 34

17
 10/3/2025

Simple Linear Regression

Mean Squared Error Exam_Score Predicted Error Squared
(y) (ŷ) e=(y−ŷ) error (e2)
(MSE) Example 2:
35 36.763636
42 42 793939
42.793939
50 48.824242
55 54.854545
62 60.884848
70 66.915152
72 72.945455
78 78.975758
85 85.006061
90 91.036364

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 35

Simple Linear Regression

Mean Squared Error Exam_Score Predicted Error Squared
(y) (ŷ) e=(y−ŷ) error (e2)
(MSE) Example 2:
35 36.763636 -1.763636
42 42 793939
42.793939 -0
0.793939
793939
50 48.824242 1.175758
55 54.854545 0.145455
62 60.884848 1.115152
70 66.915152 3.084848
72 72.945455 -0.945455
78 78.975758 -0.975758
85 85.006061 -0.006061
90 91.036364 -1.036364

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 36

18
 10/3/2025

Simple Linear Regression

Mean Squared Error Exam_Score Predicted Error Squared
(y) (ŷ) e=(y−ŷ) error (e2)
(MSE) Example 2:
35 36.763636 -1.763636 3.110415
42 42 793939
42.793939 -0
0.793939
793939 0 630340
0.630340
50 48.824242 1.175758 1.382398
55 54.854545 0.145455 0.021159
62 60.884848 1.115152 1.243559
70 66.915152 3.084848 9.512272
72 72.945455 -0.945455 0.893878
78 78.975758 -0.975758 0.952106
85 85.006061 -0.006061 0.000037
90 91.036364 -1.036364 1.074056

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 37

Simple Linear Regression

Mean Squared Error Exam_Score Predicted Error Squared
(y) (ŷ) e=(y−ŷ) error (e2)
(MSE) Example 2:
35 36.763636 -1.763636 3.110415
42 42 793939
42.793939 -0
0.793939
793939 0 630340
0.630340
50 48.824242 1.175758 1.382398
55 54.854545 0.145455 0.021159
62 60.884848 1.115152 1.243559
70 66.915152 3.084848 9.512272
72 72.945455 -0.945455 0.893878
78 78.975758 -0.975758 0.952106
85 85.006061 -0.006061 0.000037
Interpretation:
90 91.036364 -1.036364 1.074056
Since, MSE is too small this
model is a good model Σ 18.820219

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 38

19
 10/3/2025

Mean Squared Error (MSE) Code

Example 2

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 39

Multiple Linear Regression

(MLR)

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 40

20
 10/3/2025

Multiple Linear regression

Linear Regression:
 Simple Linear Regression − only addresses two features.
 Multiple Linear Regression − Deals with more than one feature at once
F1: size of the
house
Independent Variables
F2: No. of
Bedrooms
Size of No. of Locality Condition Price
the Bed of the
F3: Locality house rooms house

F4: Condition of
the house

F5: Price of
House ? Dependent Variable

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 41

Multiple Linear regression

Equation of MLR:

y = a0+a1x1+a2x2+a3x3+ …… anxn + є
Where,
 a0 : ‘y’ Intercept
 a1 , a2 , a3 , … an : Slope or Coefficient of regression
 x1 , x2 , x3 , … xn : Independent / Predictor variables
 Є : Error

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 42

21
 10/3/2025

Multiple Linear regression

Equation of MLR:
Size of No. of Locality Conditio Price
the Bed n of the
house rooms house
F1: size of the
house

F2: No. of
Bedrooms

F3: Locality

F4: Condition of
the house Price of House =
F5: Price of a0 + a1(Size of the house) + a2(No.
House ? of Bed rooms) + a3(Locality) +
a4(Condition of the house)

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 43

Multiple Linear regression

Assumptions of Regression Model:

y The relationship
 Linearity: p between dependent
p and
independent variables should be linear

 Multivariate normality: Multiple Regression assumes that the

residuals are normally distributed.

 Lack of Multi-colinearity: It is assumed that there is little or

no multicollinearity between the independent variables.

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 44

22
 10/3/2025

Multiple Linear regression

Numerical Example: TV Mobile
Weekly
Sales
(x1) (x2)
 Equation: (Y)

Weeklyy Sales = a0+a1((TV)+a

) 2((Mobile)) 1 4 1

 Matrices for TV and Mobile will be as: 2 5 6

3 8 8

4 2 12

6 10 ?

Bi
Bias I t
Intersect
t

Coeff. Of
 Equation of Y-Intersect and Slopes will be TV

as: Coeff. Of
Mobile

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 45

Multiple Linear regression

Numerical Example: TV Mobile
Weekly
Sales
(x1) (x2)
 Matrix of coefficients is given as: (Y)
1 4 1

2 5 6

3 8 8

4 2 12

6 10 ?

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 46

23
 10/3/2025

Multiple Linear regression

Numerical Example: TV Mobile
Weekly
Sales
(x1) (x2)
 Matrix of coefficients is given as: (Y)
1 4 1

2 5 6

3 8 8

4 2 12

6 10 ?

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 47

Multiple Linear regression

Numerical Example: TV Mobile
Weekly
Sales
(x1) (x2)
 Matrix of coefficients is given as: (Y)
1 4 1

2 5 6

3 8 8

4 2 12

6 10 ?

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 48

24
 10/3/2025

Multiple Linear regression

Numerical Example: TV Mobile
Weekly
Sales
(x1) (x2)
 Matrix of coefficients is given as: (Y)
1 4 1

2 5 6

3 8 8

4 2 12

6 10 ?

Intersect

Coeff. Of
TV

Coeff. Of
Mobile

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 49

Multiple Linear regression

Numerical Example: TV Mobile
Weekly
Sales
(x1) (x2)
 Hence, the Multiple Linear Regression (Y)
model is as: 1 4 1

Weekly Sales = 2 5 6

-1.69+3.48(TV)+0.05(Mobile) 3 8 8

4 2 12

= -1.69+3.48(6)+0.05(10) 6 10 ?

= 19.69
19 69

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 50

25
 10/3/2025

Multiple Linear Regression

Multiple Linear Regression Code

Example

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 51

Polynomial Regression

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 52

26
 10/3/2025

Polynomial Regression

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 53

Polynomial Regression

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 54

27
 10/3/2025

Polynomial Regression

n = No. of
elements

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 55

Polynomial Regression
Numerical Example:
 Apply Polynomial Regression on
given a dataset and find the
equation.
 Also predict the value of y when x =
7.

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 56

28
 10/3/2025

Polynomial Regression
Numerical Example:

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 57

Polynomial Regression
Numerical Example:

Equation:

When x = 7, y = -0.75+0.95(7)+0.75(49) = 42.65

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 58

29
 10/3/2025

Polynomial Regression
Advantages
 Captures non-linear relationships.
 More flexible than simple linear regression.

Disadvantages
 Overfitting if polynomial degree is too high.
 Sensitive to outliers.
 Interpretation becomes harder for high-degree polynomials.

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 59

Polynomial Regression

Polynomial Regression Code Example

R. B. Mannade, Asst. Professor Govt. College of Engineering, Aurangabad(MS) | 60

30