Scribd
Upload
6 views18 pages

Linear Regression Basics and Models

The document discusses predictive analytics focusing on regression and classification, particularly linear regression. It provides examples of predicting car mileage based on various features and explains the regression model, including the formulation of the regression line and the method of least squares for fitting the model. The document also outlines the normal equations used to estimate the coefficients in the regression model.

Uploaded by

mihami4533
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
6 views18 pages

Linear Regression Basics and Models

The document discusses predictive analytics focusing on regression and classification, particularly linear regression. It provides examples of predicting car mileage based on various features and explains the regression model, including the formulation of the regression line and the method of least squares for fitting the model. The document also outlines the normal equations used to estimate the coefficients in the regression model.

Uploaded by

mihami4533
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Predictive Analytics

Regression and Classification


Module 2

Sourish

Chennai Mathematical Institute


Regression
Motivating Examples of Linear Regression
Ex 1 Given the different features of a new prototype car, can you
predict the mileage or ‘miles per gallon’ of the car?
Motivating Examples of Regression

Ex Given the different features of a new prototype car, can you


predict the mileage or ‘miles per gallon’ of the car?

I mpg cyl disp hp wt


Mazda RX4 21.0 6 160 110 2.620
Mazda RX4 Wag 21.0 6 160 110 2.875
Datsun 710 22.8 4 108 93 2.320
Hornet 4 Drive 21.4 6 258 110 3.215
.....
Prototype ? 4 120 100 3.200

I Note that your objective is to predict the variable mpg.


Plot the data

30
25
mpg

20
15
10

2 3 4 5

wt
Regression Line
mpg=β0 +β1 wt+

30
25
mpg

20
15
10

2 3 4 5

wt
Regression Plane
mpg=β0 +β1 wt+β2 disp+

35
30
25
mpg

disp
500
20

400
300
15

200
100
10

0
1 2 3 4 5 6

wt
Regression Model

I Given a vector of inputs X T = (X1 , X2 , X3 ), we predict the


output Y via model

Y = β0 + X1 β1 + X2 β2 + X3 β3 + .

The term β0 is the intercept.


I Often it is convenient to include the constant variable 1 in
X , include β0 in the vector of coefficients β = (β1 , β2 , β3 )

I We have data about y and X

I How can we estimate β = (β1 , β2 , β3 )?


Regression Model
I Given a vector of inputs X T = (X1 , X2 , . . . , Xp ), we predict
the output Y via model
p
X
Y = β0 + Xj βj + .
j=1

The term β0 is the intercept, also known as the bias in


machine learning.
I Often it is convenient to include the constant variable 1 in
X , include β0 in the vector of coefficients β = (β1 , · · · , βp )

I We have data about y and X

I How can we estimate β = (β1 , · · · , βp )?


Regression Line
mpg=35 - 5wt+

30
25
mpg

20
15
10

2 3 4 5

wt
Regression Line
mpg=39 - 6wt+

30
25
mpg

20
15
10

2 3 4 5

wt
Choice of β
(β0 = 35, β1 = −5) and (β0 = 39, β1 = −6)

−2
−4
−6
β1

−8
−10

30 35 40 45

β0
Choice of β
However, thousands of choices are there, which one is best?

−2
−4
−6
β1

−8
−10

30 35 40 45

β0
How do we fit Linear Regression Models?

I Consider the model

y n×1 = X n×p β p×1 + n×1

I Many different methods, most popular is least squares.

I minimize the residual sum of squares

RSS(β) = (y − X β)T (y − X β)
Xn
= (yi − xiT β)2
i=1
n
X
= ε2i = T 
i=1
Residual Sum of Square : Surface

I RSS(β) is a quadratic function of the parameters

I Its minimum always exists, but may not be unique.


How do we fit Regression models?

I Differentiate RSS(β) with respect to β and equate to 0

∂RSS(β)
=0
∂β
∂
=⇒ (y − X β)T (y − X β) = 0
∂β
=⇒ −2X T (y − X β) = 0
=⇒ X T X β = X T y Normal Equations

I X T X is p × p matrix,
I So normal equations have p unknown and p equations.
How do we fit Regression models?

I Normal Equations

XTXβ = XTy

I X T X is p × p matrix,
I So normal equations have p unknown and p equations.

I Solving the equations, we have

β̂ = (X T X )−1 X T y

I Least Squares method provides analytical solution


Thank You

You might also like