IIMT 2641 Introduction to Business Analytics
Module 3: Linear Regression
Topic 1: Simple Linear Regression
1
Bordeaux wine
▪ Large differences in price and quality between years, although wine is
produced in a similar way
▪ Meant to be aged, so hard to tell if wine will be good when it is on the
market
▪ Expert tasters predict which ones will be good
▪ Can analytics be used to come up with a different system for judging wine?
3
Predicting the quality of wine
▪ March 1990 - Orley Ashenfelter, a Princeton economics professor, claims
he can predict wine quality without tasting the wine
4
Building a model
▪ Ashenfelter used a method called linear regression
– Predicts an outcome variable, or dependent variable
– Predicts using a set of independent variables
5
Building a model
▪ Dependent variable:
– Typical price in 1990-1991 wine auctions (approximates quality)
– Conduct logarithmic transformation
❑ A better linear fit
▪ Independent variables:
– Age of wine (in 1990)
❑ Older wines are more expensive
– Weather
❑ Average Growing Season Temperature (AGST)
❑ Harvest Rain
❑ Winter Rain
– Population of France
6
The wine data (1952 - 1978)
8
The wine data (1952 - 1978)
Quick Question: What is the relationship between harvest rain, average
growing season temperature, and wine prices?
9
Baseline model (?)
10
Baseline model (Take the mean)
11
One-Variable Linear Regression
12
Simple Regression Model
The population model of y with one predictor variable x is:
𝑦 = 𝛽0 + 𝛽1 𝑥 + ε
▪ y is the dependent variable (DV)
▪ x is the independent variable (IV)
▪ Regression Function
▪ E[Y|x] = 𝛽0 + 𝛽1𝑥 is the mean of Y given x
▪ 𝛽0 is the y-intercept (value of E[Y|0] when x=0)
▪ 𝛽1 is the slope for x, which is the change in E[Y|x] for a unit increase in x
▪ Random errors (not required)
▪ Random errors are a random sample from 𝑁 0, 𝜎𝜀 Random samples are i.i.d. or
▪ Each observation has a random error independent and identically
▪ The output does not show these, but it does estimate distributed random variables
▪ The random errors and IV (X) are uncorrelated
▪ These assumptions are important for effective business analytics
13
Estimated Regression Function
▪ Estimates the regression model with n observations (xi,yi) for i = 1, …, n
▪ The estimated or predicted value of y given x is:
𝑦ො = 𝑏0 + 𝑏1 𝑥
▪ 𝑏0 is the sample estimate of the population intercept 𝛽0
▪ 𝑏1 is the sample estimate of the population slope 𝛽1
𝑏0 and 𝑏1 are sample statistics
ത
(similar to 𝑋)
and have sampling distributions
14
One-Variable Linear Regression
15
Data and Predicted Values
▪ What is the observed y when x = 1?
▪ What is the predicted y when x = 1?
▪ What is the observed y when x = 4?
▪ What is the predicted y when x = 4?
16
Estimated Model and Residuals
▪ Residuals are the difference between the observed values of 𝑦 and
predicted values of 𝑦ො
– 𝑟 = y - 𝑦ො
– Each observation has one observed y, one predicted 𝑦,
ො and one residual r.
▪ The residuals are errors between the observed and predicted values.
y3
𝑦ො4
𝑟3 = 𝑦3 − 𝑦ො3
y1
𝑦ො2 𝑟4 = 𝑦4 − 𝑦ො4
𝑦ො3
𝑟1 = 𝑦1 − 𝑦ො1
y4
𝑦ො1 y2
𝑟2 = 𝑦2 − 𝑦ො2
17
Computing Residuals
r3
r4
r1 r2
▪ What is the residual r2 at x = 2?
▪ What is the residual r3 at x = 3?
18
Ordinary Least Squares Criterion or (OLS)
The least squares line finds the estimates 𝑏0 and 𝑏1 of the coefficients to
minimize the sums-of-squares error for a sample {(xi,yi)} with n observations:
𝑆𝑆𝐸 = σ𝑛𝑖=1 𝑦𝑖 − 𝑦ො𝑖 2
𝑦ො𝑖 = 𝑏0 + 𝑏1 𝑥𝑖 for 𝑖 = 1, … , 𝑛
Why squared?
The sum of residuals could be zero.
𝑛
𝑆𝑆𝐸(𝑏0 , 𝑏1 ) = 𝑦𝑖 − 𝑏0 − 𝑏1 𝑥𝑖 2
σ𝑛𝑖=1 𝑥𝑖 − 𝑥ҧ 𝑦𝑖 − 𝑦ത
𝑖=1 𝑏1 =
𝜕𝑆𝑆𝐸(𝑏0 , 𝑏1 ) σ𝑛𝑖=1 𝑥𝑖 − 𝑥ҧ 2
= 0 𝑏0 = 𝑦ത − 𝑏1 𝑥ҧ
𝜕𝑏0
𝜕𝑆𝑆𝐸(𝑏0 , 𝑏1 ) 𝑥:ҧ sample average of independent variable
= 0
𝜕𝑏1 𝑦:
ത sample average of dependent variable
Do not need to memorize.
19
Estimate a linear model H0: AGST Coefficient = 0 versus HA: AGST
(One Variable ) Coefficient ≠ 0
Estimated Standard Errors t-score = (Estimated Coefficient – 0)/(Standard Error)
intercept and for estimated
slope coefficients
Two-Tail Test: p-value = 2*P(T<-|t-score|)
Coefficient of Determination: R-Squared
20
One-Variable Linear Regression
𝑦=
ො -3.4178 + 0.6351*AGST
21
Estimate a linear model
(One Variable )
• Estimated model for price:
𝑦=
ො -3.4178 + 0.6351*AGST
• The predicted LogPrice increases by
0.6351 for every 1 degree increase in
average growing season temperature.
• If AGST = 15, then 𝑦=ො ?
• If AGST = 18, then 𝑦=
ො ?
• If AGST = 20, then 𝑦=
ො ?
22
T-Tests for the Coefficients: H0: j = 0 versus HA: j ≠ 0
Two-Tail Test for the Slope
(Very important. Can you predict Y from X?)
H0: 1 = 0 versus HA: 1 ≠ 0
• t-score = (coefficient – 0)/([Link])
• t-score = (0.6351-0)/0.1509 = 4.208
• p_value = 2*P(T < -|4.208|)
=2*[Link](-4.208, 23, 1) < .001
• df = n-1-#IV = df Error under Sum of squares
df = 23
23
How well the model fits data
▪ The simplest commonly used measure of fit is R2 (the coefficient of
determination): R2 = 1 − SSE/SST
– SSE = σn i=1 yi − yො i 2 : sum of squared errors
❑ Variation of Y that cannot be explained by the regression
– SST = σn ത 2 : total sum of squares
i=1 yi − y
❑ Total amount of variation of Y around its mean
❑ “Error” generated by a baseline model without any inputs
– Decomposition of variation of Y:
𝑛
❑ σ
𝑖=1 𝑦𝑖 − 𝑦ത 2 = σ𝑛𝑖=1 𝑦𝑖 − 𝑦ො𝑖 2 + σ𝑛𝑖=1 𝑦ො𝑖 − 𝑦ത 2
Total variation Unexplained variation Explained variation
R2 = Proportion of the variance in DV is explained by the
regression model.
24
Coefficient of Determination: R-Squared
• R-Squared is a measure of fit
• Bigger R-Squared indicates better fit all
else being equal
• 43.5% of the variation of prices is
explained by the simple regression on
AGST
• 0 < R-Squared < 1
25
Use each variable on its own
▪ R2 =0.44 using Average growing season temperature (Variable
Significant, 0.001)
▪ R2 =0.32 using Harvest rain (Variable Significant, 0.01)
▪ R2 =0.22 using France Population (Variable Significant, 0.05)
▪ R2 =0.20 using Age (Variable Significant, 0.05)
▪ R2 =0.02 using Winter rain (Not Significant)
▪ Multivariate linear regression allows us to use more than one
variable to potentially improve our predictive ability.
26