IIMT 2641 Introduction to Business Analytics

Module 3: Linear Regression

Topic 2: Multiple Linear Regression

1
The wine data (1952 - 1978)

2
Multiple Regression Model simple IV x 1
multiple IV x K
The population model of y with k independent variables (IV) is:
s Kum
𝑦 = 𝛽0 + 𝛽1 𝑥1 + 𝛽2 𝑥2 + ⋯ + 𝛽𝑘 𝑥𝑘 + ε
0in iXk
▪ y is the dependent variable (DV) ideally Xi
▪ x1, x2, …, xk are independent variables (IV) uncorrelated
▪ Regression Function
▪ E[Y|x] = 𝛽0 + 𝛽1 𝑥1 + 𝛽2 𝑥2 + ⋯ + 𝛽𝑘 𝑥𝑘 is the mean of Y given x1, x2, …, xk
▪ 𝛽0 is the y-intercept
Hint fixik
▪ 𝛽𝑗 is the slope for xj for j = 1, 2, …, k
y p po e Ei
▪ Random errors  (not required) F
V
sq
s
▪ Random errors are a random sample from 𝑁 0, 𝜎𝜀
Random samples are i.i.d. or
independent and identically
▪ Each observation has a random error term distributed random variables
▪ The output does not show these, but it does estimate 
– The random errors are uncorrelated with the IV
Ei 1 Xc Xk
– These assumptions are critical for effective business analytics

3
Multiple Regression: Visually with 2 predictors

Xi Xz Y
Regression function is a
y Sample plane in 3 dimensions.
observation
yi

T
ෝ𝒊
𝒚 Residual 𝑟𝑖 = 𝑦𝑖 − 𝑦ො𝑖

𝑦ො𝑖 = 𝑏0 + 𝑏1 𝑥1,𝑖 + 𝑏2 𝑥2,𝑖 x2i

i x2

x1i

x1
f

4
Estimate a linear model: H0: Population Coefficient = 0 versus HA:
Two Variables Population Coefficient ≠ 0

Estimated Standard Errors

intercept and for estimated
AGS1
F
tstat = (Estimated Coefficient – 0)/(Standard Error)

0.602 0 0.11128 5.415

slope coefficients
Two-Tail Test: p-value = 2*P(T<-|tstat|)
a

coeffient
individual

Coefficient of Determination: R-Squared

ones Adjusted R-Squared: Consider number

as of variables.
i
K

SSE = 2.97 < SSE1 = 5.73 F-Test

5
Estimate a linear model:
Two Variables
Estimated
intercept and
slope

6
Estimate a linear model:
Two Variables
Estimated • The predicted LogPrice increases by 0.6 for each
HGST season
degree increase of average growing
intercept and
slope temperature if Harvest rain is held constant
ms
• The predicted LogPrice increases by -0.00457 for
each additional millimetre of Harvest rain if
u
average growing season temperature is held o
constant.

Notice Multiple liner regression

inerees x try I unit
I increas y by Bi
o
increase x by1 unit
while holding Xz Xk
constant

7
incressen y by
Estimate a linear model:
Two Variables
AGST: tstat = ?

• Two-tail p-value = ?
• Conclusion: ?
• df = ?

AGST: tstat = ?

• Two-tail p-value = ?
• Conclusion: ?
8
Estimate a linear model:
Two Variables
𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 𝑆𝑙𝑜𝑝𝑒−0 0.60262−0
AGST: tstat = 𝑠𝑡𝑑.𝑒𝑟𝑟𝑜𝑟
= 0.11128
= 5.415

ea
• Two-tail p-value = 2*P(T<-|tstat|) = 2*[Link](-5.415, 22,1) = 1.94e-5
• Conclusion: Reject H0 at the 0.001 level ofe significance.
• df = n – 1 - #IV = 25 – 1- 2 = 22
o o o

e i

𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 𝑆𝑙𝑜𝑝𝑒−0 −0.00457−0

HarvestRain: tstat = 𝑠𝑡𝑑.𝑒𝑟𝑟𝑜𝑟
= 0.00101
= −4.525
a
a
• Two-tail p-value = 2*P(T<-|tstat|) = 2*[Link](-4.525, 22,1) = 0.000167
• Conclusion: Reject H0 at the 0.001 level of significance.
9
o
 a
Estimate a linear model:
Two Variables
at
H0: all of the slopes are zero versus HA: one or more of the slopes are non-zero
n
H0: 1 = … n= k = 0 versus HA: j ≠ 0 for one or more j

Factoids
1. The null hypothesis means the model is no better than using the mean 𝑦ത to
predict price for wines, while the alternative hypothesis is that some of the IV help
to predict Y better than 𝑦, ത but it does not specify which ones.
2. If H0 is true, the F-statistic has a F distribution. [Link]
eat 1
or How
[Link]
3. Reject H0 if p-value < 0.05
4. If you cannot reject the H0, then your model is really bad
5. Some say, “the model is significant if you reject H 0,” but it may not be a “good”
model.

f test
specific non new

g coefficient
in
10
D
Estimate a linear model (All Variables)

i
IV b
k R
or
pe adj
py
n T
Variable
R t
Multiple

D Tig cnotrequired

N p l 2
a
11 SSE = 1.73
CAdjusted R-squared (consider number of variables) d
R-squared

Variables R2
AGST 0.44
AGST, Harvest rain 0.71
AGST, Harvest rain, Age 0.79
AGST, Harvest rain, Age, Winter rain 0.83
AGST, Harvest rain, Age, Winter rain, Population 0.83
X r
To

• Adding more variables can improve the model R

• The in-sample R2 never declines.
• Diminishing returns as more variables are added

12
Problem with in-sample R-squared

• A major problem with 𝐑𝟐 is adding another independent variable to an

equation can never decrease 𝐑𝟐.

R2 = 1 − SSE/SST Ii
• Adding a variable will not change SST.
E ix i
• Adding a variable will, in most cases, decrease SSE and increase 𝐑𝟐.
• Even if the added variable is nonsensical, 𝐑𝟐 will increase unless the
new coefficient is exactly zero.

13
Overfit the model Yf I I
Peel
x 3
A model can be “overly complicated” if we excessively add independent
variables to fit sample data. too IV Rt v 99
w Iv's 0.98
• Example: If a univariate regression model has only 2 data points, you
will always get an 𝐑𝟐 of 1.
• Fail to fit additional data/accurately predict future observations zoolo
too
Solve the problem:
Adjusted R-squared (consider number of variables)
Out-of-sample testing.
It xI
• Training set: Build model based on a subsample (e.g., 70%) of data random
f
• Test set: Use remainder (e.g., 30%) of the dataset to assess model sample
p pi 70
estimated p p
Yi Got b Xi based on training set
Oo o.o
plug X Xiao into trainedmodel
14 X Xo
 MSE C mean squared errors
Select variables FEED
7
average f Y Yi t i t Yoo Yoo
l I f
sum4
SSE
Not all available variables should always be used. ites In
Adding more variables requires more data to diminish influence of noise.

• Overfitting: high R2 on data used to build the model, but bad

performance on unseen data

How to choose what variables to use?

15
Refine the model
We need to remove insignificant independent variables one at a time.
▪ Multicollinearity: correlation between independent variables
T_
▪ If there is high correlation, some significant variables seem insignificant
because they essentially represent the same [Link]
▪ If one variable is removed, the other could be significant.

16
 Cove Y Y
Correlation
Cov n Var Ef Y n ft
a more general variance

poster
(not required)
x
A measure of the linear relationship (co-movement) between variables
▪ +1 = perfect positive linear relationship
Y x
▪ 0 = no linear relationship
core Y Y Ef
Y axing
D
▪ -1 = perfect negative linear relationship
2
6 61 Etty axis
y x e i
Letitia um e ti
Effect
Yg l E Px
17
x EGt eenY e
Example of Correlation
Correlation between Harvest rain and Average growing season temperature
= -0.0645
dose to 0 almost no relationship

18
Example of Correlation
Correlation between Age of wine (years) and Population of France (in
thousands) = -0.9945
egative relations

19
e
Multicollinearity: Correlations among IV
▪
e
In a perfect world, we would like the IVs to be uncorrelated
– Results in the most efficient use of information
–
I
Adding or subtracting a variable to the model would not change the estimated
coefficients of the other variables
▪ In this world, IVs are often correlated or “collinear”
▪ If the correlations among IVs are too large, then estimated coefficients cannot
be trusted due to multicollinearity
▪ Variance inflation factors (VIF) checks for multicollinearity
i – VIF < 10: no problem
– VIF > 10: problem
❑ Drop variable with large VIF
or combine variables together

WinterRain AGST HarvestRain Age FrancePop

1.298801 1.274536 1.116584 97.219725 98.252693
20
Estimate a linear model (Re-run the model by leaving out
FrancePop)

VIF WinterRain AGST HarvestRain Age

21 1.241993 1.225811 1.113615 1.069950 e lo
 22

What has changed?

▪ All of our independent variables are significant!
▪ By removing an independent variable, all of our coefficient estimates
adjusted slightly.
Tghly correlated
▪ R decreases slightly from 0.8294 to 0.8286, while adjusted R2 increases
2

from 0.7845 to 0.7943.

– If we removed Age and FrancePop at the same time (they were both
insignificant in the original model), R2 would decrease to 0.7537.
Predictive ability

Given that it may improve predictive

e
ability of the model by removing highly
correlated variables, we need a criterion to select among models.
▪ Our wine model with all variables has in-sample 𝐑𝟐 = 0.83.
M
▪ Tells accuracy on the training data that we used to build the model
▪ But how well does the model perform on the new (test) data?
7
– Bordeaux wine buyers profit from being able to predict the quality of wine
before it matures.

popa AGST
price age
Rain

23
Make predictions
not used in building regression model
Predict unknown value of the dependent variable
▪ Plug new value of the independent variables into the estimated regression
equation
Prediction model: Informed guesses about some unobserved property based on
observed properties

y b t b X t bvxrt ble Xk

bo t b Xi new t bk Xnew
X neg o
d
predicted value

24
Out–of–sample R2

Variables
AGST 0.44 OO
In – sample R2 Out - of – sample R2
0.79
AGST, Harvest rain 0.71
C
-0.08
AGST, Harvest rain, Age 0.79 0.53
AGST, Harvest rain, Age, Winter rain 0.83 0.79
AGST, Harvest rain, Age, Winter rain, 0.83 0.76
Population

SST Ely 772

SST Yi F
A at
training
training sample
sample mean mean
25
Comparison between in-sample and out-of-sample R2

• Better in-sample R2 does not imply better out-of-sample R2 .

• Need more data to be conclusive
• Out-of-sample R2 can be negative
This happens when we use the training set mean to calculate SST.

26
Steps for Building Good Regression Models
1. Check F-test. H0: all slopes = 0 vs HA: some slopes ≠ 0
– Stop if p-value > 0.05: you cannot reject H0  y to predict
❑ H0 means IV are not useful in prediction the DV
– Continue if p-value < 0.05 ☺

1
2. Check T-Tests for each IV. H0: slope = 0 vs HA: slope ≠ 0

i
– If some IV have p-value > 0.05 , remove the IV with the maximum p-value
and refit the model move
– Continue if p-value < 0.05 for all IVs in the model ☺
3. See VIF to check the Multicollinearity problem
4. Pick the model with largest Adjusted R-Squared if multiple models
satisfy steps 1-3
5. Can you explain your model?

27
R output
▪ Regress Price (DV)
– Model, Linear Regression,
– DV: LogPrice, IV: AGST, Harvest Rain, Age; Number of observations: 25

s Estinatedcoetf
[Link]

[Link] y.t
0 0250
0.00822
us
？ ？ ？
Lene
257 PITE It statt
？ 3 4
n 0.00905

28
Exercises

[Link]

RI l
variance
j ？ ？ ？ explain
total variance
？
I
▪ What is the estimated model? use significant IVs
LogPrice = -1.478 + 0.532*AGST – 0.005*HarvestRain + 0.025*Age
▪ What proportion of the variance in price is explained by the
regression model?
29
s
R-Squared = 0.79
Exercises

t
individual
F test
？ ？ ？
joint
？

5
▪
0
What is the standard error of the coefficient of AGST?
0.0995
H
▪ What is the test result for testing H0 that the slopes for AGST, HarvestRain,
I
and Age are 0 versus Ha that one or more of the slopes are non-zero?

2.596 157 0.05

30 F-test. Reject H0 because p-value < 0.05
Exercises

？ ？ ？

Test H0: slope Age = 0 versus HA: slope Age ≠ 0 at a significance level of 0.001

o
tstat = 2.875
Degree of freedom: 25 – 1 – 3 = 21
P-value = 2P(T<-|tstat|) > 2P(T>3.819) = 0.001 0 00905
P-value for two-tail test > 0.001
31
Do not reject H0 570.001
 g f and I f l

Exercises

Price
？ ？ ？ clog
price
？ exponential

▪ What is the prediction of LogPrice if Age is 35, HarvestRain is 40, and

AGST is 15.6?
LogPrice = -1.478 + 0.532*15.6 – 0.005*40 + 0.025*35 = 7.496
m
p p p
32
plugging in
Introduction to Statistics
The service time of patients in an urgent care clinic is known to be
exponentially distributed with a mean service time of 17 minutes per patient.
The standard deviation of the service time is also 17 minutes. Healthcare
regulations require an average service time of no more than 20 minutes per
patient during the auditor’s visit, or else the clinic will be fined. If during the
auditor’s visit to the clinic, a random sample 100 patient service times are
observed, what is the probability the clinic will NOT get fined?

33
Introduction to Statistics
The service time of patients in an urgent care clinic is known to be
exponentially distributed with a mean service time of 17 minutes per patient.
The standard deviation of the service time is also 17 minutes. Healthcare
regulations require an average service time of no more than 20 minutes per
patient during the auditor’s visit, or else the clinic will be fined. If during the
auditor’s visit to the clinic, a random sample 100 patient service times are
observed, what is the probability the clinic will NOT get fined?

𝟏𝟕
ഥ ∼ 𝑵(𝟏𝟕, (
Following central limit theorem, 𝑿 )^𝟐).
𝟏𝟎𝟎
𝟏𝟕
ഥ < 𝟐𝟎 = 𝒏𝒐𝒓𝒎. 𝒅𝒊𝒔𝒕 𝟐𝟎, 𝟏𝟕,
P 𝑿 ,𝟏 = 𝑷 𝒁 ≤ 𝟏. 𝟕𝟔 =
𝟏𝟎𝟎
𝟎. 𝟗𝟔.

34