SMMD
Class 9
Richard P. Waterman
Wharton
May 23, 2016
May 19, 2016
1 / 22
Table of contents I
1
What you need to know from last time
Todays material
Collinearity
Definition
What is sxk (adjusted) ?
The market model
The consequences of collinearity
Diagnostics for collinearity
Hypothesis testing in multiple regression
Review
May 19, 2016
2 / 22
Last time
What is multiple regression?
The difference between marginal and partial association.
The scatterplot matrix and its uses.
Use leverage plots to diagnose multiple regressions.
May 19, 2016
3 / 22
Todays material
We will work over the material in Stine, 24.2 & 23.4, BAUR Ch. 5.
Collinearity Stine, 24.2. Chapter 5 in case book
Hypothesis testing for regression slopes Stine, 23.4 and Chapter 5 in
the case book.
May 19, 2016
4 / 22
Collinearity
Definition: correlation between the X-variables.
Consequence: it is difficult to establish which of the X-variables are
most important in the regression (they all look the same).
Visually the regression plane becomes very unstable (sausage in
space, legs on the table).
Key formula:
1
Multiple regression: SE (bk )
n sxk (adjusted)
Contrast this to simple regression where
1
Simple regression: SE (b1 )
n sx1
The difference is in whether the standard deviation of x is adjusted.
May 19, 2016
5 / 22
What is sxk (adjusted) ?
The unique variation in xk that is not explained by the other
explanatory variables.
The standard deviation of the residuals from the regression of xk
against all of the other xs.
May 19, 2016
6 / 22
The market model
Consider the regression of Apple return against the Equal weighted
market return.
May 19, 2016
7 / 22
Interpretations
The slope: when the market goes up by an additional one percent,
then Apple can be expected to increase by 1.09%.
The intercept: on days when the market doesnt move, Apple can be
expected to fall by 0.05% (but it is not significant).
R 2 : 16% of the risk in Apple is explained by the market.
1 - R 2 : 84% of the risk in Apple is not explained by the market, that
is, it is specific to Apple.
May 19, 2016
8 / 22
Getting ready for collinearity
The standard deviation of Equal weighted market return is
0.0072256.
The standard error of the slope is approximately:
se(b1 )
0.018157
1
RMSE
1
= 0.1427215
sx1
0.0072256
n
310
Of course, you can see the exact standard error on the output:
0.142952, which is really close to 0.1427215, proving our
approximation is very good.
May 19, 2016
9 / 22
Introducing a second correlated variable
The regression slope for Equal weighted return has switched
sign, its standard error has exploded and it is no longer statistically
significant.
This is what gross collinearity can do to a regression analysis.
May 19, 2016
10 / 22
Why did the standard error explode?
Find sxk (adjusted) .
Regress Equal weighted return against Market return and find the
standard deviation of the residuals. Of course, thats just the RMSE in
this regression of x1 against x2 .
May 19, 2016
11 / 22
Putting it together
All terms refer to the Equal weighted return variable.
Model
SRM
MRM
sd
0.007226
0.002175
[Link] slope
0.1429
0.4675
The standard error of the slope has increased because the standard
deviation of x has become much smaller.
Recall that the standard deviation of x is in the denominator of the
standard error formula.
The standard deviation fell from 0.007226 to 0.002175 and the
standard error increased from 0.1429 to 0.4675.
May 19, 2016
12 / 22
The Variance Inflation Factor
This is a numeric summary of the extent of collinearity in a multple
regression.
Each variable gets its own VIF.
The VIF is the price you pay for collinearity: the increase of the
variance in the estimated regression coefficient, due to the presence of
collinearity.
An approximation with a nice interpretation:
2
sxk
VIF (Xk )
.
sxk (adjusted)
The exact formula:
VIF (Xk ) =
1
1
R 2 (Xk v .X1 ,
, Xk1 )
When all the xs are all uncorrelated the VIFs are all 1 (perfection).
As the collinearity increases so do the VIFs. VIFs above 10 are a
warning signal to take some action.
May 19, 2016
13 / 22
VIFS in the market model
Using the approximation:
VIF (Xk )
From the JMP
sxk
sxk (adjusted)
2
=
0.007226
0.002175
2
= 11.038.
output:
Note that the approximation (11.038) is close to the true value (11.0678),
so the VIFs interpretation as the ratio of the standard deviations is
justified.
1
You get the VIFs by right clicking in the parameter estimates table, choosing
columns, then VIF
May 19, 2016
14 / 22
Diagnostics for collinearity
Thin ellipses in the scatterplot matrix. (High correlation.)
Counter-intuitive signs on the slopes.
Large standard errors on the slopes (theres little information on
them).
2 Collapsed
leverage plots.
High Variance Inflation Factors. The increase in the variance of the
slope estimate due to collinearity.
VIF (Xk ) =
1
1
R 2 (Xk v .X1 ,
, Xk1 )
Insignificant t-statistics even though over all regression is significant
(ANOVA F-test).
2
To be discussed in the next class
May 19, 2016
15 / 22
Fix ups for collinearity
Ignore it. OK if sole objective is prediction in the range of the data.
Combine collinear variables in a meaningful way.
Delete variables. OK if extremely correlated.
May 19, 2016
16 / 22
Hypothesis testing in multiple regression
Recall the regression assumptions. If they are seriously broken, then these
tests could be misleading.
Three flavors. They all test whether slopes are equal to zero or not. They
differ in the number of slopes we are looking at simultaneously.
1
Test a single regression coefficient (slope).
Test all the regression coefficients at once.
Test a subset of the regression coefficients (more than one, but not
all of them the Partial F-test).
May 19, 2016
17 / 22
Test a single regression coefficient (slope)
Stine: p.613.
Look for the t-statistic.
The hypothesis test in English: does this variable add any explanatory
power to the model that already includes all the other X-variables?
Small p-value says YES, big p-value says NO.
May 19, 2016
18 / 22
Test all the regression coefficients at once
Stine: p. 611.
It is sometimes called the whole model test.
Look for the F-statistic in the ANOVA table.
The hypothesis test in English: do any of the X-variables in the model
explain any of the variability in the Y-variable?
Small p-value says YES, big p-value says NO.
Note that the test does not identify which variables are important.
If you answer this question as NO then its back to the drawing board
none of your variables are any good!
May 19, 2016
19 / 22
Test a subset of the regression coefficients
See formula on p.152 of BAUR. An example using the custom test
dialog is on the LMS.
The test in English: do any of the X-variables in the subset under
consideration explain any of the variability in Y?
We read the p-value from the JMP output to answer this one.
You must be able to answer this question: why not do a whole bunch of
t-tests rather than one partial F-test? Answer: the partial F-test is an
honest simultaneous test (see p. 135 of BAUR).
May 19, 2016
20 / 22
Writing out the hypothesis tests formally
Consider a model with Y , X1 , X2 and X3
An example of a test for a single coefficient would be:
H0 : 2 = 0 v .H1 : 2 6= 0.
The whole model test would be:
H0 : 1 = 2 = 3 = 0 v .
H1 : at least one of 1 , 2 or 3 6= 0.
An example of the partial-F test would be:
H0 : 2 = 3 = 0 v . H1 : at least one of 2 or 3 6= 0.
May 19, 2016
21 / 22
Review
Introduced collinearity; definition, diagnostics and fix-ups.
Introduced the various tests in multiple regression.
May 19, 2016
22 / 22