Statistics Micro Mini
Multiple Regression
January 5-9, 2008
Beth Ayers
January 6, 2009 - morni 1
ng session
Tuesday 9am-12pm Session
• Critique of An Experiment in Grading
Papers
• Review of simple linear regression
• Introduction to Multiple regression
‒ Assumptions
‒ Model checking
‒ R2
‒ Multicollinearity
January 6, 2009 - morni 2
ng session
Simple Linear Regression
• Both the response and explanatory variable are
quantitative
• Graphical Summary
‒ Scatter plot
• Numerical Summary
‒ Correlation
‒ R2
‒ Regression equation
‒ Response = ¯0 + ¯1 ¢ explanatory
• Test of significance
‒ Test significance of regression equation coefficients
January 6, 2009 - morni 3
ng session
Scatter plot
• Shows relationship between two
quantitative variables
‒ y-axis = response variable
‒ x-axis = explanatory variable
January 6, 2009 - morni 4
ng session
Correlation and R2
• Correlation indicates the strength and
direction of the linear relationship between
two quantitative variables
‒ Values between -1 and +1
• R2 is the fraction of the variability in the
response that can be explained by the linear
relationship with the explanatory variable
‒ Values between 0 and +1
• Correlation2 = R2
• Large values of each depend on the field
January 6, 2009 - morni 5
ng session
Linear Regression Equation
• Linear Regression Equation
‒ Response = ¯0 + ¯1 * explanatory
‒ ¯0 is the intercept
‒ the value of the response variable when the
explanatory variable is 0
‒ ¯1 is the slope
‒ For each 1 unit increase in the explanatory
variable, the response variable increases by ¯1
• ¯0 and ¯1 are most often found using
least squares estimation
January 6, 2009 - morni 6
ng session
Assumptions of linear regression
• Linearity
‒ Check my looking at either observed vs. predicted or
residual vs. predicted plot
‒ If non-linear, predictions will be wrong
• Independence of errors
‒ Can often be checked by knowing how data was
collected. If not sure can use autocorrelation plots.
• Homoscedasticity (constant variance)
‒ Look at residuals versus predicted plot
‒ If non-constant variance predictions will have wrong
confidence intervals and estimated coefficients may be
wrong
• Normality of errors
‒ Look at normal probability plot
‒ If non-normal confidence intervals and estimated
coefficients will be wrong
January 6, 2009 - morni 7
ng session
Assumptions of linear regression
• If the assumptions are not met, the
estimates of ¯0, ¯1, their standard
deviations, and estimates of R2 will be
incorrect
• Maybe possible to do transformations to
either the explanatory or response
variable to make the relationship linear
January 6, 2009 - morni 8
ng session
Hypothesis testing
• Want to test if there is a significant
linear relationship between the variables
‒ H0 = there is no linear relationship between
the variables (¯1 = 0)
‒ H1 = there is a linear relationship between
the variables (¯1 ≠ 0)
• Testing ¯0 = 0 may or may not be
interesting and/or valid
January 6, 2009 - morni 9
ng session
Monday’s Example
• Curious if typing speed (words per
minute) affects efficiency (as measured
by number of minutes required to finish
a paper)
• Graphical display
January 6, 2009 - morni 10
ng session
Sample Output
• Below is sample output for this
regression
January 6, 2009 - morni 11
ng session
Numerical Summary
• Numerical summary
‒ Correlation = -0.946
‒ R2 = 0.8944
‒ Efficiency = 85.99 – 0.52*speed
• For each additional word per minute
typed, the number of minutes needed to
complete an assignment decreases by
0.52 minutes
• The intercept does not make sense since
it corresponds to a speed of zero words
per minute
January 6, 2009 - morni 12
ng session
Interpretation of r and R2
• r = -0.946
‒ This indicates a strong negative linear
relationship
• R2 = 89.44
‒ 89.44% of the variability in efficiency can be
explained by words per minute typed
January 6, 2009 - morni 13
ng session
Hypothesis test
• To test the significance of ¯1
‒ H0 = there is no linear relationship between
the speed and efficiency (¯1 = 0)
‒ H1 = there is a linear relationship between
the speed and efficiency (¯1 ≠ 0)
• Test statistic: t = -20.16
• P-value = 0.000
• In this case, testing ¯0 = 0 is not
interesting; however it may be in some
experiments
January 6, 2009 - morni 14
ng session
Checking Assumptions
• Checking assumptions
‒ Plot on left: residual vs. predicted
‒ Want to see no pattern
‒ Plot on right: normal probability plot
‒ Want to see points fall on line
January 6, 2009 - morni 15
ng session
Another Example
• Suppose we have an explanatory and
response variable and would like to
know if there is a significant linear
relationship
• Graphical display
January 6, 2009 - morni 16
ng session
Numerical Summary
• Numerical summary
‒ Correlation = 0.971
‒ R2 = 0.942
‒ Response = -21.19 + 19.63*explanatory
• For each additional unit of the
explanatory variable, the response
variable increases by 19.63 minutes
• When the explanatory variable has a
value of 0, the response variable has a
value of -21.19
January 6, 2009 - morni 17
ng session
Hypothesis testing
• To test the significance of ¯1
‒ H0 = there is no linear relationship between
the explanatory and response (¯1 = 0)
‒ H1 = there is a linear relationship between
the explanatory and response (¯1 ≠ 0)
• Test statistic: t = 49.145
• P-value = 0.000
• It appears as though there is a
significant linear relationship between
the variables
January 6, 2009 - morni 18
ng session
Sample Output
• Sample output for this example, we can
see both coefficients are highly
significant
January 6, 2009 - morni 19
ng session
Checking Assumptions
• Checking assumptions
‒ Plot on left: residual vs. predicted
‒ Want to see no pattern
‒ Plot on right: normal probability plot
‒ Want to see points fall on line
January 6, 2009 - morni 20
ng session
Example 6 (cont)
• Checking assumptions
‒ In the residual vs. predicted plot we see that the
residual values are higher for lower and higher
predicted values and lower for values in the middle
‒ In the normal probability plot we see that the
points are falling off the lines at the two ends
• This indicates that one of the assumptions was
not met!
• In this case the is a quadratic relationship
between the variables
• With experience you’ll be able to determine what
relationships are present given the residual versus
predicted plot
January 6, 2009 - morni 21
ng session
Data with Linear Prediction Line
• When we add the predicted linear
relationship, we can clearly see misfit
January 6, 2009 - morni 22
ng session
Multiple Linear Regression
• Use more than one explanatory variable
to explain the variability in the response
variable
• Regression Equation
‒ Y = ¯0 + ¯1¢X1 + ¯2¢X2 + . . . + ¯N¢XN
• ¯j is the change in the response variable
(Y) when Xj increases by 1 unit and all
the other explanatory variables remain
fixed
January 6, 2009 - morni 23
ng session
Exploratory Analysis
• Graphical Display
‒ Look at the scatter plot of the response
versus each of the explanatory variables
• Numerical Summary
‒ Look at the correlation matrix of the response
and all of the explanatory variables
January 6, 2009 - morni 24
ng session
Assumptions of Multiple Linear
Regression
• Same as simple linear regression!
‒ Linearity
‒ Independence of errors
‒ Homoscedasticity (constant variance)
‒ Normality of errors
• Methods of checking assumptions are
also the same
January 6, 2009 - morni 25
ng session
R2adj
• R2 is the fraction of the variation in the
response variable that can be explained by
the model
• When variables are added to the model, R2
will increase or stay the same (it will not
decrease!)
‒ Use R2adj which adjusts for the number of
variables
‒ Check to see if there is a significant increase
• R2adj is a measure of the predictive power of
our model, how well do the explanatory
variables collectively predict the response
January 6, 2009 - morni 26
ng session
Inference in Multiple Regression
• Step 1
‒ Does the data provide evidence that any of
the explanatory variables are important in
predicting Y?
‒ No – none of the variables are important, the
model is useless
‒ Yes – at least one variable is important, move
to step 2
• Step 2
‒ For each explanatory variable Xj: does the
data provide evidence that Xj has a significant
linear effect with Y, controlling for all the
other variables
January 6, 2009 - morni 27
ng session
Step 1
• Test the overall hypothesis that at least
one of the variables is needed
‒ H0: none of the explanatory variables are
important in predicting the response variable
‒ H1: at least one of the explanatory variables
is important in predicting the response
variable
• Formally done with an F-test
‒ We will skip the calculation of the F-statistic
and p-value as they are given in output
January 6, 2009 - morni 28
ng session
Step 2
• If H0 is rejected, test the significance of
each of the explanatory variables in the
presence of all of the other explanatory
variables
• Perform a T-test for the individual
effects
‒ H0: Xj is not significant to the model
‒ H1: Xj is significant to the model
January 6, 2009 - morni 29
ng session
Example
• Earlier we looked at how typing speed
and efficiency are linearly related
• Now we want to see if adding GPA (on a
0-5 point scale) as an explanatory
variable will make the model more
predictive of efficiency
January 6, 2009 - morni 30
ng session
Graphical displays
January 6, 2009 - morni 31
ng session
Numerical Summary
Efficienc Words per GPA
y minute
Efficiency 1.00 -0.95 -0.92
Words per 1.00 0.96
minute
GPA 1.00
January 6, 2009 - morni 32
ng session
Sample Output
January 6, 2009 - morni 33
ng session
Step 1 – Overall Model Check
• For our example with words per minute
and GPA, the F-test yields
‒ F-statistic: 207.4
‒ P-value = 0.0000
• Interpretation, at least one of the
variables (words per minute and GPA)
are important in predicting efficiency
January 6, 2009 - morni 34
ng session
Step 2
• Test significance of words per minute
‒ T-statistic: -4.67
‒ P-value = 0.0000
• Test significance of GPA
‒ T-statistic: -1.33
‒ P-value = 0.1900
• Conclusions
‒ Words per minute is significant but GPA is not
‒ In this case we ended up with a simple linear
regression with words per minute as the only
explanatory variable
January 6, 2009 - morni 35
ng session
Looking at R2adj
• R2adj (wpm and GPA) = 89.39
• R2adj (wpm) = 89.22
• Adding GPA to the model only
raised the R2adj by 0.17%, not
nearly enough to justify adding
GPA to the model
‒ This agrees with the hypothesis
testing on the previous page
January 6, 2009 - morni 36
ng session
Automatic methods
• Model Selection – compare models to
determine which best fits the data
• Uses one of several criteria (R2adj, AIC
score, BIC score) to compare models
• Often use stepwise regression
‒ Start with no variables, add variables one at
a time until there is no significant change in
the selection criteria
‒ Start with all variables, remove variables one
at a time until there is no significant change
in the selection criteria
• Packages have built in methods for this
January 6, 2009 - morni 37
ng session
Multicollinearity
• Collinearity refers to the linear
relationship between two explanatory
variables
• Multicollinearity is more general and
refers to the linear relationship between
two or more explanatory variables
January 6, 2009 - morni 38
ng session
Multicollinearity
• Perfect multicollinearity – one of the
variables is a perfect linear function of
other explanatory variables, one of the
variables must be dropped
‒ Example: using both inches and feet
• Near-perfect multicollinearity – occurs
when there are strong, but not perfect
linear relationships among the
explanatory variable
‒ Example: Height and arm spread
January 6, 2009 - morni 39
ng session
Collinearity Example
• An instructor wants to predict final exam
grade and has the following explanatory
variables
‒ Midterm 1
‒ Midterm 2
‒ Diff = Midterm 2 – Midterm 1
• Diff is a perfect linear function of
Midterm 1 and Midterm 2
‒ Drop diff from the model
‒ Use Diff but neither Midterm 1 or Midterm 2
January 6, 2009 - morni 40
ng session
Indicators of Multicollinearity
• Moderate to high correlations among the
explanatory variables in the correlation
matrix
• The estimates of the regression
coefficients have surprising and/or
counterintuitive values
• Highly inflated standard errors
January 6, 2009 - morni 41
ng session
Indicators of Multicollinearity
• The correlation matrix alone isn’t always
enough
• Can calculate the tolerance, a more
reliable measure of multicollinearity
‒ Run the regression with Xj as the response
versus the rest of the explanatory variables
‒ Let R2j be the be the R2 value from this
regression
‒ Tolerance (Xj) = 1 – R2j
‒ Variance Inflation Factor (VIF)= 1/Tolerance
• Do more checking if the tolerance is less
than 0.20 or VIF is greater than 5
January 6, 2009 - morni 42
ng session
Back to Example
• Use GPA as the response and words per
minute as the explanatory
‒ R2 = 0.91
‒ Tolerance (GPA) = 0.09
‒ Well below 0.30!
• Adding GPA to the regression equation
does not add to the predictive power of
the model
January 6, 2009 - morni 43
ng session
What can be done?
• Drop the correlated variables!
• Interpretations of coefficients will be
incorrect if you leave all variables in the
regression.
• Do model selection (same as that on
slide 37)
January 6, 2009 - morni 44
ng session
Example
• Suppose we have an online math tutor and
classroom performance variables and we’d like
to predict final exam scores.
• Math tutor variables
‒ Time spent on the tutor (minutes)
‒ Number of problems solved correctly
• Classroom variable
‒ Pre-test score
• Response variable
‒ Final exam score
January 6, 2009 - morni 45
ng session
Example
•Exploratory analysis – correlation matrix
‒ The correlation between pretest and number
correct seems high
Final Pretest Number Time
Score Correct
Final Score 1.00 0.85 0.82 0.37
Pretest 1.00 0.90 0.01
Number 1.00 0.03
Correct
Time 1.00
January 6, 2009 - morni 46
ng session
Example
•Exploratory analysis
‒ linear relationship between time and final
is not strong
January 6, 2009 - morni 47
ng session
Example
• Run the linear regression using pretest,
number correct, and time as linear
predictors of final score
January 6, 2009 - morni 48
ng session
Step 1
• Test the overall hypothesis that at least one of
the variables is needed
‒ H0: none of the explanatory variables are
important in predicting the response variable
‒ H1: at least one of the explanatory variables is
important in predicting the response variable
• F-statistic = 95.56
• P-value = 0.0000
• At least one of the three explanatory variables is
important in predicting final exam score
January 6, 2009 - morni 49
ng session
Step 2
• Test significance of pretest score
‒ T-statistic: 4.88
‒ P-value = 0.0000
• Test significance of number correct
‒ T-statistic: 1.99
‒ P-value = 0.0524
• Test significance of time
‒ T-statistic: 6.45
‒ P-value = 0.0000
• Conclusions
‒ Pretest score and time are significant but number
correct is not
January 6, 2009 - morni 50
ng session
Example
• This is not surprising given the high
correlation (0.90) between pretest score
and number correct
• Formally show
‒ Number Correct ~ Pretest + Time
‒ R2 = 0.8044
‒ Tolerance = 1 – 0.8044 = 0.1956
‒ Lower than 0.20
‒ VIF = 1/0.1956 = 5.11
‒ VIF is greater than 5
January 6, 2009 - morni 51
ng session
Model Selection
• Why was test number correct and not pretest
chosen as insignificant? Depends on which
variable adds more to the predictive power of
the regression equation
• Doing stepwise regression will yield more
information
• Depending on the criteria used, some model
selection procedures dropped number correct
and others kept all three variables
‒ If we decide to drop number correct we will have
to rerun the regression
January 6, 2009 - morni 52
ng session
Rerunning the regression
• New output
January 6, 2009 - morni 53
ng session
Steps 1 and 2
• Step 1
‒ F-statistic = 133
‒ P-value = 0.0000
• Step 2
‒ Test significance of pretest score
‒ T-statistic: 14.93
‒ P-value = 0.0000
‒ Test significance of time
‒ T-statistic: 6.34
‒ P-value = 0.0000
January 6, 2009 - morni 54
ng session
Example
• Conclusion – both pretest score and time
are important predictors of final exam
score
• R2adj = 84.34
‒ 84% of the variability in final exam score is
explained by pretest score and time
January 6, 2009 - morni 55
ng session
Check Assumptions
• There may be a slight pattern to the
residual vs. fitted plot, but overall the
plots look good
January 6, 2009 - morni 56
ng session
Interpretation
• The final regression equation is:
Final - 8.16 0.59 pretest 0.29 time
• For each additional point on the pretest, a
student’s predicted final exam score increases by
0.59 points, holding time on the tutor constant
• For each additional minute on the tutor, a
student’s predicted final exam score increases by
0.29 points, holding pretest score constant
January 6, 2009 - morni 57
ng session
Notes on Example
• If either pretest or time was found to be
non-significant, we would have rerun the
regression again
• Multiple regression often takes several
regressions before we are done
• The built in automatic model selection in
statistical packages will do these in one
step!
January 6, 2009 - morni 58
ng session
Alternate Ending
• What if we had dropped pretest instead of
number correct?
• The regression equation would be:
Final 12.58 0.43 number correct 0.29 time
January 6, 2009 - morni 59
ng session
Steps 1 and 2
• Step 1
‒ F-statistic = 88.52
‒ P-value = 0.0000
• Step 2
‒ Test significance of number correct score
‒ T-statistic: 12.09
‒ P-value = 0.0000
‒ Test significance of time
‒ T-statistic: 5.19
‒ P-value = 0.0000
January 6, 2009 - morni 60
ng session
Check the Assumptions
• On the residual vs. predicted there is a
slight pattern. I’d recommend dropping
the outlier and rerunning the regression.
January 6, 2009 - morni 61
ng session
Notes
• We can see that both pretest and time
are significant but that the assumptions
might be questionable
• However, when the R2adj of this model
with the previous model we see the
different
‒ R2adj (pretest, time) = 84.34
‒ R2adj (Number correct, time) = 78.13
• This model with pretest describes more of
the variability in final exam scores
January 6, 2009 - morni 62
ng session
Another Example
•Suppose we have 4 explanatory variables (X1, X2,
X3, X4) and we have our response variable Y
•X1 and X3 appear to be highly correlated
Y X1 X2 X3 X4
Y 1.00 -0.36 0.76 -0.38 0.54
X1 1.00 -0.33 0.98 0.09
X2 1.00 -0.34 -0.12
X3 1.00 0.08
X4 1.00
January 6, 2009 - morni 63
ng session
Exploratory Analysis
• Appears reasonable that each of the 4
explanatory variables may have a linear
relationship with the response variable
January 6, 2009 - morni 64
ng session
Example
• Start by running the regression with all
four explanatory variables
January 6, 2009 - morni 65
ng session
Steps 1 and 2
• Step 1
‒ F-statistic = 1900
‒ P-value = 0.0000
• Step 2
‒ Test significance of X1
‒ T-statistic: -9.04
‒ P-value = 0.0000
‒ Test significance of X2
‒ T-statistic: 207.21
‒ P-value = 0.0000
‒ Test significance of X3
‒ T-statistic: 0.88
‒ P-value = 0.3817
‒ Test significance of X4
‒ T-statistic: 181.57
‒ P-value = 0.0000
January 6, 2009 - morni 66
ng session
Conclusions
• Variable X3 is not significant in predicting
Y
• Calculate the tolerance for X3
‒ X3 ~ X1 + X2 + X4
‒ R2 = 0.96
‒ Tolerance = 0.04
‒ VIF = 25
• Remove X3 from the regression and
rerun!
January 6, 2009 - morni 67
ng session
Updated Regression
• R2adj= 99.94
‒ Note that the R2adj is the same as the
regression with all four variables
January 6, 2009 - morni 68
ng session
Steps 1 and 2
• Step 1
‒ F-statistic = 2675
‒ P-value = 0.0000
• Step 2
‒ Test significance of X1
‒ T-statistic: -42.62
‒ P-value = 0.0000
‒ Test significance of X2
‒ T-statistic: 208.82
‒ P-value = 0.0000
‒ Test significance of X4
‒ T-statistic: 181.46
‒ P-value = 0.0000
January 6, 2009 - morni 69
ng session
Things to Note
• When we reran the regression without X3,
the changes in the regression equation
and step 2 of the analysis were mostly to
X1
• This is not surprising since it was X1 and
X3 which were highly correlated
January 6, 2009 - morni 70
ng session
Check Assumptions
• I would probably delete the low two
observations in the residual vs. fitted plot
and rerun
January 6, 2009 - morni 71
ng session
After removing observations
•Step 1 significant
•All three variables significant in Step 2
Y 16.51 4.98 X1 9.96 X 2 15.02 X 4
January 6, 2009 - morni 72
ng session
Outliers
• Removing observations in a linear
regression is often subjective
• Many packages will indicate observations
which are possible outliers
• Running a regression with and without
the observations and comparing them is
best
January 6, 2009 - morni 73
ng session