Chapter 4: Multiple Regression Analysis – Inference
Introductory Econometrics
Taehoon Kim
Chapter Objectives
By the end of this chapter, you should be able to:
4.1 Explain the classical linear model assumptions, and why normality is needed for
exact inference.
4.2 Describe the sampling distribution of OLS estimators and their standardized
versions.
4.3 State a null hypothesis for a population parameter and distinguish one–sided vs.
two–sided alternatives.
Chapter Objectives
4.4 Write down the formula for a t statistic for a given null.
4.5 Use software or tables to obtain critical values for one– and two–tailed t tests.
4.6 Record and interpret p–values from regression output.
4.7 Construct and interpret confidence intervals for parameters.
Chapter Objectives
4.8 Distinguish practical (economic) from statistical significance.
4.9 Express a null involving several parameters.
4.10 Explain the purpose of multiple–hypothesis tests, especially exclusion restrictions.
4.11 Produce SSR and R 2 forms of the F statistic.
Statistical Inference and Sampling Distributions
In multiple regression we want to:
▶ Test hypotheses about population parameters.
▶ Construct confidence intervals.
We already know:
▶ E(β̂j ) and Var(β̂j ) under MLR.1–MLR.5.
For exact tests we also need the sampling distribution of the estimators, which requires
an additional assumption about ui .
Assumption MLR.6 (Normality)
MLR.6 (Normality of the error terms)
ui ∼ N (0, σ 2 ) independently of xi1 , . . . , xik , i = 1, . . . , n.
▶ Unobserved factors are normally distributed around the population regression
function.
▶ The form and variance of the error distribution do not depend on the explanatory
variables.
Together with MLR.1–MLR.5 these are the classical linear model (CLM) assumptions.
Discussion of the Normality Assumption
Arguments for approximate normality:
▶ The error term is often a sum of many small, independent shocks.
▶ By the central limit theorem, such sums tend toward normality.
Concerns:
▶ How many factors enter u and how independent are they?
▶ Individual components may be very heterogeneous.
Discussion of the Normality Assumption
▶ Normality of ui is an empirical question; we usually hope the distribution is “close
enough”.
▶ In some applications, normality is impossible:
▶ Wages or prices (nonnegative, often bounded below).
▶ Counts such as number of arrests.
▶ Binary outcomes such as unemployment status.
▶ Transformations like log(y ) can make normality more plausible.
▶ Under CLM, OLS is not only BLUE but also the best (even among nonlinear)
unbiased estimator.
Normal Sampling Distributions
Under MLR.1–MLR.6:
β̂j ∼ N βj , Var(β̂j ) ,
and the standardized estimator
β̂j − βj
zj = ∼ N (0, 1).
sd(β̂j )
Here sd(β̂j ) uses the true error variance σ 2 .
t Distribution for Standardized Estimators (Theorem 4.2)
In practice we replace σ 2 by σ̂ 2 , so
q
se(β̂j ) = Var(
d β̂j )
is used. Under MLR.1–MLR.6:
β̂j − βj
tβj = ∼ tn−k−1 .
se(β̂j )
▶ The t distribution is centered at zero with heavier tails than the standard normal.
▶ As n − k − 1 becomes large, tn−k−1 approaches N (0, 1).
Null and Alternative Hypotheses
Most often we test whether a coefficient is zero:
H0 : β j = 0 vs. H1 : βj ̸= 0 (two–sided)
or a one–sided alternative:
H1 : βj > 0 or H1 : βj < 0.
Interpretation (example):
▶ H0 : βj = 0: after controlling for other regressors, xj has no effect on y .
The t Statistic
For H0 : βj = 0 the t statistic is
β̂j
tβj = .
se(β̂j )
More generally, for H0 : βj = aj :
β̂j − aj
t= .
se(β̂j )
If H0 is true, then
t ∼ tn−k−1 .
The t statistic measures how many estimated standard errors the estimate lies away
from the null value.
One–Sided Test: βj > 0
For H0 : βj = 0 vs. H1 : βj > 0:
▶ Choose significance level α (often 0.05).
▶ Find critical value cα such that
P tn−k−1 > cα = α.
▶ Reject H0 if tβj > cα .
Example (illustrative): with n − k − 1 = 28, a 5% one–sided test uses c0.05 ≈ 1.701;
reject if t > 1.701.
Example: Wage Equation (1 of 2)
Estimate:
log(wage) = 0.284 + 0.092 educ + 0.0041 exper + 0.022 tenure,
(0.104) (0.007) (0.0017) (0.003),
with n = 526 and R 2 = 0.316.
Test H0 : βexper = 0 vs. H1 : βexper > 0.
Example: Wage Equation (2 of 2)
0.0041
texper = ≈ 2.41.
0.0017
Degrees of freedom:
df = n − k − 1 = 526 − 3 − 1 = 522.
Critical values (approximate):
c0.05 ≈ 1.645, c0.01 ≈ 2.326.
Because 2.41 > 1.645, we reject H0 at the 5% (and even the 1%) level: experience has
a positive and statistically significant effect on wages.
One–Sided Test: βj < 0
For H0 : βj = 0 vs. H1 : βj < 0:
▶ Find cα such that
P tn−k−1 < cα = α.
▶ Reject H0 if tβj < cα .
Example: with 18 degrees of freedom and α = 0.05, c0.05 ≈ −1.734; reject if
t < −1.734.
Example: Student Performance and School Size
Model:
math10 = 2.2774 + 0.00046 totcomp + 0.048 staff − 0.00020 enroll,
(6.113) (0.00010) (0.040) (0.00022),
with n = 408, R 2 = 0.0541.
Test whether smaller school size improves performance:
H0 : βenroll = 0 vs. H1 : βenroll < 0.
Example: Student Performance and School Size
Compute
−0.00020
tenroll = ≈ −0.91.
0.00022
Degrees of freedom:
df = n − k − 1 = 408 − 3 − 1 = 404.
Critical values:
c0.05 ≈ −1.65, c0.15 ≈ −1.04.
Since t = −0.91 is not less than c0.15 , we cannot reject H0 even at the 15% level: no
clear effect of school size in this linear specification.
Example: Student Performance and School Size
Alternative specification:
math10 = −207.66 + 21.16 log(totcomp) + 3.98 log(staff ) − 1.29 log(enroll),
(47.80) (4.06) (1.69) (0.69),
with n = 408, R 2 = 0.0654.
Test
H0 : βlog(enroll) = 0 vs. H1 : βlog(enroll) < 0.
Example: Student Performance and School Size
−1.29
tlog(enroll) = ≈ −1.87.
0.69
With df = 404, c0.05 ≈ −1.65.
Since −1.87 < −1.65, we reject H0 at the 5% level: larger school size has a
statistically significant negative effect on test scores in this log specification.
A 10% increase in enrollment reduces the pass rate by about
−0.129 × 10% = −1.29 percentage points.
Two–Sided Alternatives
For
H0 : β j = 0 vs. H1 : βj ̸= 0
we reject when |tβj | is large:
▶ Choose α, find critical value cα/2 such that
P |tn−k−1 | > cα/2 = α.
▶ Reject H0 if |tβj | > cα/2 .
Example: with df = 40 and α = 0.05, c0.025 ≈ 2.02; reject if |t| > 2.02.
Example: Determinants of College GPA
Regression:
colGPA = 1.39 + 0.412 hsGPA + 0.015 ACT − 0.083 skipped,
(0.33) (0.094) (0.011) (0.026),
with n = 141, R 2 = 0.234.
Two–sided tests:
0.412 0.015 −0.083
thsGPA = ≈ 4.38, tACT = ≈ 1.36, tskipped = ≈ −3.19.
0.094 0.011 0.026
At the 1% level, hsGPA and skipped are statistically significant; ACT is not even at
10%.
“Statistically Significant” Variables
If |tβj | exceeds the critical value in a two–sided test, xj is said to be statistically
significant.
With large df and using the standard normal:
|t| > 1.645 ⇒ significant at 10% level,
|t| > 1.96 ⇒ significant at 5% level,
|t| > 2.576 ⇒ significant at 1% level.
General Hypotheses about a Coefficient
To test
H0 : βj = aj vs. H1 : βj ̸= aj ,
use
β̂j − aj
t= .
se(β̂j )
The procedure is the same as before, except that we subtract aj rather than 0.
Example: Campus Crime and Enrollment
Model:
log(crime) = −6.63 + 1.27 log(enroll),
(1.03) (0.11),
with n = 97, R 2 = 0.585.
Hypothesis: a 1% increase in enrollment increases crime by 1%:
H0 : βlog(enroll) = 1 vs. H1 : βlog(enroll) ̸= 1.
1.27 − 1
t= ≈ 2.45.
0.11
With df = 95, c0.025 ≈ 1.985, so we reject H0 at 5%: the elasticity is significantly
larger than one.
p-values for t Tests
▶ The p–value is the smallest significance level at which H0 would still be rejected.
▶ For a two–sided test,
p–value = P |tdf | > |tobs | .
▶ A small p–value is strong evidence against H0 ; a large p–value is evidence in favor
of H0 .
▶ H0 is rejected at any α such that p–value < α.
Example: if |tobs | = 1.85 and P(tdf > 1.85) = 0.0359, then
p–value = 2 × 0.0359 = 0.0718.
Economic vs. Statistical Significance
▶ A statistically significant coefficient may be economically trivial.
▶ Always discuss the magnitude and units (or percentage effect) of the estimate.
▶ If a variable is statistically and economically important but has an unexpected
sign, reconsider the model specification.
▶ An insignificant coefficient may be due to small sample size or multicollinearity;
think before dropping variables.
Confidence Intervals
From Theorem 4.2, a 95% confidence interval for βj is
β̂j ± c0.025 se(β̂j ),
where c0.025 is the 2.5% upper critical value of tn−k−1 .
Interpretation:
▶ The interval endpoints are random.
▶ In repeated samples, 95% of such intervals will contain the true βj .
Confidence Intervals
For typical levels:
99% CI: β̂j ± c0.005 se(β̂j ),
95% CI: β̂j ± c0.025 se(β̂j ),
90% CI: β̂j ± c0.05 se(β̂j ).
Relationship with tests:
▶ H0 : βj = aj is rejected at level α in a two–sided test iff aj lies outside the
corresponding (1 − α) confidence interval.
Example: Firms’ R&D Expenditures
Model:
log(rd) = 4.38 + 1.084 log(sales) + 0.0217 profmarg ,
(0.47) (0.060) (0.0128),
with n = 32, R 2 = 0.918 and df = 29.
95% confidence intervals (using c0.025 ≈ 2.045):
1.084 ± 2.045(0.060) = [0.961, 1.21],
0.0217 ± 2.045(0.0128) = [−0.0045, 0.0479].
Sales elasticity is precisely estimated and clearly > 0; the profit–margin effect is
imprecise and not significantly different from zero.
Linear Combinations of Parameters
Example: returns to education at two–year vs. four–year colleges:
log(wage) = β0 + β1 jc + β2 univ + β3 exper + u.
We want to test whether the return to four–year college exceeds that of two–year:
H0 : β 1 − β 2 = 0 vs. H1 : β1 − β2 < 0.
A natural statistic is
β̂1 − β̂2
t= .
se(β̂1 − β̂2 )
Linear Combinations of Parameters
The variance of the difference is
q
se(β̂1 − β̂2 ) = Var(β̂1 ) + Var(β̂2 ) − 2 Cov(β̂1 , β̂2 ).
Software often does not report Cov(β̂1 , β̂2 ) directly.
Alternative: define a new regressor
totcoll = jc + univ
and rewrite:
log(wage) = β0 + θ1 jc + β2 totcoll + β3 exper + u,
where θ1 = β1 − β2 . Then test H0 : θ1 = 0 using the usual t statistic on jc.
Estimation Results for the College Example
Estimated regression:
log(wage) = 1.472 − 0.0102 jc + 0.0769 totcoll + 0.0049 exper ,
(0.021) (0.0069) (0.0023) (0.0002),
with n = 6,763, R 2 = 0.222.
The t statistic for jc:
−0.0102
t= ≈ −1.48.
0.0069
This is not significant at the 5% level (two–sided); the 95% CI is
−0.0102 ± 1.96(0.0069) = [−0.0237, 0.0003].
Testing Multiple Linear Restrictions: The F Test
Example: baseball player salaries:
log(salary ) = β0 + β1 years + β2 gamesyr + β3 bavg + β4 hrunsyr + β5 rbisyr + u.
Test that performance measures have no effect:
H0 : β3 = 0, β4 = 0, β5 = 0 (q = 3 restrictions).
Unrestricted and Restricted Models
Unrestricted model (full regression) gives SSR:
SSRur = 183.186, n = 353, R 2 = 0.6278.
Restricted model (dropping bavg , hrunsyr , rbisyr ):
log(salary ) = 11.22 + 0.0713 years + 0.0222 gamesyr + u,
with
SSRr = 198.311.
The F statistic:
(SSRr − SSRur )/q
F = ∼ Fq, n−k−1 under H0 .
SSRur /(n − k − 1)
Rejection Rule for the F Test
▶ An F variable is always nonnegative; larger values indicate stronger evidence
against H0 .
▶ For significance level α and degrees of freedom (q, n − k − 1), choose cα such that
P(Fq,n−k−1 > cα ) = α.
▶ Reject H0 if F > cα .
Example: in the baseball regression,
F ≈ 9.55,
and with q = 3, n − k − 1 = 347,
F3,347;0.01 ≈ 3.78.
We overwhelmingly reject H0 even at the 1% level.
Overall Significance of a Regression
For
y = β0 + β1 x1i + · · · + βk xki + ui ,
to test
H0 : β1 = β2 = · · · = βk = 0,
the restricted model is y = β0 + u (regression on constant only).
The F statistic can be written as
(SSRr − SSRur )/k R 2 /k
F = = 2
.
SSRur /(n − k − 1) (1 − R )/(n − k − 1)
Testing General Linear Restrictions with F
Example: rationality of house price assessments:
log(price) = β0 + β1 log(assess) + β2 log(lotsize) + β3 log(sqrft) + β4 bdrms + u.
Hypothesis:
H0 : β1 = 1, β2 = 0, β3 = 0, β4 = 0,
i.e., conditional on assessed value, other housing characteristics should not matter if
assessments are rational.
Testing General Linear Restrictions with F
Unrestricted regression: full model above, with
SSRur = 1.822.
Restricted regression:
log(price) = β0 + β1 log(assess) + u,
i.e., regress log(price) on log(assess) only; here
SSRr = 1.880.
F statistic:
(SSRr − SSRur )/q (1.880 − 1.822)/4
F = = ≈ 0.661.
SSRur /(n − k − 1) 1.822/(88 − 4 − 1)
With q = 4, df2 = 83, F4,83;0.05 ≈ 2.50; since 0.661 < 2.50, we cannot reject H0 .
Summary
▶ Added the normality assumption (MLR.6) to obtain the classical linear model.
▶ Derived the normal and t sampling distributions of OLS estimators.
▶ Constructed and interpreted t tests, p–values, and confidence intervals.
▶ Distinguished statistical significance from economic significance.
▶ Tested hypotheses about single coefficients, linear combinations, and sets of
coefficients using t and F statistics.
▶ Expressed the overall significance test in terms of R 2 .