Relationship between Simple and Multiple Regression
We consider four OLS regressions:
Short Regression: Y on X.
→ The fitted line is: 𝑌̂𝑠ℎ𝑜𝑟𝑡 = 𝑎𝑠ℎ𝑜𝑟𝑡 + 𝑏𝑋,𝑠ℎ𝑜𝑟𝑡 𝑋
→ The residual is: 𝑢̂𝑠ℎ𝑜𝑟𝑡 = 𝑌 − 𝑌̂𝑠ℎ𝑜𝑟𝑡 .
Long Regression: Y on X and Z
→ The fitted line is: 𝑌̂𝑙𝑜𝑛𝑔 = 𝑎𝑙𝑜𝑛𝑔 + 𝑏𝑋,𝑙𝑜𝑛𝑔 𝑋 + 𝑏𝑧 𝑍
Auxiliary Regression: Z on X yields slope c, and residuals 𝑢̂𝑎𝑢𝑥 .
Residual Regression: 𝑢̂𝑠ℎ𝑜𝑟𝑡 on 𝑢̂𝑎𝑢𝑥 yields slope d.
1
Result 1 (Omitted Variable Rule):
The slope of “Short” is related to the slopes of “Long” via:
(OVR) 𝑏𝑋,𝑠ℎ𝑜𝑟𝑡 = 𝑏𝑋,𝑙𝑜𝑛𝑔 + c𝑏𝑧 .
Remark: That 𝑏𝑋,𝑠ℎ𝑜𝑟𝑡 ≠ 𝑏𝑋,𝑙𝑜𝑛𝑔 should not come as a surprise
because while 𝑏𝑋,𝑠ℎ𝑜𝑟𝑡 measures ΔY/ΔX (a “total” derivative),
𝑏𝑋,𝑙𝑜𝑛𝑔 measures ∂Y/∂X (a “partial” derivative), with Z held
constant.
Result 2 (Residual Regression Rule):
(RRR) d = bz.
This is a special case of the Frisch-Waugh Theorem (S&W Apdx. 6.3).
2
Verification of the OVR in the California School example:
With Y = TestScore, X = STR, Z = PctEL:
Short Regression: Y on X yields slope 𝑏𝑋,𝑠ℎ𝑜𝑟𝑡 = –2.28.
Long Regression: Y on X and Z yields a pair of slopes,
bX,long = – 1.10 and bz = – 0.65.
Auxiliary Regression: Z on X yields slope c = 1.81.
(OVR) 𝑏𝑋,𝑠ℎ𝑜𝑟𝑡 = bX,long + cbz
–2.28 = – 1.10 + (1.81)(– 0.65)
Please verify! (Equality is not exact because of rounding.)
We see that when PctEL is omitted, STR takes the credit for its
negative effect on test scores (has a “bigger” negative coefficient).
3
Verification of the RRR in the California School example:
With Y = TestScore, X = STR, Z = PctEL:
* Long regression of Y on X and Z:
. reg testscr str pctel
------------------------------------------------------------------------------
testscr | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
str | -1.101296 .3802783 -2.90 0.004 -1.848797 -.3537945
pctel | -.6497768 .0393425 -16.52 0.000 -.7271112 -.5724423
_cons | 686.0322 7.411312 92.57 0.000 671.4641 700.6004
------------------------------------------------------------------------------
* Short regression of Y on X:
. reg testscr str
------------------------------------------------------------------------------
testscr | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
str | -2.279808 .4798256 -4.75 0.000 -3.22298 -1.336637
_cons | 698.933 9.467491 73.82 0.000 680.3231 717.5428
------------------------------------------------------------------------------
. predict resshort, residuals
4
. * Auxiliary regression of Z on X:
. reg PctEL str
------------------------------------------------------------------------------
PctEL | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
str | 1.813719 .4643735 3.91 0.000 .9009206 2.726517
_cons | -19.85405 9.162604 -2.17 0.031 -37.86458 -1.843531
------------------------------------------------------------------------------
. predict resaux, residuals
. * Residual regression:
. reg resshort resaux
------------------------------------------------------------------------------
resshort | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
resaux | -.6497768 .0392955 -16.54 0.000 -.7270181 -.5725354
_cons | -2.28e-09 .7049498 -0.00 1.000 -1.385689 1.385689
------------------------------------------------------------------------------
5
Result 2 - Residual Regression Rule:
(RRR) d = bz = – 0.65.
As we saw, the RRR suggests a way computing a multiple
regression slope in steps. Since one of these steps is “short,” this
approach illustrates what “long” achieves. In what follows we
reexamine the RRR in this broader context.
What Multiple Regression achieves:
The switch from SR to MR is an important step that offers hope in
our quest to discover the causal effect (i.e., unbiased/consistent
estimate) of STR on TestScores.