Topic 3: Nuts & Bolts of Linear Regression

ECON30130: Econometrics

Dr. Enda Hargaden

Spring 2025
Please remember that reading slides is like watching a
movie with the sound off. You will likely miss out on key
details if you rely on slides to “understand the script”.
The “Classical Model” and Gauss-Markov

Under “reasonable” assumptions, we have the Gauss-Markov Theorem.

OLS delivers estimates with lower variance than any other unbiased estimator. The
theoretical underpinning for OLS: there is no estimator that gives you an unbiased
estimate of the true β with lower variance.

This is sometimes, annoyingly, called BLUE Best Linear Unbiased Estimator. This
is catchy but stupid. We will discuss violations of these assumptions in the this
segment of the module.

These assumptions are that the errors are mean zero, homoskedastic, and
uncorrelated with each other.
Heteroskedasticity
An intuitive definition of homoskedasticity

Homoskedastic errors: var(ui ) = σ 2

e.g. σ 2 = 6.8

Heteroskedastic errors: var(ui ) = σi2

i.e. variance is not well captured by a single number.
Heteroskedasticity— what is it?
Example of homoskedasticity
400

200
Residuals

-200

-400
0 1000 2000 3000 4000
Some explanatory variable, e.g. x2
Heteroskedasticity— what is it?
Clear example of heteroskedasticy

400

200
Residuals

-200

-400
0 1000 2000 3000 4000
Some explanatory variable, e.g. x2
Heteroskedasticity— more examples
Consequences

Recall the formula for a standard error:

\
se( βˆ1 ) = qP
σ̂
n
i=1 (xi − x̄)
2

1. This is based on the assumption of homoskedasticity

2. Thus heteroskedasticity implies that ‘standard SEs’ will be wrong

3. No guarantee OLS is MVUE

4. Inference (t-tests, etc.) all wrong

Tests for heteroskedasticity

It is thus important to check if your results suffer from heteroskedasticity.

The classical test for this, which is based on the squared residuals, was developed
by two Australian econometricians, Breusch and Pagan, in 1979.
Breusch-Pagan (BP) test

The test has three steps:

1. Obtain residuals from a regression Y = βˆ0 + βˆ1 X1 + βˆ2 X2 + û

2. Square the residuals: Z = û2

3. Regress Z = αˆ0 + αˆ1 X1 + αˆ2 X2

This lets us test the null hypothesis that α0 = α1 = α2 = 0. This will give us an R2
for this regression, and the statistic the BP test uses is NR2 .

You might think this is an F-test. Not quite: NR2 ∼ χ2 with k degrees of freedom.
Enter Hal White (RIP)

Hal White’s PhD dissertation essentially solved the het-

eroskedasticity problem.

His famous 1980 Econometrica paper not only introduced a

huge generalization of B-P, but also derived the ‘appropriate’
SE’s in the case of heteroskedasticity.

White was tipped for the Nobel Prize in 2011, but didn’t win
that year. He sadly passed away in 2012.
Step 1: The Generalization

Much like in B-P, run a regression and square the residuals:

1. Square the residuals: Z = û2

2. But rather than just regressing Z on the X variables, themselves, regress Z on

the X variables, their squares, and their cross-products:

Z = αˆ0 + αˆ1 X1 + αˆ2 X2 + αˆ3 X21 + αˆ4 X22 + αˆ5 X1 X2

3. Test whether the α’s are jointly insignificant, again knowing that NR2 ∼ χ2 .
Step 2: The ‘Robust’ SEs

Hal White didn’t stop there.

He then derived appropriate standard errors, robust (in very large samples) to
essentially any form of heteroskedasticity. You don’t want to see the formula.
A quick note on consistency

h i
We are very familiar with the idea of unbiasedness: E β̂ = β

There is a related notion of consistency. This is where there is a bias, but that the
bias goes away in large samples. Any estimator that satisfies this criterion is called
‘consistent’. White’s SEs are consistent. They are sometimes called
Heteroskedastic Consistent, or HC errors.
Implementation of Heteroskedasticity-Consistent SEs

Naive SEs
1 library ( wooldridge )
2 data('wage1 ')
3 mymodel <- lm(lwage ~ educ + exper + south + female , data=wage1)

White ‘HC’ SEs

1 mymodel <- lm(lwage ~ educ + exper + south + female , data=wage1)
2 library ( sandwich )
3 library ( lmtest )
4 mymodel _ robust <- coeftest (mymodel , vcov = vcovHC )

This adjusts the variance covariance matrix (vcov) to be Heteroskedastic

Consistent (HC). It doesn’t change the point estimates.
Comparison across the two models
Naive SEs White SEs

educ 0.090∗∗∗ 0.090∗∗∗

(0.007) (0.008)

exper 0.009∗∗∗ 0.009∗∗∗

(0.001) (0.001)

south −0.069∗ −0.069∗

(0.039) (0.038)

female −0.347∗∗∗ −0.347∗∗∗

(0.038) (0.038)

Constant 0.524∗∗∗ 0.524∗∗∗

(0.108) (0.124)

Observations 526 526

R2 0.356 0.356
Adjusted R2 0.351 0.351

Note: ∗ p<0.1; ∗∗ p<0.05; ∗∗∗ p<0.01

Serial Correlation

(or Autocorrelation)
Autocorrelation

Autocorrelation/serial correlation is when the errors are correlated with each other.
It is a separate issue to having non-standard variance.

corr(ui , uj ) ̸= 0

or equally
corr(ut , ut−1 ) ̸= 0
A simple case of serial correlation

Suppose your errors persist through time. A simple case is when they are linearly
dependent on one prior period.

ut = ρut−1 + ϵt
where |ρ| < 1

(If |ρ| ≥ 1, the series is unstable and difficult to study.)

Positive or negative autocorrelation

If ρ < 0
• Then there is a negative relationship between ut and ut−1
• We call this ‘negative autocorrelation’

If ρ > 0
• Then there is a positive relationship between ut and ut−1
• We call this ‘positive autocorrelation’
What autocorrelation looks like

Positive autocorrelation is when ut−1 is higher than expected, ut tends to stay

high:

Can you come up with your own definition of negative autocorrelation?

Second correlation need not only be “first order”

Second Order:
ut = ρ1 ut−1 + ρ2 ut−2 + ϵt

Third Order:
ut = ρ1 ut−1 + ρ2 ut−2 + ρ3 ut−3 + ϵt

Causes?
Lots of things. Possibly nature.
General consequences of autocorrelation

Recall the assumption for the OLS SEs that the errors are Normally distributed.
One implication of this is independence. Autocorrelation is a violation of
independence. Thus, just like heteroskedasticity:

1. No change to OLS coefficients, but

2. No guarantee OLS is MVUE

3. Inference (t-tests, etc.) all wrong. In the case of positive autocorrelation (the
most common case), SEs will be much too small and t-stats too large.
Lagged dependent variables and autocorrelation

One specific example is serial correlation and lagged dependent (Y) variables.

If there is a lagged dependent variable, and autocorrelation follows certain forms

(e.g. first-order), then OLS point estimates (not just the SEs) will be biased.
Where autocorrelation can bias β̂

yt = β0 + β1 yt−1 + ut (1)
ut = ρut−1 + ϵt (2)

Notice that the error term ut can be substituted out from Equation (1):

yt = β0 + β1 yt−1 + ρut−1 + ϵt

But of course yt−1 is correlated with ut−1 (think about it, the outcome at time t − 1
must be correlated with shocks/the error at time t − 1), and so OLS will be biased.

This is because an explanatory variable (i.e. yt−1 ) is correlated with the error term
ut .
Takeaways from this example

1. Autocorrelation causes bias in the OLS SEs

2. In general, autocorrelation does not cause bias in the OLS β̂s

3. But it can if you are using lagged dependent variables

How to Deal with Autocorrelation

Unfortunately, there is no “Hal White” for autocorrelated errors. The closest-thing

is Newey-West (1987). These are like White standard errors (in that they’re
asymptotically consistent) but they require lag specification.

As they require lag specification, I will not go into implementing them here.
However, the sandwich package can help you.

(Fun fact: both Newey and West studied under Jerry Hausman, who also supervised
Hal White’s dissertation.1 )

1
Other notable students of his include Paul Krugman and Ben Bernanke.
Panel Data & Fixed effects
Panel Data

Data with repeated observations of the same unit (person, company, state, etc.) is
called panel data. An example is the monthly unemployment rate in every county.

Generally speaking, panel data allows much more flexible specifications than
cross-sectional data or time-series data alone.
What is the marginal effect here?
What is the marginal effect here?
Simpson’s Paradox

Marginal effects — the slopes — will be incorrectly estimated (even to the point
of having the wrong sign!) if you force multiple groups to have a common intercept.

This is a specific form of omitted variable bias. Reverse sign is an extreme version
of β̂ ̸= β.
A key advantage of panel data

Panel Data permits individual-specific intercepts.

Consider the standard cross-sectional model:

Yi = α + β1 X1i + β2 X2i + ui

With panel data, we have more observations so we generalize this:

Yi = αi + β1 X1i + β2 X2i + ui
Fixed Effects/Unobserved heterogeneity model

Yi = αi + β1 X1 + β2 X2 + ui
These individual- or unit-specific intercepts are called fixed effects. They capture
any and all time invariant features (e.g. genetics of people, constitutions for
countries, history of firms, etc.) of that unit. This is really powerful.

Furthermore, we don’t even have to directly observe these features (e.g. the
genetics of the individual). They’re automatically captured.
Time Fixed Effects

Suppose you have panel data on countries and taxes, 1960–2023. You’re
considered that all states suffered a big shock in 2009, and another in 2020, and
don’t want this affecting your relationship of interest.

Throw in a year fixed effect.

In fact, throw in fixed effects for all years. Extract all that business cycle variation
from your core relationship.
The importance of getting your intercepts right
Countries differ in their number of parties
Taxes and Number of Parties

55
Sweden
Denmark

Government Revenue (% GDP)

50
Norway

France Netherlands
Austria Finland Belgium
45
New Zealand
Germany Luxembourg
UK Canada
40

Ireland Iceland
Italy
35

Portugal
Switzerland
USA Spain
Greece
30

Australia
Japan

2 3 4 5 6 7
Parties in Parliament (average)

Average Taxes Fitted values

Political Science regression

Table: Effects on Tax Receipts

Without Country FE With Country FE

(1) (2) (3) (4)
Eff. parties parliament 2.256∗∗∗ 1.654∗∗ -0.506 -0.374
(0.747) (0.687) (0.589) (0.438)
Year FE Yes Yes Yes Yes
Macro controls No Yes No Yes
Country FE No No Yes Yes
N 981 850 981 850
Standard errors in parentheses
∗
p < 0.10, ∗∗ p < 0.05, ∗∗∗ p < 0.01
First differences
First-differences

A second panel data technique that is feasible under panel data is called
first-differences.
First-differences

Suppose you believe the true model is an unobserved heterogeneity model which,
in full notation, is:
Yit = αi + β1 X1,it + β2 X2,it + uit

Ireland held censuses in 2022 and 2016. So, to make this more concrete, let’s
consider two example observations:

WagesAnn,2022 = αAnn + β1 EducAnn,2022 + β2 ExperAnn,2022 + uAnn,2022

and

WagesAnn,2016 = αAnn + β1 EducAnn,2016 + β2 ExperAnn,2016 + uAnn,2016

Let’s manipulate those to observations into changes

Let’s omit the error terms for a second.

2022 version: WagesAnn,2022 = αAnn + β1 EducAnn,2022 + β2 ExperAnn,2022 (1)

2016 version: WagesAnn,2016 = αAnn + β1 EducAnn,2016 + β2 ExperAnn,2016 (2)

Subtract Equation 2 from Equation 1, to get change in Wages, which we’ll call
“∆Wages”
∆WagesAnn,2022 = WagesAnn,2022 − WagesAnn,2016

So we have

∆WagesAnn,2022 = ∆αAnn + β1 ∆EducAnn,2022 + β2 ∆ExperAnn,2022

Intercepts don’t change

∆WagesAnn,2022 = ∆αAnn + β1 ∆EducAnn,2022 + β2 ∆ExperAnn,2022

Recall that αAnn is Ann’s intercept. It doesn’t have a time subscript. It’s something
‘fundamental’ to her, it’s ‘time-invariant’. It is the same in 2016 and in 2022.

All this is to say ∆αAnn = 0 so it drops out of the first-difference equation.

∆WagesAnn,2022 = ∆αAnn +β1 ∆EducAnn,2022 + β2 ∆ExperAnn,2022

| {z }
=0
Moral of the Story

∆WagesAnn,2022 = β1 ∆EducAnn,2022 + β2 ∆ExperAnn,2022

What do we learn here?

By doing a “first difference” transformation of the data, and running OLS in

differences rather than in levels, we automatically remove the unobserved,
unit-specific heterogeneity.

→ We don’t even need to know Ann’s fundamentals! (DNA, etc.)

Is this any use?

“We just learned that fixed effects can account of unit-specific heterogeneity. Why
first difference at all?”

Great question! There are two main advantages of running first differences:
1. The β̂ in the FD world is theoretically identical to the β̂ estimated in FE world
(the βs are the same). So you should run both, and make sure your estimates
are similar in magnitude. If they’re not, you have a problem.

2. The β̂ in FD is consistent under slightly weaker conditions than the β̂ in FE.

This distinction is subtle, and hopefully is irrelevant for your personal usage.
See above.
A little more interpretation
Interpreting Dummy Variables

Recall the binary nature of dummy variables.

• Holding other variables constant, moving from zero to one is associated with
increasing Y by β̂, or

• Holding other variables constant, observations with X = 1 have outcomes that

are β̂ higher than observations with X = 0.
Y and log(X)

Y = β0 + β1 log(X1 ) + β2 X2
Suppose βˆ1 = 5.
Literally then a one-unit increase in the log of X1 is associated with increasing Y by
5. However, a more useful interpretation exploits a special feature of logs.

In particular, here the interpretation is a 1 percentage increase in X is

βˆ1
associated with a 100 unit increase Y.

Alternatively, a 1% increase in X is associated with a 0.05-unit increase in Y.

log(Y) and X

log(Y) = β0 + β1 X1 + β2 X2
Suppose βˆ1 = 0.62.
Literally then a one-unit increase in X1 is associated with the log of Y by 0.62.

However, a more useful interpretation is that increasing X by one unit is

associated with a 100 × βˆ1 percent change in Y.

Alternatively, increasing X by one-unit increases Y by 62%.

log(Y) and log(X)

log(Y) = β0 + β1 log(X1 ) + β2 X2

This is everyone’s favourite: it’s an elasticity!

For example, suppose βˆ1 = 2.5. The interpretation is that a 1% change in X is

associated with a 2.5% change in Y.
Interaction Effects
Interaction effects example

Suppose you run an experiment (“treatment”) teaching teenagers about the risks of
pregnancy. You might think the effects of this will differ between males and
females. You could plausibly run two regressions, one on males and one on
females. But in this case it might be better to estimate:

Y = α + β1 Treatment + β2 (Treament × Female) + u

where βˆ1 will show whether the treatment had a significant effect for men, and βˆ2
whether the effect for men was different from that for women.

Mention the need for βˆ1 + βˆ2 . Mention Jason’s paper.

Bobonis et al. (AER, 2016) on audits and corruption
 Variable selection
Which variables should I include in my model?
Opening Gambit

You’re searching for truth.

That is not achieved by focusing on statistical significance, maximizing R2 , or

anything else.

Truth, first and foremost.

Which variables should be included in our regression?

Suppose we want to understand rents in Dublin.

We should include the number of bedrooms in the house, the crime in the area,
and whether it comes with amenities like a bus route. Including these factors will
likely improve our predictions, and help us better understand rental prices.

But should we include a variable for e.g. length of the window-sills?

There is a case for including such a variable

Recall omitted variable bias.

Omitting an importanth variable
i will give leave us with a biased estimate of the true
β, or more formally E β̂ ̸= β.

We call this a “biased estimate”. Bias exists unless:

1. There is no omitted variable, or

2. Included and omitted variables are uncorrelated

Included variables

In a perfect world, theory informs you which variables you should include in a
regression. Should we include this variable or not?

Including variables that should be in there will improve the model, even if they’re
not directly relevant to our parameter of interest. They will still ‘control’ for some of
the residual variance.

Do not simply include variables based on naive statistical tests.

Choosing Independent Variables Example 1

Suppose you want to estimate GPA, and your friend has collected these data:

GPA College GPA

HGPA High school GPA
MSAT Math SAT
VSAT Verbal SAT
GREEK Dummy variable for Greek life
HRS Average number of hours studying per week
JOCK Dummy variable for Athletics
PRIV Dummy variable for private high school
lnEX Log number of college courses completed
GPA = f(?, ?, ?, ?) + u
Irrelevant variables

In general, you should be somewhat generous in the inclusion of variables, but do

not push it to extremes.

Regressions of rent in Dublin should probably exclude window sill length variables.

Including variables unnecessarily will generally increase the variance of our

estimates.
Choosing Independent Variables Example 2

You work for Prime Energy Drink, and estimate the price elasticity of demand for
the product by offering selected students the opportunity to purchase the drink at
different prices (50c, e1, e2).

You also happen to know these students’ parental income. Should you include this
variables in the analysis?
Why Can’t I Just Include Every Variable?
Why Can’t I Just Include Every Variable?

Well, firstly, k ≤ n by simple mathematics.

If you have two equations, the most unknowns you can solve for is two. You cannot
obtain unique solutions for three unknowns.

If you have n observations, you must have k or fewer variables. This is why panel
data allows individual individual fixed effects. It is also why cross-country growth
regressions are largely a dead-end: we have about 200 countries, but more than
200 “causes” of growth.
 Multicollinearity

or

Including variables unnecessarily will

generally increase the variance of our
estimates.
Perfect multicollinearity – indeterminant matrices

Try
 to calculate
 the determinant of the following matrix:
10 70
3 21

The two columns are perfectly correlated (they are multiples of each other), so
there is no way to disentangle the effect of one variable from the other.

Story: two guys pushing a boulder up a mountain. Dummy variable trap.

Imperfect multicollinearity – it’s a start

If two variables are imperfectly correlated, OLS can at least back out separate
estimates. These estimates carry some difficulties though.

Calculate
  the determinants
 of thefollowing two matrices.
8 5 8 5
3 2 =? 3.1 2 =?
Imperfect multicollinearity – it’s a start

If two variables are imperfectly correlated, OLS can at least back out separate
estimates. These estimates carry some difficulties though.

Calculate
  the determinants of the 
following two matrices.
8 5 8 5
3 2 =1 3.1 2 = 0.5

A very small change in the data ends up having a big effect on the estimates.
Consequences of imperfect multicollinearity

1. Instability of coefficients (though they remain unbiased)

2. Increase in variance
Illustration of instability of estimates

See R
Detection of multicollinearity

1. Correlations between variables

2. Variance Inflation Factors

Variance Inflation Factors

Suppose we have a model

Y = α + β1 X1 + β2 X2 + β3 X3

and suspect X3 is a candidate for multicollinearity. That means it’s well correlated
with at least one of the other variables, X1 or X2 .

Then a regression of X3 on X1 and X2 would have a high R2 .

Variance Inflation Factors

From that regression, we can calculate a Variance Inflation Factor (VIF) for X3 with
the following formula:
1
VIF =
1 − R2
Variance Inflation Factors

Indeed, we can calculate a vif for all variables in a regression.

Unlike some quantities like a t-statistic, there is no formal guide for what
constitutes a large or small VIF. A rule of thumb of VIF ≥ 5 indicates a serious
problem.

The car library includes a vif(mymodelname) command to calculate VIFs.

Some takeaways for empirical work

• Perfect multicollinearity (MC) is a mathematical problem

• Imperfect MC is a tricky issue with no obvious answer

• You should check if MC is an issue by probing the robustness of your results

• In this light, you should see MC as part of the specification issue covered
previously

• Good theoretical modeling helps mitigate MC before it emerges

Akaike Information Criterion (AIC)

One objective way to compare model fit, named for Hirotugu Akaike
(“A-kah-ee-kay”), is to use the AIC.

This is comparable to (but better than) comparing models based on R̄2 . R can
calculate the AIC for a regression you run (and the similar Bayesian Information
Criterion), by entering AIC(mymodelname) after a regression. A lower AIC means a
better fit.
Some Overall Conclusions
Perhaps more art than science

Bad Idea “Hmm, my R2 is a little low, I should run variations of the regressions
until we get nice t-stats, and the best R̄2 ”. This is unscientific and unethical. Don’t
do it.

Good Idea Sounds strange given the above point, but you should run a number of
variations on the regression. There is a big difference between trying to poke
holes in your results, and trying to find the one result that looks ‘best’.
Recall this slide about variable inclusion

Suppose you want to estimate GPA, and your friend has collected these data:

GPA College GPA

HGPA High school GPA
MSAT Math SAT
VSAT Verbal SAT
GREEK Dummy variable for Greek life
HRS Average number of hours studying per week
JOCK Dummy variable for Athletics
PRIV Dummy variable for private high school
lnEX Log number of college courses completed
GPA = f(?, ?, ?, ?) + u
Some tentative conclusions

1. Theory should drive your model selection

2. Regressions (almost always) suffer from omitted variables

3. Err on the side of caution

4. Statistical measures can guide model selection

5. Never hunt for statistical significance

6. Do not emphasize goodness of fit

One Last Thing: Nonlinearity in X
Polynomials

Although we have focused on linear models, we can also include polynomials (e.g.
X2 ) in our regression. This is still a linear model.

Suppose you believe in diminishing returns. That is, the first unit of X has a big
effect on Y, but each additional unit has a smaller effect. This is a common
assumption in economics. We can capture this with a quadratic term:

Y = β 0 + β 1 X + β 2 X2 + u

where β1 > 0 and β2 < 0.