Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Linear Regression with One Regressor: Inference

Osman DOĞAN

1 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Linear Regression with One Regressor: Inference

Linear Regression with One Regressor: Inference

1 Simple hypothesis testing (or significance testing)
2 Confidence intervals for a regression coefficient
3 Regression when the regressor is binary
4 Heteroskedasticity vs Homoskedasticity
5 The Gauss-Markov theorem

Readings:
1 Stock and Watson (2020, Chapter 5),
2 Doğan and Taşpınar (2025, Chapter 15).

2 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Simple hypothesis testing

A hypothesis is simply a claim and a simple hypothesis in the regression

analysis means a claim involving a single coefficient, often about the slope of
the population regression function.

The empirical analysis may require the researcher to test a hypothesis on β1 .

For example, in our class-size test-scores example, a taxpayer’s claim could be

that cutting class size will not help boost test scores.

In other words, under this claim, the slope of the population regression function
is zero, i.e., β1 = 0.

Can you reject the taxpayer’s hypothesis that β1 = 0, or should you accept it,
at least tentatively pending further new evidence?

3 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Simple hypothesis testing

Our objective is to test a hypothesis, like β1 = 0, using our sample to reach a

tentative conclusion whether the hypothesis is correct or incorrect.

To this end, we will start by stating the null and the alternative hypotheses.

The null hypothesis will state the claim to be tested and we use H0 to denote
it. In our example, H0 : β1 = 0.

The alternative hypothesis will state the scenario when the null hypothesis does
not hold and we use H1 to denote it:
1 If H1 : β1 ̸= 0 it is called a two-sided alternative.
2 If H1 : β1 < 0 or H1 : β1 > 0, it is called a one-sided alternative.

4 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Simple hypothesis testing

In order to decide between H0 and H1 , we need a test statistics that will make
use of the sample data and help us to reach a tentative conclusion.

We will use the t-statistic, which is defined as

estimator-hypothesized value
t= (1)
standard error of the estimator
Thus, the t-statistic for testing H0 : β1 = c, where c is the hypothesized value,
takes the following form:

βb1 − c
t= , (2)
SE(βb1 )

where SE(βb1 ) is the square root of the variance of βb1 .

To compute t-statistic, we need to figure out SE(βb1 ).
Also, since t-statistic will inherit the uncertainty due to sampling, we will have
to figure out its sampling distribution to reach a tentative conclusion.

5 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Simple hypothesis testing

Recall from Chapter 4 that under the least squares assumptions, in large
samples, we have

A
  1 var ((Xi − µX )ui )
βb1 ∼ N β1 , σβ2b1 , where σβ2b1 = . (3)
n (var(Xi ))2

The idea is to replace the unknown terms in σβ2b with their sample
1
counterparts. Thus, an estimator of σβ2b is
1

1
Pn 2 2
1 n−2 i=1 (Xi − X̄) ubi
bβ2b1
σ = n
2 (4)
n 1
P
i=1 (Xi − X̄)
2
n
q
Then, SE(βb1 ) in (2) can be computed through SE(βb1 ) = σbβ2b .
1

6 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Simple hypothesis testing

Also, under the least squares assumptions, in large samples, we have

A
  βb1 − β1 A
βb1 ∼ N β1 , σβ2b1 ⇒ ≡ Z ∼ N (0, 1) (5)
σβb1

Also, (5) suggests that if H0 is true then, the sampling distribution of the
A
t-statistic can be approximated by N (0, 1) in large samples, i.e., t ∼ N (0, 1).

Recall the types of errors you can make in a testing problem (Chapter 3)
□ we reject a TRUE H0 , i.e., Type-I error,

□ we fail to reject a FALSE H0 , i.e., Type-II error.

We choose a (significance) level for committing the Type-I error, and then
minimize the likelihood of committing Type-II error.

The conventional significance levels are 1%, 5% and 10%.

We can now use the level and the sampling distribution of the test to define
the rejection regions.
7 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Simple hypothesis testing

Figure 1 illustrates the rejection regions according to the type of H1 .

1 In each case, the shaded blue area is the level of test.
2 Depending on the type of H1 , zα , −zα , −zα/2 and zα/2 are critical values.
Finally, we will reject H0 if the t-statistic value falls in the rejection regions
(the shaded blue areas).

Example 1
Consider H1 : β1 ̸= c in Figure 1c, and assume that α = 5%. Then, the critical
value on the left tail is the 2.5th percentile of N (0, 1), which is
−zα/2 = −1.96, and the critical value on the right tail is the 97.5th percentile
of N (0, 1), which is zα/2 = 1.96. In R, we can use the following:
qnorm (0.025 , mean = 0 , sd =1 , lower . tail = TRUE ) = -1.96

β̂1 −c
Assume that we obtained t = SE( β̂1 )
= 2. Then, we will reject H0 because the
test statistic value is in the rejection region. i.e., t = 2 > zα/2 = 1.96.

8 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

(a) H0 : β1 = c versus H1 : β1 > c (b) H0 : β1 = c versus H1 : β1 < c

(c) H0 : β1 = c versus H1 : β1 ̸= c

Figure 1: Rejection Regions

9 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Hypothesis Testing

Alternatively, we can also calculate a p-value to decide between H0 and H1 :


PH0 (t > |tcalc |) for H1 : β1 > c,

p-value = PH0 (t < −|tcalc |) for H1 : β1 < c,

PH0 (|t| > |tcalc |) for H1 : β1 ̸= c.


where tcalc is the value of the test statistic obtained from (2).

PH0 means this probability is calculated from the distribution of the test
statistic under the null hypothesis.

Since the asymptotic distribution of t-statistic is N (0, 1), we have


PH0 (t > |tcalc |) = 1 − Φ(|tcalc |) for H1 : β1 > c,

p-value = PH0 (t < −|tcalc |) = Φ(−|tcalc |) for H1 : β1 < c,

PH0 (|t| > |tcalc |) = 2 × Φ(−|tcalc |) for H1 : β1 ̸= c.


10 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Hypothesis Testing

What does the p-value tell us?

It gives the likelihood of obtaining a test statistic value that is more extreme
than the actual one when the null is correct.

Hence, the smaller p-value, the less likely that the null is correct. If the level is
chosen as 5%, then reject H0 if p-value is less than 5%.

Example 2
Consider H1 : β1 ̸= c in Figure 1c, and assume that α = 5%. Assume that we
obtained t = 2 from (2). Then,

p-value = PH0 (|t| > 2) = PH0 (t > 2) + PH0 (t < −2) = 2 × Φ(−2).

In R, we can compute the p-value in the following way:

2 * pnorm ( -2 , mean = 0 , sd =1 , lower . tail = TRUE ) = 0.045

Since p-value = 0.045 < α = 0.05, we reject H0 .

11 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Simple hypothesis testing

Consider the test score example:

TestScorei = β0 + β1 STRi + ui . (6)

In R, the lm function computes the t-statistic for each regressor and provides
the corresponding p-values for a two sided null hypothesis. The estimation
results are given in Listing 1.
Consider H0 : β1 = 0 versus H1 : β1 ̸= 0 and assume α = 5%. Then, we can
compute t-statistic as

β̂1 − 0 −2.28
t= = ≈ −4.75 (7)
SE(β̂1 ) 0.48

The critical value is zα/2 = −1.96. Since t = −4.75 < zα/2 = −1.96. We
reject H0 and conclude that β1 is statistically significant at 5% level.

Alternatively, we can compute p-value = 2 × Φ(−|tcalc |) = 2 × Φ(−4.75).

2 * pnorm ( -4.75 , mean = 0 , sd =1 , lower . tail = TRUE ) =2.034166 e -06

12 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Simple hypothesis testing

Listing 1: Estimation results

library ( stargazer )
library ( readxl )

CAschool = read _ excel ( " caschool . xlsx " , col _ names = TRUE , skip = 0)
results = lm ( testscr ∼ str , data = CAschool )
summary ( results )

Call :
lm ( formula = testscr ∼ str , data = mydata )

Residuals :
Min 1Q Median 3Q Max
-47.727 -14.251 0.483 12.822 48.540

Coefficients :
Estimate Std . Error t value Pr ( >| t |)
( Intercept ) 698.9330 9.4675 73.825 < 2e -16 * * *
str -2.2798 0.4798 -4.751 2.78 e -06 * * *
---
Signif . codes : 0 ’* * * ’ 0.001 ’* * ’ 0.01 ’* ’ 0.05 ’. ’ 0.1 ’ ’ 1

Residual standard error : 18.58 on 418 degrees of freedom

Multiple R - squared : 0.05124 , Adjusted R - squared : 0.04897
F - statistic : 22.58 on 1 and 418 DF , p - value : 2.783 e -06

13 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Confidence Interval for β1

Recall that a 95% confidence is, equivalently:

1 The set of points that cannot be rejected at the 5% significance level,
2 A set-valued function of the data (an interval that is a function of the data) that
contains the true parameter value 95% of the time in repeated samples.
Suppose the level of the (two-sided) test is 5%. We know that the critical
value (in absolute value) is 1.96.

Then,
 the 95% CI for β1 refers
 to the set of values for β1 such that
b1 −β1
β
P {β1 : SE(βb ) ≤ 1.96} = 0.95. Hence,
1

βb1 − β1 βb1 − β1
{β1 : ≤ 1.96} = {β1 : −1.96 ≤ ≤ 1.96}
SE(βb1 ) SE(βb1 )
= {β1 : βb1 − 1.96 SE(βb1 ) ≤ β1 ≤ βb1 + 1.96 SE(βb1 )}
h i
Thus, the 95% CI for β1 is βb1 − 1.96 × SE(βb1 ), βb1 + 1.96 × SE(βb1 ) .

14 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Confidence Interval for β1

Similarly, you can construct 90% and 99% CIs. You just need to change the
critical value, 1.64 and 2.57, respectively.

Notice that the CI is an estimator and is a random variable.

In other words, from one sample to the next, it provides different values for the
lower bound and upper bound of the interval.

By construction, this interval estimator retains the true (unknown) value of β1

in 95% of the intervals constructed under repeated sampling.

It does not mean that for the sample at hand the likelihood of the true
(unknown) value of β1 being in the interval constructed is 95%.

If the hypothesized value under H0 is in the CI, we fail to reject the H0 .

Otherwise, we reject H0 .

15 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Confidence Interval for β1

Example 3
Consider the test score example in (6). The estimation results are provided in
Listing 1. Then, the 95% CI for β1 is β̂1 ± 1.96 × SE(β̂1 ), which gives

[−3.22, −1.34] .

The 95% CI for β1 can be used to construct a 95% CI for the predicted effect
of a general change in X.

Recall that the expected change in Y associated with this change in X is β1 ∆x.

The 95% CI for β1 ∆x is

h    i
βb1 − 1.96 SE(βb1 ) ∆x, βb1 + 1.96 SE(βb1 ) ∆x .

16 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Regression when X is Binary

Sometimes a regressor can be binary–that is, it takes on only two values, 0 and
1. For example, X can be
1 a worker’s sex: X = 1 if female, X = 0 otherwise,
2 whether a school district is urban or rural: X = 1 if urban, X = 0 otherwise),
3 whether the district’s class size is small or large: X = 1 if small, X = 0
otherwise.
A binary variable is also called an dummy variable or sometimes a indicator
variable.

The mechanics of regression with a binary regressor are the same as if it is

continuous.

17 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Regression when X is Binary

Suppose you have a variable Di that equals either 0 or 1, depending on
whether the student-teacher ratio is less than 20:

1, if the student-teacher ratio in i’th district < 20
Di =
0, if the student-teacher ratio in i’th district ≥ 20.
Let the population regression model with Di as the regressor is
Yi = β0 + β1 Di + ui , i = 1, 2, . . . , n. (8)

Because Di is not continuous, it is not useful to think of β1 as a slope; indeed,

because Di can take on only two values.

The best way to interpret β1 in a regression with a binary regressor is to

consider, one at a time, the two possible cases, Di = 0 and Di = 1.
Under the least squares assumptions,
1 if the student-teacher ratio is high, then Di = 0, and we have
E(Yi |Di = 0) = β0 ,
2 if the student-teacher ratio is low, then Di = 1, and we have
E(Yi |Di = 1) = β0 + β1 .
18 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Regression when X is Binary

Hence,

E(Yi |Di = 1) − E(Yi |Di = 0) = β0 + β1 − β0 = β1

Thus, β1 is the difference between two population means.

In our example, it is the difference between the mean test score in districts with
low student-teacher ratios and the mean test score in districts with high
student-teacher ratios.

Listing 2 provides the estimation result for (8). The results show that
β̂1 = 7.37.

Thus, the students in districts with the STR less than 20 on average have 7.37
more points than the students in districts with the STR larger or equal to 20.

19 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Regression when X is Binary

Listing 2: Dummy variable regression results: R

CAschool $ D = ( CAschool $ str < 20) # create the dummy variable

results = lm ( testscr ∼ D , data = CAschool )
summary ( results )

Call :
lm ( formula = testscr ∼ D , data = mydata )

Residuals :
Min 1Q Median 3Q Max
-50.601 -14.047 -0.451 12.841 49.399

Coefficients :
Estimate Std . Error t value Pr ( >| t |)
( Intercept ) 649.979 1.388 468.380 < 2e -16 * * *
DTRUE 7.372 1.843 3.999 7.52 e -05 * * *
---
Signif . codes : 0 ’* * * ’ 0.001 ’* * ’ 0.01 ’* ’ 0.05 ’. ’ 0.1 ’ ’ 1

Residual standard error : 18.72 on 418 degrees of freedom

Multiple R - squared : 0.03685 , Adjusted R - squared : 0.03455
F - statistic : 15.99 on 1 and 418 DF , p - value : 7.515 e -05

20 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Regression when X is Binary

Listing 3: Dummy variable regression results: Python

import numpy as np
import pandas as pd
import statsmodels . api as sm
import statsmodels . formula . api as smf

# Import data
CAschool = pd . read _ excel ( " data / caschool . xlsx " )
# Create the dummy variable D
CAschool [ " D " ] = CAschool [ " str " ] < 20

# Specify the model

model2 = smf . ols ( formula = ’ testscr∼D ’ , data = CAschool )
# Use the fit method to obtain parameter estimates
result2 = model2 . fit ()
# Print the estimation results
print ( result2 . summary () )

21 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Heteroskedasticity vs. Homoskedasticity

Definition 1
The error term ui is homoskedastic if the variance of the conditional
distribution of ui given Xi is constant for i = 1, 2, . . . , n, and in particular does
not depend on Xi . Otherwise, the error term ui is heteroskedastic.

Recall the least squares assumptions did not say anything about the the
conditional distribution of ui given Xi . In other words, we have been assuming
heteroskedasticity implicitly.

And we derived the variance of the sampling distribution of the OLS estimator
under heteroskedasticity.
1 var ((Xi − µX )ui )
σβ2b1 =
n (var(Xi ))2
Under homoskedasticity, this formula simplifies to
1 var ((Xi − µX )ui ) 1 var ((Xi − µX )) var (ui ) 1 var (ui )
σβ2b1 = = = .
n (var(Xi ))2 n (var(Xi ))2 n var(Xi )

22 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Heteroskedasticity vs. Homoskedasticity

23 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Heteroskedasticity vs. Homoskedasticity

Then, under homoskedasticity, SE(βb1 ) is given by

s Pn s Pn
1 1
q
2 1 n−2
b2i
i=1 u n−2
b2i
i=1 u
σ
bβb1 = σ bβb = = n .
n n1 n
P P
i=1 (Xi − X̄)
2
i=1 (Xi − X̄)
1 2

This formula is called the homoskedasticity-only standard errors. The lm

function by default calculates standard errors according to this formula.

Heteroskedasticity-robust standard errors (or Eicker-Huber-White standard

errors) can be obtained using the vcovHC function from the sandwich package.

# install . packages (" sandwich ")

library ( sandwich )
r1 <- lm ( testscr ∼ str , data = mydata )
vcov <- vcovHC ( r1 , type = " HC1 " )
robust _ se <- sqrt ( diag ( vcov ) )

stargazer ( r1 , r1 ,
se = list ( NULL , robust _ se ) ,
type = " text " ,
column . labels = c ( " Homoskedastic " ,
" Heteroskedastic " ) ,
align = TRUE )

24 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Heteroskedasticity vs. Homoskedasticity

According the estimation results in Listing 4, the heteroskedastic standard

errors are relatively larger.

Listing 4: Homeskedastic and heteroskedastic models

============================================================
Dependent variable :
-----------------------------
testscr
Homoskedastic Heteroskedastic
(1) (2)
------------------------------------------------------------
str -2.280 * * * -2.280 * * *
(0.480) (0.519)

Constant 698.933 * * * 698.933 * * *

(9.467) (10.364)

------------------------------------------------------------
Observations 420 420
R2 0.051 0.051
Adjusted R2 0.049 0.049
Residual Std . Error ( df = 418) 18.581 18.581
F Statistic ( df = 1; 418) 22.575 * * * 22.575 * * *
============================================================
Note : *p <0.1; * *p <0.05; * * *p <0.01)

25 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Heteroskedasticity vs. Homoskedasticity

Listing 5: Homeskedastic and heteroskedastic models: Python

import numpy as np
import pandas as pd
import statsmodels . api as sm
import statsmodels . formula . api as smf
from statsmodels . iolib . summary2 import summary _ col

# Specify the model

model = smf . ols ( formula = ’ testscr∼str ’ , data = CAschool )
# Use the fit method to obtain parameter estimates
result = model . fit ( cov _ type = " HC1 " )
# Print the estimation results
print ( result . summary () )

26 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Heteroskedasticity vs. Homoskedasticity

If the errors are either homoskedastic or heteroskedastic and you use

heteroskedastic-robust standard errors, you are OK.

If the errors are heteroskedastic and you use the homoskedasticity-only formula
for standard errors, your standard errors will be wrong (the
homoskedasticity-only estimator of the variance of β̂1 is inconsistent if there is
heteroskedasticity).

The two formulas coincide (when n is large) in the special case of

homoskedasticity.

Therefore, Stock and Watson (2020) suggest that we should always use
heteroskedasticity-robust standard errors.

27 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

The Gauss-Markov theorem

Recall the three least squares assumptions. Now, add the homoskedasticity
assumption on the error term.

Then, it can be shown analytically that the OLS estimator is the most precise
estimator (conditional on X’s) within the class of linear unbiased estimators.

Often stated as the OLS estimator is Best Linear (conditionally) Unbiased

Estimator (BLUE).

This results is known as the Gauss-Markov theorem. See Appendix 5.2 for a
proof.

The Gauss-Markov theorem has two important limitations.

□ Homoskedasticity assumption is not likely to hold in practice. Then, the OLS
estimator is no longer BLUE.
□ There are estimators that are not linear and conditionally unbiased, and under
some conditions, these other estimators are more efficient than OLS.

28 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Normality of the error term

So far we have not specified the distribution of the error term, we just made
assumptions on its moments.

Addition to the four assumptions earlier, we assume also that the

ui ∼ N (0, σu2 ).

Then, the OLS estimator βb1 is exactly distributed as normal with mean β1 and
σβ2b under homoskedasticity.
1

Also, the t-statistic now exactly distributed as Student t with (n − 2) degrees

of freedom.

For large n, the Student t distribution with (n − 2) degrees of freedom is

almost the same as the standard normal distribution.

For inference, this creates a difference if the sample size very small, say n < 30:
the Student t critical values can be a fair bit larger than the standard normal
critical values.

29 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Practical implications

If n < 50 and you really believe that, for your application, ui is homoskedastic
and normally distributed, then use the tn−2 instead of the N (0, 1) critical
values for hypothesis tests and confidence intervals.

In most econometric applications, there is no reason to believe that ui is

homoskedastic and normal - usually, there are good reasons to believe that
neither assumption holds.

Fortunately, in modern applications, n > 50, so we can rely on the large-n

results presented earlier, based on the CLT, to perform hypothesis tests and
construct confidence intervals using the large-n normal approximation.

30 / 31
Outline Significance testing Confidence intervals Regression with binary X Heteroskedasticity Efficiency of the OLS References

Bibliography I

Doğan, Osman and Süleyman Taşpınar (2025). Introduction to Econometrics

with Python. url: [Link]
Stock, James H. and Mark W. Watson (2020). Introduction to Econometrics.
Fourth. Pearson.

31 / 31