Advanced Statistics

Quentin Lippmann

1 / 46
1 Last Week

2 The simple linear regression model

3 Expectation and Variance of OLS Estimators

4 Let’s recap

2 / 46
Last Week

• What are the different data types

3 / 46
Last Week

• What are the different data types

• Does a very high correlation between two variables imply a causal

relationship?

3 / 46
Last Week

• What are the different data types

• Does a very high correlation between two variables imply a causal

relationship?
• What are the two assumptions needed to interpret causally a
coefficient of a simple linear regression?

3 / 46
Last Week

• What are the different data types

• Does a very high correlation between two variables imply a causal

relationship?
• What are the two assumptions needed to interpret causally a
coefficient of a simple linear regression?
• What is the OLS estimator? What is the intuition?

3 / 46
Last Week

• What are the different data types

• Does a very high correlation between two variables imply a causal

relationship?
• What are the two assumptions needed to interpret causally a
coefficient of a simple linear regression?
• What is the OLS estimator? What is the intuition?

• How do you derive the OLS estimator?

3 / 46
Last Week

• What are the different data types

• Does a very high correlation between two variables imply a causal

relationship?
• What are the two assumptions needed to interpret causally a
coefficient of a simple linear regression?
• What is the OLS estimator? What is the intuition?

• How do you derive the OLS estimator?

• What is the coefficient of determination?

3 / 46
Last Week

• What are the different data types

• Does a very high correlation between two variables imply a causal

relationship?
• What are the two assumptions needed to interpret causally a
coefficient of a simple linear regression?
• What is the OLS estimator? What is the intuition?

• How do you derive the OLS estimator?

• What is the coefficient of determination?

3 / 46
1 Last Week

2 The simple linear regression model

3 Expectation and Variance of OLS Estimators

4 Let’s recap

4 / 46
Non-linear relationships

• Suppose we estimate the impact of education on hourly wages

• We obtain wage
 = 12 + 0.8educ
• How do we interpret 0.8?

5 / 46
Non-linear relationships

• Suppose we estimate the impact of education on hourly wages

• We obtain wage
 = 12 + 0.8educ
• How do we interpret 0.8? When educ increases by 1, wage increases
by 0.8
• Does it seem correct?

5 / 46
Non-linear relationships

• Suppose we estimate the impact of education on hourly wages

• We obtain wage
 = 12 + 0.8educ
• How do we interpret 0.8? When educ increases by 1, wage increases
by 0.8
• Does it seem correct? No

• Assume that each year of schooling increases wages by a fixed

percent
• How do we model that?

5 / 46
Non-linear relationships

• Suppose we estimate the impact of education on hourly wages

• We obtain wage
 = 12 + 0.8educ
• How do we interpret 0.8? When educ increases by 1, wage increases
by 0.8
• Does it seem correct? No

• Assume that each year of schooling increases wages by a fixed

percent
• How do we model that?

• We update the model: log(wage) = β0 + β1 educ + u

• We have: %∆wage = (100β1)∆educ

5 / 46
Non-linear relationships

• Suppose we estimate the impact of education on hourly wages

• We obtain wage
 = 12 + 0.8educ
• How do we interpret 0.8? When educ increases by 1, wage increases
by 0.8
• Does it seem correct? No

• Assume that each year of schooling increases wages by a fixed

percent
• How do we model that?

• We update the model: log(wage) = β0 + β1 educ + u

• We have: %∆wage = (100β1)∆educ
• Interpret β
c1 = 0.08

5 / 46
Non-linear relationships
Proof
ln(Y ) = β0 + β1 X

6 / 46
Non-linear relationships
Proof
ln(Y ) = β0 + β1 X

∂(ln(Y )) 1 ∂Y ∂Y
= β1 ⇔ · = β1 ⇔ = β1 ∂X
∂X Y ∂X Y

6 / 46
Non-linear relationships
Proof
ln(Y ) = β0 + β1 X

∂(ln(Y )) 1 ∂Y ∂Y
= β1 ⇔ · = β1 ⇔ = β1 ∂X
∂X Y ∂X Y

For a small change ∆X in X , the corresponding change in Y is

approximately:

6 / 46
Non-linear relationships
Proof
ln(Y ) = β0 + β1 X

∂(ln(Y )) 1 ∂Y ∂Y
= β1 ⇔ · = β1 ⇔ = β1 ∂X
∂X Y ∂X Y

For a small change ∆X in X , the corresponding change in Y is

approximately:
∆Y ∆Y
≈ β 1 ∆X ⇔ ≈ β1 ∆X
Y Y

6 / 46
Non-linear relationships
Proof
ln(Y ) = β0 + β1 X

∂(ln(Y )) 1 ∂Y ∂Y
= β1 ⇔ · = β1 ⇔ = β1 ∂X
∂X Y ∂X Y

For a small change ∆X in X , the corresponding change in Y is

approximately:
∆Y ∆Y
≈ β 1 ∆X ⇔ ≈ β1 ∆X
Y Y

The left side is the definition of percentage change in Y . Multiplying

both sides by 100%:

6 / 46
Non-linear relationships
Proof
ln(Y ) = β0 + β1 X

∂(ln(Y )) 1 ∂Y ∂Y
= β1 ⇔ · = β1 ⇔ = β1 ∂X
∂X Y ∂X Y

For a small change ∆X in X , the corresponding change in Y is

approximately:
∆Y ∆Y
≈ β 1 ∆X ⇔ ≈ β1 ∆X
Y Y

The left side is the definition of percentage change in Y . Multiplying

both sides by 100%:

%∆Y ≈ (100β1 )∆X

6 / 46
Non-linear relationships
Illustration

• Left: log(wage) = β0 + β1 educ + u

• Right: wage = β0 + β1 educ + u

7 / 46
Non-linear relationships

• Suppose we estimate log(salary ) = β0 + β1 log(sales) + ²

• Say salary of CEO as a function of sales

• How do we interpret β1 ?

8 / 46
Non-linear relationships

• Suppose we estimate log(salary ) = β0 + β1 log(sales) + ²

• Say salary of CEO as a function of sales

• How do we interpret β1 ? Elasticity

• Recall the definition of elasticity:

Percentage change in Y ∂Y /Y
Elasticity = =
Percentage change in X ∂X /X
• Derive log(salary ) with respect to log(sales) from the original
specification

∂log(salary ) ∂salary /salary

= = β1
∂log(sales) ∂sales/sales

8 / 46
To summarize

Model Dependent (Y) Independent (X) Interpretation of β1

Level-Level Y X ∆Y = β1 ∆X
Level-Log Y ln(X) ∆Y = (β1 /100)%∆X
Log-Level ln(Y) X %∆Y = (100β1 )∆X
Log-Log ln(Y) ln(X) %∆Y = β1 %∆X
Table 1: Interpretation of β1 in different regression models

9 / 46
Non-linear relationships: squared terms

• The effect of X on Y may not be constant

• Example: wages increase with experience, but at a decreasing rate
• How do you model that?

10 / 46
Non-linear relationships: squared terms

• The effect of X on Y may not be constant

• Example: wages increase with experience, but at a decreasing rate
• How do you model that?

• We can model this with a squared term

10 / 46
Non-linear relationships: squared terms

• The effect of X on Y may not be constant

• Example: wages increase with experience, but at a decreasing rate
• How do you model that?

• We can model this with a squared term

y = β0 + β1 X + β2 X 2 + u

10 / 46
Non-linear relationships: squared terms

• The effect of X on Y may not be constant

• Example: wages increase with experience, but at a decreasing rate
• How do you model that?

• We can model this with a squared term

y = β0 + β1 X + β2 X 2 + u

• Interpretation
∂y
= β1 + 2β2 X
∂X

• If β2 < 0: diminishing returns (concavity)

• If β2 > 0: accelerating effect (convexity)
• When does the shift occur?

10 / 46
Non-linear relationships: squared terms
An example

11 / 46
What does linear mean?

• A model is linear in the parameters if the dependent variable (Y) can

be expressed as a linear combination of the parameters β0 and β1
• Examples of linear models
• log(y ) = β0 + β1 x + u

12 / 46
What does linear mean?

• A model is linear in the parameters if the dependent variable (Y) can

be expressed as a linear combination of the parameters β0 and β1
• Examples of linear models
• log(y ) = β0 + β1 x + u
p
• log(y ) = β0 + β1 x + u

12 / 46
What does linear mean?

• A model is linear in the parameters if the dependent variable (Y) can

be expressed as a linear combination of the parameters β0 and β1
• Examples of linear models
• log(y ) = β0 + β1 x + u
p
• log(y ) = β0 + β1 x + u
• log(y ) = β0 + β1 x 2 + u

12 / 46
What does linear mean?

• A model is linear in the parameters if the dependent variable (Y) can

be expressed as a linear combination of the parameters β0 and β1
• Examples of linear models
• log(y ) = β0 + β1 x + u
p
• log(y ) = β0 + β1 x + u
• log(y ) = β0 + β1 x 2 + u

• Examples of non linear models

12 / 46
What does linear mean?

• A model is linear in the parameters if the dependent variable (Y) can

be expressed as a linear combination of the parameters β0 and β1
• Examples of linear models
• log(y ) = β0 + β1 x + u
p
• log(y ) = β0 + β1 x + u
• log(y ) = β0 + β1 x 2 + u

• Examples of non linear models

• y = β 0 x β1 + u

12 / 46
What does linear mean?

• A model is linear in the parameters if the dependent variable (Y) can

be expressed as a linear combination of the parameters β0 and β1
• Examples of linear models
• log(y ) = β0 + β1 x + u
p
• log(y ) = β0 + β1 x + u
• log(y ) = β0 + β1 x 2 + u

• Examples of non linear models

• y = β 0 x β1 + u
• y = β 0 e β1 x + u

12 / 46
What does linear mean?

• A model is linear in the parameters if the dependent variable (Y) can

be expressed as a linear combination of the parameters β0 and β1
• Examples of linear models
• log(y ) = β0 + β1 x + u
p
• log(y ) = β0 + β1 x + u
• log(y ) = β0 + β1 x 2 + u

• Examples of non linear models

• y = β 0 x β1 + u
• y = β 0 e β1 x + u
β x
• y = 0 +u
β1 + x

• Important: models can be linearized

• y = exp β0 +β1 x +u ⇔ log(y ) = β0 + β1 x + u
• y ≈ β0 x β1 + u ⇔ log(y ) = log(β0 ) + β1 log(x ) + u

12 / 46
The unit of measurement
An example
The unit of measurement
An example

• Suppose we have two variables

• y corresponds to the annual salary in thousand of euros of CEO
• If salary = 856.3, it means that the salary is 856,300 euros
• x corresponds to a measure of profitability of equity (roe)
• If roe = 10, it means that the return is 10%

• We wish to study: salary = β0 + β1 roe + u

• We obtain salary
ƒ = 963.191 + 18.501roe
• What is the average salary if roe = 0?

13 / 46
The unit of measurement
An example

• Suppose we have two variables

• y corresponds to the annual salary in thousand of euros of CEO
• If salary = 856.3, it means that the salary is 856,300 euros
• x corresponds to a measure of profitability of equity (roe)
• If roe = 10, it means that the return is 10%

• We wish to study: salary = β0 + β1 roe + u

• We obtain salary
ƒ = 963.191 + 18.501roe
• What is the average salary if roe = 0?
• How does the salary increases when roe increases by 1?

13 / 46
The unit of measurement
An example
The unit of measurement
An example

• Suppose we have two variables

• y corresponds to the annual salary in thousand of euros of CEO
• If salary = 856.3, it means that the salary is 856,300 euros
• x corresponds to a measure of profitability of equity (roe)
• If roe = 10, it means that the return is 10%

• We wish to study: salary = β0 + β1 roe + u

• We obtain salary
ƒ = 963.191 + 18.501roe
• What is the average salary if roe = 0?
• How does the salary increases when roe increases by 1?
• If a given company has a roe = 30, will the salary of the COE be
1,518,221 euros?

13 / 46
An example
The unit of measurement

• Suppose we decide to measure the salary in euros

• If salary = 856300, it means that the salary is 856,300 euros

• How will β
c1 and β
c0 change?

14 / 46
An example
The unit of measurement

• Suppose we decide to measure the salary in euros

• If salary = 856300, it means that the salary is 856,300 euros

• How will β
c1 and βc0 change?
• We have: salary = 963191 + 18501roe
ƒ
• β
c0 and β
c1 are multiplied by 1000

14 / 46
An example
The unit of measurement

• Suppose we decide to measure the roe in percent roedec = roe/100

• If roedec = 0.23, it means that the return is 23%

• How will β
c1 and β
c0 change?

15 / 46
An example
The unit of measurement

• Suppose we decide to measure the roe in percent roedec = roe/100

• If roedec = 0.23, it means that the return is 23%

• How will β
c1 and β c0 change?
• We have: salary
ƒ = 963.191 + 1850.1roe
• β
c1 is multiplied by 100

15 / 46
The unit of measurement

• If the explained variable is multiplied by c

16 / 46
The unit of measurement

• If the explained variable is multiplied by c

• The constant and slope coefficients will be multiplied by c

16 / 46
The unit of measurement

• If the explained variable is multiplied by c

• The constant and slope coefficients will be multiplied by c

• If the explanatory variable is divided or multiplied by c

16 / 46
The unit of measurement

• If the explained variable is multiplied by c

• The constant and slope coefficients will be multiplied by c

• If the explanatory variable is divided or multiplied by c

• The slope coefficient related to this variable will be multiplied or
divided by c

16 / 46
The unit of measurement

• If the explained variable is multiplied by c

• The constant and slope coefficients will be multiplied by c

• If the explanatory variable is divided or multiplied by c

• The slope coefficient related to this variable will be multiplied or
divided by c

• Question: how will the coefficient of determination change?

16 / 46
The unit of measurement

• If the explained variable is multiplied by c

• The constant and slope coefficients will be multiplied by c

• If the explanatory variable is divided or multiplied by c

• The slope coefficient related to this variable will be multiplied or
divided by c

• Question: how will the coefficient of determination change?

• It will remain the same

16 / 46
Time for a quizz

17 / 46
1 Last Week

2 The simple linear regression model

3 Expectation and Variance of OLS Estimators

4 Let’s recap

18 / 46
Expectation of OLS Estimators

• We wish to estimate a given relationship y = β0 + β1 x + u

• Using OLS and a sample of data, we obtain β

c0 and β
c1

• Question: are they good estimators?

19 / 46
Expectation of OLS Estimators

• We wish to estimate a given relationship y = β0 + β1 x + u

• Using OLS and a sample of data, we obtain β

c0 and β
c1

• Question: are they good estimators?

• Question: what is a bad estimator?

19 / 46
An example
• We are interested in the relationship between study time and test
scores

• Question: what trend do you notice?

20 / 46
An example
• We are interested in the relationship between study time and test
scores

• Question: what trend do you notice?

• Question: how can we describe the relationship mathematically?
20 / 46
An example
• Question: what do you think of these estimators?

• Answer

21 / 46
An example
• Question: what do you think of these estimators?

• Answer
• Red =

21 / 46
An example
• Question: what do you think of these estimators?

• Answer
• Red = overestimation
• Green =

21 / 46
An example
• Question: what do you think of these estimators?

• Answer
• Red = overestimation
• Green = underestimation
• Blue =

21 / 46
An example
• Question: what do you think of these estimators?

• Answer
• Red = overestimation
• Green = underestimation
• Blue = wrong sign

21 / 46
An example

• Question: how different is the OLS estimator?

• It is unbiased

22 / 46
Unbiasedness of an estimator

An estimator θb of a parameter θ is said to be unbiased if its expected

value is equal to the true value of the parameter being estimated
Formally:

E [θb] = θ
Implication:

23 / 46
Unbiasedness of an estimator

An estimator θb of a parameter θ is said to be unbiased if its expected

value is equal to the true value of the parameter being estimated
Formally:

E [θb] = θ
Implication: if we repeat our sampling process many times and calculate
θb each time, the average of all these estimates would converge to the
true value θ

23 / 46
Understanding unbiasedness
Let’s take an example
1 Two quick questions:

24 / 46
Understanding unbiasedness
Let’s take an example
1 Two quick questions:
X : How many minutes do you take to get to campus?

24 / 46
Understanding unbiasedness
Let’s take an example
1 Two quick questions:
X : How many minutes do you take to get to campus?
Y : On a scale of 1–10, how much do you like the campus?
Assume: Y = β0 + β1 X + ²

24 / 46
Understanding unbiasedness
Let’s take an example
1 Two quick questions:
X : How many minutes do you take to get to campus?
Y : On a scale of 1–10, how much do you like the campus?
Assume: Y = β0 + β1 X + ²
2 Using the full group: draw the OLS line ⇒ population line

24 / 46
Understanding unbiasedness
Let’s take an example
1 Two quick questions:
X : How many minutes do you take to get to campus?
Y : On a scale of 1–10, how much do you like the campus?
Assume: Y = β0 + β1 X + ²
2 Using the full group: draw the OLS line ⇒ population line
3 Take two random samples of 5 students and draw OLS lines ⇒ sample
line

24 / 46
Understanding unbiasedness
Let’s take an example
1 Two quick questions:
X : How many minutes do you take to get to campus?
Y : On a scale of 1–10, how much do you like the campus?
Assume: Y = β0 + β1 X + ²
2 Using the full group: draw the OLS line ⇒ population line
3 Take two random samples of 5 students and draw OLS lines ⇒ sample
line

• Questions
• How do the sample lines compare to the population line?

24 / 46
Understanding unbiasedness
Let’s take an example
1 Two quick questions:
X : How many minutes do you take to get to campus?
Y : On a scale of 1–10, how much do you like the campus?
Assume: Y = β0 + β1 X + ²
2 Using the full group: draw the OLS line ⇒ population line
3 Take two random samples of 5 students and draw OLS lines ⇒ sample
line

• Questions
• How do the sample lines compare to the population line?
• Why do the slopes differ?

24 / 46
Understanding unbiasedness
Let’s take an example
1 Two quick questions:
X : How many minutes do you take to get to campus?
Y : On a scale of 1–10, how much do you like the campus?
Assume: Y = β0 + β1 X + ²
2 Using the full group: draw the OLS line ⇒ population line
3 Take two random samples of 5 students and draw OLS lines ⇒ sample
line

• Questions
• How do the sample lines compare to the population line?
• Why do the slopes differ?
• If we averaged the slopes from all possible samples of size 5, where
would that average line be?

24 / 46
Understanding unbiasedness
Let’s take an example
1 Two quick questions:
X : How many minutes do you take to get to campus?
Y : On a scale of 1–10, how much do you like the campus?
Assume: Y = β0 + β1 X + ²
2 Using the full group: draw the OLS line ⇒ population line
3 Take two random samples of 5 students and draw OLS lines ⇒ sample
line

• Questions
• How do the sample lines compare to the population line?
• Why do the slopes differ?
• If we averaged the slopes from all possible samples of size 5, where
would that average line be?

• Why sample estimates differ from the population?

24 / 46
Understanding unbiasedness
Let’s take an example
Understanding unbiasedness
Let’s take an example
1 Two quick questions:
X : How many minutes do you take to get to campus?
Y : On a scale of 1–10, how much do you like the campus?
Assume: Y = β0 + β1 X + ²
2 Using the full group: draw the OLS line ⇒ population line
3 Take two random samples of 5 students and draw OLS lines ⇒ sample
line

• Questions
• How do the sample lines compare to the population line?
• Why do the slopes differ?
• If we averaged the slopes from all possible samples of size 5, where
would that average line be?

• Why sample estimates differ from the population?

• Because of the sampling process
• Across many samples: the average equals the truth = unbiasedness

24 / 46
Properties of Unbiased Estimators

1 The bias of an unbiased estimator is zero:

Bias(θb) = E [θb] − θ = 0

2 Unbiasedness ensures that our estimator does not systematically

overestimate or underestimate the true parameter

3 However: other properties of estimators are desirable

25 / 46
Unbiasedness of OLS estimator
yi = β0 + β1 xi + ui
The OLS estimator for the slope (β1 ) is given by:
P
(xi − x )(yi − y )
βb1 =
(xi − x )2
P

We need to show that E [βb1 ] = β1

26 / 46
Unbiasedness of OLS estimator
yi = β0 + β1 xi + ui
The OLS estimator for the slope (β1 ) is given by:
P
(xi − x )(yi − y )
βb1 =
(xi − x )2
P

We need to show that E [βb1 ] = β1

Proof:
1) First, let’s substitute yi = β0 + β1 xi + ui into our estimator:

(xi − x )((β0 + β1 xi + ui ) − y )
P
βb1 =
(xi − x )2
P

26 / 46
Unbiasedness of OLS estimator
yi = β0 + β1 xi + ui
The OLS estimator for the slope (β1 ) is given by:
P
(xi − x )(yi − y )
βb1 =
(xi − x )2
P

We need to show that E [βb1 ] = β1

Proof:
1) First, let’s substitute yi = β0 + β1 xi + ui into our estimator:

(xi − x )((β0 + β1 xi + ui ) − y )
P
βb1 =
(xi − x )2
P

2) Simplify:
(xi − x )(β0 + β1 xi + ui − y )
P
βb1 =
(xi − x )2
P

(xi − x )(β0 − y ) + (xi − x )(β1 xi ) + (xi − x )(ui )

P P P
=
(xi − x )2
P

26 / 46
 X
3) Note that (xi − x ) = 0, so the first term becomes zero:
β1 (xi − x )(xi ) + (xi − x )(ui )
P P
βb1 =
(xi − x )2
P

27 / 46
 X
3) Note that (xi − x ) = 0, so the first term becomes zero:
β1 (xi − x )(xi ) + (xi − x )(ui )
P P
βb1 =
(xi − x )2
P

4) Simplify further: P
(xi − x )(ui )
β1 = β1 + P
b
(xi − x )2

27 / 46
 X
3) Note that (xi − x ) = 0, so the first term becomes zero:
β1 (xi − x )(xi ) + (xi − x )(ui )
P P
βb1 =
(xi − x )2
P

4) Simplify further: P
(xi − x )(ui )
β1 = β1 + P
b
(xi − x )2
5) Now, let’s take the expected value of both sides:
P
(xi − x )(ui )
· ¸
E [ β1 ] = E β1 + P
b
(xi − x )2
·P
(xi − x )(ui )
¸
= β1 + E
(xi − x )2
P

27 / 46
 X
3) Note that (xi − x ) = 0, so the first term becomes zero:
β1 (xi − x )(xi ) + (xi − x )(ui )
P P
βb1 =
(xi − x )2
P

4) Simplify further: P
(xi − x )(ui )
β1 = β1 + P
b
(xi − x )2
5) Now, let’s take the expected value of both sides:
P
(xi − x )(ui )
· ¸
E [ β1 ] = E β1 + P
b
(xi − x )2
·P
(xi − x )(ui )
¸
= β1 + E
(xi − x )2
P

6) One of the key assumptions of OLS is that E [ui /x ] = 0 for all i. By

the linearity of expectation:
P
(xi − x )E [ui ]
E [βb1 ] = β1 +
(xi − x )2
P

27 / 46
 X
3) Note that (xi − x ) = 0, so the first term becomes zero:
β1 (xi − x )(xi ) + (xi − x )(ui )
P P
βb1 =
(xi − x )2
P

4) Simplify further: P
(xi − x )(ui )
β1 = β1 + P
b
(xi − x )2
5) Now, let’s take the expected value of both sides:
P
(xi − x )(ui )
· ¸
E [ β1 ] = E β1 + P
b
(xi − x )2
·P
(xi − x )(ui )
¸
= β1 + E
(xi − x )2
P

6) One of the key assumptions of OLS is that E [ui /x ] = 0 for all i. By

the linearity of expectation:
P
(xi − x )E [ui ]
E [βb1 ] = β1 +
(xi − x )2
P

7) Final step
P
(xi − x )(0)
E [βb1 ] = β1 + P = β1 + 0 = β1
(xi − x )2
27 / 46
Unbiasedness of OLS estimator

• Unbiasedness refers to the distribution of estimators

• No conclusion on a given estimator

• We will never be sure that we are close to the true value

28 / 46
Variance of OLS estimator

• Question: To what extent β

c1 can deviate from β1 on average?

29 / 46
Variance of OLS estimator

• Question: To what extent β

c1 can deviate from β1 on average? We
need to estimate the variance

29 / 46
Variance of OLS estimator

• To what extent β
c1 can deviate from β1 on average?

• We assume homoscedasticity: Var (u/x ) = σ2

• It is different from heteroscedasticity where Var (u/x ) depends on x

30 / 46
Variance of OLS estimators

31 / 46
Variance of OLS Estimators
The OLS estimator for β1 is:
P
(xi − x )(yi − y )
βb1 =
(xi − x )2
P

32 / 46
Variance of OLS Estimators
The OLS estimator for β1 is:
P
(xi − x )(yi − y )
βb1 =
(xi − x )2
P

1) Substitute yi = β0 + β1 xi + ui into the formula for βb1 :

(xi − x )((β0 + β1 xi + ui ) − (β0 + β1 x + u))

P
βb1 =
(xi − x )2
P

32 / 46
Variance of OLS Estimators
The OLS estimator for β1 is:
P
(xi − x )(yi − y )
βb1 =
(xi − x )2
P

1) Substitute yi = β0 + β1 xi + ui into the formula for βb1 :

(xi − x )((β0 + β1 xi + ui ) − (β0 + β1 x + u))

P
βb1 =
(xi − x )2
P

2) Simplify: P
(xi − x )(ui − u)
βb1 = β1 +
(xi − x )2
P

32 / 46
Variance of OLS Estimators
The OLS estimator for β1 is:
P
(xi − x )(yi − y )
βb1 =
(xi − x )2
P

1) Substitute yi = β0 + β1 xi + ui into the formula for βb1 :

(xi − x )((β0 + β1 xi + ui ) − (β0 + β1 x + u))

P
βb1 =
(xi − x )2
P

2) Simplify: P
(xi − x )(ui − u)
βb1 = β1 +
(xi − x )2
P

3) Therefore: P
(xi − x )(ui − u)
βb1 − β1 =
(xi − x )2
P

32 / 46
Variance of OLS Estimators
The OLS estimator for β1 is:
P
(xi − x )(yi − y )
βb1 =
(xi − x )2
P

1) Substitute yi = β0 + β1 xi + ui into the formula for βb1 :

(xi − x )((β0 + β1 xi + ui ) − (β0 + β1 x + u))

P
βb1 =
(xi − x )2
P

2) Simplify: P
(xi − x )(ui − u)
βb1 = β1 +
(xi − x )2
P

3) Therefore: P
(xi − x )(ui − u)
βb1 − β1 =
(xi − x )2
P

4) Now, let’s find the conditional variance:

µP ¯ ¶
(xi − x )(ui − u) ¯¯
Var (βb1 |X ) = Var X
(xi − x )2 ¯
P

32 / 46
Variance of OLS Estimators
5) Using the property Var (aX |Y ) = a2 Var (X |Y ) for constant a:
P
Var ( (xi − x )(ui − u)|X )
Var (βb1 |X ) =
[ (xi − x )2 ]2
P

33 / 46
Variance of OLS Estimators
5) Using the property Var (aX |Y ) = a2 Var (X |Y ) for constant a:
P
Var ( (xi − x )(ui − u)|X )
Var (βb1 |X ) =
[ (xi − x )2 ]2
P

6) Using conditionality on X:
(xi − x )2 Var (ui |X )
X X
Var ( (xi − x )(ui − u)|X ) =

33 / 46
Variance of OLS Estimators
5) Using the property Var (aX |Y ) = a2 Var (X |Y ) for constant a:
P
Var ( (xi − x )(ui − u)|X )
Var (βb1 |X ) =
[ (xi − x )2 ]2
P

6) Using conditionality on X:
(xi − x )2 Var (ui |X )
X X
Var ( (xi − x )(ui − u)|X ) =
7) Assuming homoscedasticity. Let σ2 = Var (ui |X ). Then:
σ2 (xi − x )2
P
Var (βb1 |X ) = P
[ (xi − x )2 ]2

33 / 46
Variance of OLS Estimators
5) Using the property Var (aX |Y ) = a2 Var (X |Y ) for constant a:
P
Var ( (xi − x )(ui − u)|X )
Var (βb1 |X ) =
[ (xi − x )2 ]2
P

6) Using conditionality on X:
(xi − x )2 Var (ui |X )
X X
Var ( (xi − x )(ui − u)|X ) =
7) Assuming homoscedasticity. Let σ2 = Var (ui |X ). Then:
σ2 (xi − x )2
P
Var (βb1 |X ) = P
[ (xi − x )2 ]2
8) Simplify:
σ2
Var (βb1 |X ) = P
(xi − x )2
Therefore, the conditional variance of βb1 given X is:

σ2
Var (βb1 |X ) = P
(xi − x )2

33 / 46
Variance of OLS estimators

• We have:
σ2
Var (βb1 |X ) = P
(xi − x )2

• We need to estimate σ2

34 / 46
Variance of OLS estimators

• We have:
σ2
Var (βb1 |X ) = P
(xi − x )2

• We need to estimate σ2

• Question: how do we do that?

34 / 46
Variance of OLS estimators

• We have:
σ2
Var (βb1 |X ) = P
(xi − x )2

• We need to estimate σ2

• Question: how do we do that?

• Using residuals ubi

34 / 46
Variance of OLS estimators

• We have yi = β0 + β1 xi + ui

35 / 46
Variance of OLS estimators

• We have yi = β0 + β1 xi + ui

• We also have: yi = β
c0 + β
c1 xi + ubi

35 / 46
Variance of OLS estimators

• We have yi = β0 + β1 xi + ui

• We also have: yi = β
c0 + β
c1 xi + ubi

• We obtain: ubi = yi − β
c0 − β
c1 xi = (β0 + β1 xi + ui ) − β
c0 − β
c1 xi

35 / 46
Variance of OLS estimators

• We have yi = β0 + β1 xi + ui

• We also have: yi = β
c0 + β
c1 xi + ubi

• We obtain: ubi = yi − β
c0 − β
c1 xi = (β0 + β1 xi + ui ) − β
c0 − β
c1 xi
• Which is equivalent to ubi = ui − (βc0 − β0 ) − (β
c1 − β1 )xi

• Notice how residuals ubi are related to errors ui

35 / 46
Variance of OLS estimators

• We know that σ2 = E (ui2 )

• Problem: we don’t know ui

36 / 46
Variance of OLS estimators

• We know that σ2 = E (ui2 )

• Problem: we don’t know ui

• But we have an estimator of ui with ubi

36 / 46
Variance of OLS estimators

• We know that σ2 = E (ui2 )

• Problem: we don’t know ui

• But we have an estimator of ui with ubi

1 n
• We estimate σ2 with σ ui2
X
c2 = c
n − 2 i =1

36 / 46
Variance of OLS estimators

• We know that σ2 = E (ui2 )

• Problem: we don’t know ui

• But we have an estimator of ui with ubi

1 n
• We estimate σ2 with σ ui2
X
c2 = c
n − 2 i =1
• Why n-2?
n
X n
X
• We need ubi = 0 and xi ubi = 0
i =1 i =1
• We pick at least two residuals to satisfy these conditions

36 / 46
Variance of OLS estimators

2
c1 |X ) = P σ
• We have Var (β
(xi − x )2

d (βb1 |X ) = P σ
c2
• We estimate it with: Var
(xi − x )2
v
u
u σc2 σ
• We define the standard error se(β1 ) = t P
b
c
2
=P 1
(xi − x ) (xi − x ) 2

37 / 46
1 Last Week

2 The simple linear regression model

3 Expectation and Variance of OLS Estimators

4 Let’s recap

38 / 46
Let’s recap

1 We are interested in the effect of the time spent studying on grades

39 / 46
Let’s recap

1 We are interested in the effect of the time spent studying on grades

2 We know that there exists a linear relationship in the population:

yi = β0 + β1 xi + ui

39 / 46
Let’s recap

1 We are interested in the effect of the time spent studying on grades

2 We know that there exists a linear relationship in the population:

yi = β0 + β1 xi + ui

3 We have data only on a sample of students

39 / 46
Let’s recap

1 We are interested in the effect of the time spent studying on grades

2 We know that there exists a linear relationship in the population:

yi = β0 + β1 xi + ui

3 We have data only on a sample of students

4 We estimate the relationship between the two variables using OLS

39 / 46
Let’s recap

1 We are interested in the effect of the time spent studying on grades

2 We know that there exists a linear relationship in the population:

yi = β0 + β1 xi + ui

3 We have data only on a sample of students

4 We estimate the relationship between the two variables using OLS

5 Interpretation
• β1 is causal if E (ui /xi ) = 0
• β1 is a correlation if E (ui /xi ) 6= 0

39 / 46
Let’s recap

1 We are interested in the effect of the time spent studying on grades

2 We know that there exists a linear relationship in the population:

yi = β0 + β1 xi + ui

3 We have data only on a sample of students

4 We estimate the relationship between the two variables using OLS

5 Interpretation
• β1 is causal if E (ui /xi ) = 0
• β1 is a correlation if E (ui /xi ) 6= 0

6 We estimate how far β

c1 can be β1 using Var
d (β1 )

39 / 46
Let’s recap
Raw data

40 / 46
Let’s recap
Many estimators are possible (and bad)

41 / 46
Let’s recap
OLS estimator

42 / 46
Let’s recap
OLS Estimator is unbiased

43 / 46
Let’s recap
OLS Estimators can be imprecise depending on the
data

44 / 46
Formulas to know

• OLS Estimators
n
P
(xi − x )(yi − y )
• β
c1 = i =1
n
(xi − x )2
P
i =1

45 / 46
Formulas to know

• OLS Estimators
n
P
(xi − x )(yi − y )
• β
c1 = i =1
n
(xi − x )2
P
i =1
• β
c0 = y − β
c1 x

45 / 46
Formulas to know

• OLS Estimators
n
P
(xi − x )(yi − y )
• β
c1 = i =1
n
(xi − x )2
P
i =1
• β
c0 = y − β
c1 x

• Variances
1 X n
c2 =
• σ uc2
n − 2 i =1 i

45 / 46
Formulas to know

• OLS Estimators
n
P
(xi − x )(yi − y )
• β
c1 = i =1
n
(xi − x )2
P
i =1
• β
c0 = y − β
c1 x

• Variances
1 X n
c2 =
• σ uc2
n − 2 i =1 i

d (βb1 |X ) = P σ
c2
• Var
(xi − x )2

45 / 46
Formulas to know

• OLS Estimators
n
P
(xi − x )(yi − y )
• β
c1 = i =1
n
(xi − x )2
P
i =1
• β
c0 = y − β
c1 x

• Variances
1 X n
c2 =
• σ uc2
n − 2 i =1 i

d (βb1 |X ) = P σ
c2
• Var
(xi − x )2
n
σ 1
2 x2
c P
n i
d (βb0 |X ) = P
• Var i =1
(xi − x )2

45 / 46
Time for a quizz

46 / 46