Advanced Statistics
Quentin Lippmann
1 / 42
1 Introduction: what is econometrics
2 The different steps in empirical economic analysis
3 Linear Regression
4 Ordinary Least Squares Estimator
2 / 42
What is econometrics?
• Question: What is the effect of one additional year of schooling on
earnings?
3 / 42
What is econometrics?
• Question: What is the effect of one additional year of schooling on
earnings?
• Answer:
• About 10% in the US according to David Card (2001)
• Between: 8-13% in the US according to a group of studies
3 / 42
What is econometrics?
• Question: What is the effect of one additional year of schooling on
earnings?
• Answer:
• About 10% in the US according to David Card (2001)
• Between: 8-13% in the US according to a group of studies
• How do they get this number?
3 / 42
What is econometrics?
• Question: What is the effect of one additional year of schooling on
earnings?
• Answer:
• About 10% in the US according to David Card (2001)
• Between: 8-13% in the US according to a group of studies
• How do they get this number?
• Using econometrics
3 / 42
What is econometrics?
• Question: What is the effect of having a child on salaries?
• Answer:
4 / 42
What is econometrics?
• Question: What is the effect of having a child on salaries?
• Answer:
• For men: small increase estimated to be around 6%
4 / 42
What is econometrics?
• Question: What is the effect of having a child on salaries?
• Answer:
• For men: small increase estimated to be around 6%
• For women: significant decrease in earnings that ranges from 10% to
20% of pre-child earnings
• How do they get this number?
4 / 42
What is econometrics?
• Question: What is the effect of having a child on salaries?
• Answer:
• For men: small increase estimated to be around 6%
• For women: significant decrease in earnings that ranges from 10% to
20% of pre-child earnings
• How do they get this number?
• Using econometrics
4 / 42
What is econometrics?
• Question: What is the effect of increasing the minimum wage on
employment?
• Answer:
5 / 42
What is econometrics?
• Question: What is the effect of increasing the minimum wage on
employment?
• Answer:
• How do they get this number?
5 / 42
What is econometrics?
• Question: What is the effect of increasing the minimum wage on
employment?
• Answer:
• How do they get this number?
• Using econometrics
5 / 42
What is econometrics?
• Use of statistical methods to estimate relationships between
variables
• Test theories
6 / 42
What is econometrics?
• Use of statistical methods to estimate relationships between
variables
• Test theories
• Evaluate public policies
6 / 42
What is econometrics?
• Use of statistical methods to estimate relationships between
variables
• Test theories
• Evaluate public policies
• Predict a variable
6 / 42
What is econometrics?
• Use of statistical methods to estimate relationships between
variables
• Test theories
• Evaluate public policies
• Predict a variable
• Importance of data + statistical tools
6 / 42
This course: Advanced statistics
• 36 hours: 3 hours every wednesday
• 2 hours of lectures
• 1 hour of tutorials
• Two parts
1 Basics of econometrics
2 Causal inference
• Grading
• 50% continuous assessment
• 20%: collection + oral participation
• 30%: mid-term
• 50% final exam
• My email: [Link]@[Link]
7 / 42
Step 1: build a model
• We need to have a question in mind
• It could be about testing a theory
• Or more generally about estimating the relationship between
variables
8 / 42
Step 1: build a model
An example
• Question: What drives criminality?
9 / 42
Step 1: build a model
An example
• Question: What drives criminality?
• Theory by Gary Becker: maximizing utility
9 / 42
Step 1: build a model
An example
• Question: What drives criminality?
• Theory by Gary Becker: maximizing utility
• What are the rewards?
9 / 42
Step 1: build a model
An example
• Question: What drives criminality?
• Theory by Gary Becker: maximizing utility
• What are the rewards?
• What are the costs?
9 / 42
Step 1: build a model
An example
• Question: What drives criminality?
• Theory by Gary Becker: maximizing utility
• What are the rewards?
• What are the costs?
• y = f (x1 , x2 , x3 , x4 , x5 , x6 , x7 )
9 / 42
Step 1: build a model
An example
• Question: What drives criminality?
• Theory by Gary Becker: maximizing utility
• What are the rewards?
• What are the costs?
• y = f (x1 , x2 , x3 , x4 , x5 , x6 , x7 )
• y = hours spent on criminal activities
9 / 42
Step 1: build a model
An example
• Question: What drives criminality?
• Theory by Gary Becker: maximizing utility
• What are the rewards?
• What are the costs?
• y = f (x1 , x2 , x3 , x4 , x5 , x6 , x7 )
• y = hours spent on criminal activities
• x1 = salary for criminal activities
• x2 = salary for legal jobs
• x3 = income from different sources
• x4 = probability of being arrested
• x5 = probability of being recognized guilty
• x6 = sentence if guilty
• x7 = age
9 / 42
Step 1: build a model
An example
• Question: What is the impact of professional training on wages?
10 / 42
Step 1: build a model
An example
• Question: What is the impact of professional training on wages?
• y = f (x1 , x2 , x3 )
• y = wages
• x1 = number of years of schooling
• x2 = number of years of experience on the labor market
• x3 = number of weeks of professional training
10 / 42
Step 2: build an econometric model
• y = f (x1 , x2 , x3 , x4 , x5 , x6 , x7 )
• Two problems
1 Specify the function f
2 How do we measure variables that cannot be observed?
11 / 42
Step 2: build an econometric model
• y = f (x1 , x2 , x3 , x4 , x5 , x6 , x7 )
• Two problems
1 Specify the function f
2 How do we measure variables that cannot be observed?
• Solution: crime =
β0 +β1 wage +β2 othinc +β3 freqarr +β4 freqconv +β5 avgsen +β6 age + u
• crime = measure of criminal activity
• wage = salary in a legal job
• othinc = other sources of income
• freqarr = frequency of arrests
• freqconv = frequency of convictions
• avgsen = average length for a sentence
11 / 42
Step 2: build an econometric model
• y = f (x1 , x2 , x3 , x4 , x5 , x6 , x7 )
• Two problems
1 Specify the function f
2 How do we measure variables that cannot be observed?
• Solution: crime =
β0 +β1 wage +β2 othinc +β3 freqarr +β4 freqconv +β5 avgsen +β6 age + u
• crime = measure of criminal activity
• wage = salary in a legal job
• othinc = other sources of income
• freqarr = frequency of arrests
• freqconv = frequency of convictions
• avgsen = average length for a sentence
• What contains the error term u?
11 / 42
Step 2: build an econometric model
• y = f (x1 , x2 , x3 , x4 , x5 , x6 , x7 )
• Two problems
1 Specify the function f
2 How do we measure variables that cannot be observed?
• Solution: crime =
β0 +β1 wage +β2 othinc +β3 freqarr +β4 freqconv +β5 avgsen +β6 age + u
• crime = measure of criminal activity
• wage = salary in a legal job
• othinc = other sources of income
• freqarr = frequency of arrests
• freqconv = frequency of convictions
• avgsen = average length for a sentence
• What contains the error term u?
• Everything else
• Unobserved factors: family context, moral values, etc.
• Measurement errors
11 / 42
Step 2: build an econometric model
• For example 2: y = f (x1 , x2 , x3 )
• Solution: wage = β0 + β1 educ + β2 exper + β3 training + u
• What contains the error term u?
12 / 42
Step 2: build an econometric model
• For example 2: y = f (x1 , x2 , x3 )
• Solution: wage = β0 + β1 educ + β2 exper + β3 training + u
• What contains the error term u?
• u contains innate talent, quality of studies, family context and
everything else
12 / 42
Step 3: Formulate hypotheses
• Once we have the model: we can formulate hypotheses
• Example 1: does increasing legal salary reduce criminal activities?
13 / 42
Step 3: Formulate hypotheses
• Once we have the model: we can formulate hypotheses
• Example 1: does increasing legal salary reduce criminal activities?
• We are interested in β1
• Is β1 statistically significant and negative?
13 / 42
Step 3: Formulate hypotheses
• Once we have the model: we can formulate hypotheses
• Example 1: does increasing legal salary reduce criminal activities?
• We are interested in β1
• Is β1 statistically significant and negative?
• Example 2: does professional training increase productivity?
13 / 42
Step 3: Formulate hypotheses
• Once we have the model: we can formulate hypotheses
• Example 1: does increasing legal salary reduce criminal activities?
• We are interested in β1
• Is β1 statistically significant and negative?
• Example 2: does professional training increase productivity?
• We are interested in β3
• Is it statistically significant? Positive or negative?
• β is called a parameter
13 / 42
Estimating the model: the Data
• To estimate the model, you need data
• Survey data
• Administrative data
• Observational data
• Web and social media data
• Textual data
14 / 42
Estimating the model: the Data
• To estimate the model, you need data
• Survey data
• Administrative data
• Observational data
• Web and social media data
• Textual data
• Different data structures allow to use different techniques
• What are the different data structures?
14 / 42
1. Cross-Sectional Data
• Sample of individuals, households, companies, cities observed at
one point in time
• Random sampling
• We draw randomly x individuals from a population
• Example: 2010 Time Survey data
• Run by French Statistical Institute
• Draw a random sample of individuals (10k) representative of the
French population in 2010
• Compute statistics on this sample
• Informs us on time use surveys of French people
15 / 42
2. Time Series Data
• Variables observed over time
• Examples
• GDP
• Homicide rate per year
• Turnover of a given industry
• We won’t study them
16 / 42
3. Repeated Cross-sectional Data
• Multiple cross-sectional data with different individuals
• Example
• Time use survey in 2000 + Time use survey in 2010
• Different individuals but sample representative of the same
population
• Why?
• Evaluate public policies
• Measure changes over time for a population
17 / 42
4. Panel data
• Same Individuals tracked over time
• Example: UK Understanding Society
• 40,000 households for 100,000 individuals
• Tracked over time from 2009 until today
• Data are usually difficult to obtain
18 / 42
Correlation vs Causality
• Econometrics allow to measure correlations between variables and
causal relationship
• Does correlation imply causality?
19 / 42
Correlation vs Causality
• Econometrics allow to measure correlations between variables and
causal relationship
• Does correlation imply causality?
• No
19 / 42
20 / 42
21 / 42
22 / 42
How to measure causal relationships?
• To be able to say: increasing x has a causal effect of β1 on y ceteris
paribus
• All else being equal
23 / 42
How to measure causal relationships?
• To be able to say: increasing x has a causal effect of β1 on y ceteris
paribus
• All else being equal
• You need an experimental or a quasi-experimental setting
23 / 42
How to measure causal relationships?
• To be able to say: increasing x has a causal effect of β1 on y ceteris
paribus
• All else being equal
• You need an experimental or a quasi-experimental setting
• Multiple designs and techniques
23 / 42
How to measure causal relationships?
• To be able to say: increasing x has a causal effect of β1 on y ceteris
paribus
• All else being equal
• You need an experimental or a quasi-experimental setting
• Multiple designs and techniques
• Randomized controlled trials
23 / 42
How to measure causal relationships?
• To be able to say: increasing x has a causal effect of β1 on y ceteris
paribus
• All else being equal
• You need an experimental or a quasi-experimental setting
• Multiple designs and techniques
• Randomized controlled trials
• Difference-in-differences
23 / 42
How to measure causal relationships?
• To be able to say: increasing x has a causal effect of β1 on y ceteris
paribus
• All else being equal
• You need an experimental or a quasi-experimental setting
• Multiple designs and techniques
• Randomized controlled trials
• Difference-in-differences
• Regression discontinuity
23 / 42
How to measure causal relationships?
• To be able to say: increasing x has a causal effect of β1 on y ceteris
paribus
• All else being equal
• You need an experimental or a quasi-experimental setting
• Multiple designs and techniques
• Randomized controlled trials
• Difference-in-differences
• Regression discontinuity
• Matching
23 / 42
How to measure causal relationships?
• To be able to say: increasing x has a causal effect of β1 on y ceteris
paribus
• All else being equal
• You need an experimental or a quasi-experimental setting
• Multiple designs and techniques
• Randomized controlled trials
• Difference-in-differences
• Regression discontinuity
• Matching
• Instrumental variables
23 / 42
How to measure causal relationships?
• To be able to say: increasing x has a causal effect of β1 on y ceteris
paribus
• All else being equal
• You need an experimental or a quasi-experimental setting
• Multiple designs and techniques
• Randomized controlled trials
• Difference-in-differences
• Regression discontinuity
• Matching
• Instrumental variables
• We’ll cover that later in this course
23 / 42
Time for a Quizz
24 / 42
1 Introduction: what is econometrics
2 The different steps in empirical economic analysis
3 Linear Regression
4 Ordinary Least Squares Estimator
25 / 42
The simple linear regression
• Objective: explain a variable y as a function of x
• Two main challenges:
• How do we take into account the influence of other factors on y
• How do chose the function f?
26 / 42
The simple linear regression
• Objective: explain a variable y as a function of x
• Two main challenges:
• How do we take into account the influence of other factors on y
• How do chose the function f?
• Solution: simple linear regression model y = β0 + β1 x + u
• y is called the dependent variable, explained variable, predicted
variable
• x is called the independent variable, explanatory variable, predictive
variable
26 / 42
The simple linear regression
• Objective: explain a variable y as a function of x
• Two main challenges:
• How do we take into account the influence of other factors on y
• How do chose the function f?
• Solution: simple linear regression model y = β0 + β1 x + u
• y is called the dependent variable, explained variable, predicted
variable
• x is called the independent variable, explanatory variable, predictive
variable
• What is u?
• Error term
• All the other variables that affect y and differ from x
• Also called unobserved variables
26 / 42
The simple linear regression
An example
• We want to explain wages as a function of number of years of
schooling
• Simple model: wage = β0 + β1 educ + u
• How do we interpret β0 and β1 ?
• β0 : average wage for individuals with 0 years of schooling
• β1 impact of one additional year of schooling on wages
27 / 42
The simple linear regression
An example
• We want to explain wages as a function of number of years of
schooling
• Simple model: wage = β0 + β1 educ + u
• How do we interpret β0 and β1 ?
• β0 : average wage for individuals with 0 years of schooling
• β1 impact of one additional year of schooling on wages
• Can we really interpret β1 as the causal effect of schooling on
wages?
27 / 42
The simple linear regression
An example
• We want to explain wages as a function of number of years of
schooling
• Simple model: wage = β0 + β1 educ + u
• How do we interpret β0 and β1 ?
• β0 : average wage for individuals with 0 years of schooling
• β1 impact of one additional year of schooling on wages
• Can we really interpret β1 as the causal effect of schooling on
wages?
• No!
27 / 42
The simple linear regression
• We need two assumptions:
• E (u) = 0
• Error terms are equal to zero on average
• Mild assumption
28 / 42
The simple linear regression
• We need two assumptions:
• E (u) = 0
• Error terms are equal to zero on average
• Mild assumption
• E (u/x ) = E (u)
• The expectation of u is independent of x
• u is independent of x
• Very strong assumption
28 / 42
The simple linear regression
• We need two assumptions:
• E (u) = 0
• Error terms are equal to zero on average
• Mild assumption
• E (u/x ) = E (u)
• The expectation of u is independent of x
• u is independent of x
• Very strong assumption
• What does it imply?
• Previous example: innate aptitude is independent of years of
schooling
• If this assumption is violated: β1 is not the causal effect of x
• E (y /x ) = β0 + β1 x
28 / 42
The simple linear regression
• Example 1: explain the grade as a function of attendance
• score = β0 + β1 attend + u
• In which circumstances does E (u/x ) = E (u)?
29 / 42
The simple linear regression
• Example 1: explain the grade as a function of attendance
• score = β0 + β1 attend + u
• In which circumstances does E (u/x ) = E (u)?
• Example 2: explain vote share for a politician as a function of
advertising
• voteshare = β0 + β1 advertising + u
• In which circumstances does E (u/x ) = E (u)?
29 / 42
The simple linear regression
• Example 1: explain the grade as a function of attendance
• score = β0 + β1 attend + u
• In which circumstances does E (u/x ) = E (u)?
• Example 2: explain vote share for a politician as a function of
advertising
• voteshare = β0 + β1 advertising + u
• In which circumstances does E (u/x ) = E (u)?
• Example 3: explain house prices as a function of criminality in the
neighborhood
• houseprice = β0 + β1 criminality + u
• In which circumstances does E (u/x ) = E (u)?
• It boils down to asking whether x is exogenous
29 / 42
1 Introduction: what is econometrics
2 The different steps in empirical economic analysis
3 Linear Regression
4 Ordinary Least Squares Estimator
30 / 42
Ordinary Least squares estimator
• Question: how do we estimate β0 and β1 ?
• We need a sample of data
• n individuals
• For each individual i, we observe xi and yi
• We have: yi = β0 + β1 xi + ui
• We want to estimate β c0 and β c1 to be as close as possible to
E (y /x ) = β0 + β1 x
• Population regression line : E (y /x ) = β0 + β1 x
• Sample regression line : yb = β
c0 + β
c1 x
31 / 42
Ordinary Least squares estimator
• We use ybi = β
c0 + β
c1 xi to obtain a fitted value for each observation
32 / 42
Ordinary Least squares estimator
• We use ybi = β
c0 + β
c1 xi to obtain a fitted value for each observation
• For each observation: compute the residual ubi
• ubi = yi − ybi = yi − β
c0 − β
c1 x1
32 / 42
Ordinary Least squares estimator
• We use ybi = β
c0 + β
c1 xi to obtain a fitted value for each observation
• For each observation: compute the residual ubi
• ubi = yi − ybi = yi − β
c0 − β
c1 x1
n n
ubi 2 = min (yi − β c1 xi )2
c0 − β
X X
• Objective of OLS: min
β0 ,β1 i =1 β0 ,β1 i =1
32 / 42
Ordinary Least squares estimator
• We use ybi = β
c0 + β
c1 xi to obtain a fitted value for each observation
• For each observation: compute the residual ubi
• ubi = yi − ybi = yi − β
c0 − β
c1 x1
n n
ubi 2 = min (yi − β c1 xi )2
c0 − β
X X
• Objective of OLS: min
β0 ,β1 i =1 β0 ,β1 i =1
• Intuition
• We compute the deviation between the real and the fitted values
• We then minimize the squared sum of these residuals
• Why squared? Why not absolute or power 3 or 4?
32 / 42
Ordinary Least Squares
• Invented in the early 19th century by two mathematicians studying
astronomy
(a) Carl Friedrich Gauss (b) Adrien-Marie Legendre
33 / 42
An Illustration: Fitting a straight line to a scatter
diagram with the OLS
34 / 42
Computations
n n
ubi 2 = min (yi − β c1 xi )2
c0 − β
X X
• Objective of OLS: min
β0 ,β1 i =1 β0 ,β1 i =1
35 / 42
Computations
n n
ubi 2 = min (yi − β c1 xi )2
c0 − β
X X
• Objective of OLS: min
β0 ,β1 i =1 β0 ,β1 i =1
• Question: how do we that?
35 / 42
Computations
n n
ubi 2 = min (yi − β c1 xi )2
c0 − β
X X
• Objective of OLS: min
β0 ,β1 i =1 β0 ,β1 i =1
• Question: how do we that?
• Compute partial derivatives with respect to β0 and β1 and set them
equal to zero:
n
∂ ubi 2
P
i =1
(yi − β
c0 − β
X
= −2 c1 xi ) = 0,
∂β
c0
n
∂ ubi 2
P
i =1
xi (yi − β
c0 − β
X
= −2 c1 xi ) = 0.
∂β
c1
35 / 42
Computations
• We can re-arrange the terms to obtain the normal equations
n n
nβ
c0 + β
X X
yi = c1 xi ,
i =1 i =1
n n n
β xi + β xi2 ,
X X X
xi yi = c0 c1
i =1 i =1 i =1
36 / 42
Computations
• We can re-arrange the terms to obtain the normal equations
n n
nβ
c0 + β
X X
yi = c1 xi ,
i =1 i =1
n n n
β xi + β xi2 ,
X X X
xi yi = c0 c1
i =1 i =1 i =1
• As yi − β0 − β1 xi = ui , we can write the first-order conditions as:
n
∂ ubi 2
P
n
i =1 X
= −2 ui = 0,
∂a i =1
n
∂ ubi 2
P
n
i =1 X
= −2 xi ui = 0.
∂b i =1
36 / 42
The Solutions
• We have two equations with two unknowns
• Solutions :
37 / 42
The Solutions
• We have two equations with two unknowns
• Solutions :
n
P
(xi − x )(yi − y )
c1 = i =1
β β
c0 = y − β
c1 x ,
n
− x )2
P
(xi
i =1
• Can you express β1 as a function of statistical quantities such as
covariance and variance?
37 / 42
Measuring the quality of an estimation
• Question: How well does our sample regression line fit the data?
38 / 42
Measuring the quality of an estimation
• Ultimately: we wish to explain the total variability of a variable Y
• Variability means information
n
(yi − y )2
X
SST =
i =1
39 / 42
Measuring the quality of an estimation
• Ultimately: we wish to explain the total variability of a variable Y
• Variability means information
n
(yi − y )2
X
SST =
i =1
• We are only able to obtain the explain sum of squares with X
n
(ybi − y )2
X
SSE =
i =1
39 / 42
Measuring the quality of an estimation
• Ultimately: we wish to explain the total variability of a variable Y
• Variability means information
n
(yi − y )2
X
SST =
i =1
• We are only able to obtain the explain sum of squares with X
n
(ybi − y )2
X
SSE =
i =1
• We are left with a quantity that we are unable to explain: the
residual sum of squares
n
ubi 2
X
SSR =
i =1
39 / 42
• The following relationship holds:
total sum of = explained sum + residual sum
squares of squares of squares
n n n
(Yi − Y )2 bi − Y )2 ubi2
X X X
= (Y +
i =1 i =1 i =1
SST = SSE + SSR
• These quantities are used to compute the coefficient of
determination R 2
variation in Y due to X SSE
R2 = = .
total variation in Y SST
40 / 42
The coefficient of determination
• The coefficient of determination R 2 is always included between 0
and 1
• What does it mean if it is equal to 1?
41 / 42
The coefficient of determination
• The coefficient of determination R 2 is always included between 0
and 1
• What does it mean if it is equal to 1?
• And to 0?
41 / 42
The coefficient of determination
• The coefficient of determination R 2 is always included between 0
and 1
• What does it mean if it is equal to 1?
• And to 0?
• And to 0.6?
41 / 42
The coefficient of determination
• The coefficient of determination R 2 is always included between 0
and 1
• What does it mean if it is equal to 1?
• And to 0?
• And to 0.6?
• It provides a measure of goodness of fit
• The higher it is, the better the fit
• It is used for prediction but does not say anything about causality
41 / 42
Time for a Quizz
42 / 42