Econometrics using STATA :
Part 2
Benjamin Monnery
EconomiX, Univ Paris Nanterre
M1 Economie du Droit
2017-2018
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
C ONTENT OF PART 2
When RCT is not an option, only option is to use observational /
real-life data
1. How to retrieve data ?
public sources ([Link]), data repositories, journal archives
how to clean/manipulate data sets in Stata ?
2. How to fix selection bias ?
• when there is only selection on observables (part 2)
i.e. easy problems where you know all the determinants of
assignment correlated with Y
Methods : stratification, covariate-adjustment and matching
x when there is also selection on unobservables (part 3)
Methods : IV, panel, DID, RDD...
B. Monnery (EconomiX) Econometrics using Stata II 2 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
E XAMPLE OF SELECTION ON OBSERVABLES
What’s the effect of lawyers on judicial outcomes ? e.g. Pr(conviction)
among defendants, having a lawyer is “as random” conditional on ...
• strengh of the case (evidence)
• wealth, ...
⇒ Among these determinants of treatment, strengh of case
correlates with Pr(conviction) for sure
what about wealth ? (depends on the judicial system)
Assumption : there is selection on observables (only) if
E[Yi1 |T = 1, X ] = E[Yi1 |T = 0, X ]
E[Yi0 |T = 1, X ] = E[Yi0 |T = 0, X ]
Potential outcomes are the same on average for treated and
untreated with same X
B. Monnery (EconomiX) Econometrics using Stata II 3 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
Finding Data
B. Monnery (EconomiX) Econometrics using Stata II 4 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
Access to data is necessary to answer questions
> know key sources, be able to manipulate their data
Access to novel data is (almost) necessary to publish in top scientific
journals
• good data + good method + interesting topic = top science
• “competition” for data among researchers
• difficult to teach
> be curious, follow the news, learn code
B. Monnery (EconomiX) Econometrics using Stata II 5 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
DATA G OUV
also look at INSEE, ministries’ websites...
B. Monnery (EconomiX) Econometrics using Stata II 6 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
H ARVARD DATAVERSE AND J OURNAL A RCHIVES
Many top scientific journals now require online publication of datasets
(like AER)
[Link]
B. Monnery (EconomiX) Econometrics using Stata II 7 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
C IVIL SOCIETY INITIATIVES
We will use some of their data later in the course (Diff-in-Diff)
B. Monnery (EconomiX) Econometrics using Stata II 8 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
Covariate-adjustment
B. Monnery (EconomiX) Econometrics using Stata II 9 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
I NTUITION
We want to estimate a causal treatment effect by comparing the
observed outcomes of treated and untreated people
If we think we know all the determinants X of treatment assignment T
that also relate to Y (selection on observables), we can simply
compare treated and untreated outcomes conditional on X
How to “condition on X ” ?
1. statistically control for X in a regression model (covariate
adjustment)
estimate Yi = β0 + β1 Ti + β2 Xi + i
2. use matching (e.g. propensity score matching)
3. use stratification (subclassification) :
compute differences within small groups (strata/cells) of X
⇒ Covariate-adjustement is the regression analog to stratification
B. Monnery (EconomiX) Econometrics using Stata II 10 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
In a problem of selection on observables, we want to compare treated
and untreated within subgroups with similar potential outcomes
Ex : what’s the effect of lawyers on defendants’ probability of
conviction ?
⇒ True answer ? Probably a reduction of Pr (conviction)
⇒ Problem (selection bias) : propensity to hire a lawyer and
probability of conviction are both related to strengh of evidence
against defendant
• if court has strong evidence against defendant, he is more likely
to hire a lawyer to help him
• however, he is also more likely to be convicted eventually
⇒ hence risk of selection bias due to differences in strengh of
evidence
If you can measure strengh of evidence, selection bias can be “easily”
eliminated by stratification, covariate-adjustment or matching
B. Monnery (EconomiX) Econometrics using Stata II 11 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
S TRATIFICATION
Tab 1. Sample of Defendants Tab 2. Numbers Convicted
X/T Yes No All X/T Yes No All
Strong 40 10 70 Strong 30 10 40
Weak 10 20 30 Weak 5 15 20
All 50 50 100 All 35 25 60
Stata : tab X T tab X T if Convicted==1
• Naive estimator : compare rates of conviction between Yes & No
Treated : 35/50 = 70% Untreated : 25/50 = 50%
• Naive answer : detrimental “effect” of lawyers of +20% points !
⇒ But strengh of evidence is related to both Lawyers and
Convictions : selection bias
Better estimator : stratify by (condition on) strengh of evidence
B. Monnery (EconomiX) Econometrics using Stata II 12 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
S TRATIFICATION
Tab 1. Sample of Defendants Tab 2. Numbers Convicted
X/T Yes No All X/T Yes No All
Strong 40 10 70 Strong 30 10 40
Weak 10 20 30 Weak 5 15 20
All 50 50 100 All 35 25 60
• Among Strong cases
Treated : 30/40 = 75% Untreated : 10/10 = 100%
Treatment effect : -25pp effect
• Among Weak cases
Treated : 5/10 = 50% Untreated : 15/20 = 75%
Treatment effect : -25pp effect
⇒ Hence the stratified estimator gives a treatment effect of -25 pp
B. Monnery (EconomiX) Econometrics using Stata II 13 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
S TRATIFICATION VERSUS R EGRESSIONS
Stratification solves problems of selection on observables
However in practice, it is only appropriate in the most simple
situations :
• with few variables affecting T and Y
• which are all categorical
• e.g. 1 dummy (strong/weak), 2 dummies (+rich/poor), ...
In real-life, assignment often depends on a large number of
non-dichotomic variables, i.e. need to stratify the sample within a lot
of different groups (cells/strata)
⇒ problem known as the curse of dimensionality
B. Monnery (EconomiX) Econometrics using Stata II 14 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
S TRATIFICATION VERSUS R EGRESSIONS
Problem 1 with stratification : the curse of dimensionality
Assume we want to condition on (stratify by) k dummy variables :
the number of different groups will be 2k
with k = 10, we have 21 0 = 1024 group-specific treatments effects to
compare and average (211 = 2024 , 310 = 59049)
• computation can become long
• many cells will be empty or only contain treated or untreated
observations : can’t compute group-specific effect
> makes the estimated effect less general (i.e. local) as some
observations are left-out
B. Monnery (EconomiX) Econometrics using Stata II 15 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
S TRATIFICATION VERSUS R EGRESSIONS
Problem 2 with stratification : continuous variables
In real-life, many variables are not categorical but continuous
• strong/weak and rich/poor are statistical constructions to ease
calculus
• the true underlying variables are continuous in nature
⇒ stratification makes assumptions of homogeneity within groups
Regressions can easily solve both problems : many X and mix of
categorical and continuous variables
B. Monnery (EconomiX) Econometrics using Stata II 16 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
C OVARIATE ADJUSTMENT
Goal : conditional on X , treatment should be “as random”
Key : control appropriately the effect of wealth and case strengh
• Flexible specification :
- only linear effect Yi = β0 + β1 Lawyeri + β2 Wealthi + i
- or more flexible form : logarithmic, polynomial
(Wealth2 , Wealth3 ,...), by categories/bins, linear+bins...
• Relevant data/variables :
- Use data on the “best” variables explaining treatment
assignment, instead of long-shot proxy variables
annual pre-tax income, disposable income, net wealth, gross
wealth ? Family wealth (to account for possible family support) ?
> a (linear ?) combination of several variables, or some index ?
Recall : do not condition on potential mediators (e.g. lenght of trial) as
they will capture part of the true causal effect of T on Y
B. Monnery (EconomiX) Econometrics using Stata II 17 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
A SSUMPTIONS
The key underlying assumptions :
• Conditional independance assumption (CIA, or
unconfoundedness)
Yi1 , Yi0 ⊥ T | X
CIA is not directly testable (you need to argue why it’s credible)
• Common support (or overlap)
Pr (T = 1|X ) ∈ (0, 1)
common support is easily testable
+ SUTVA
Then stratification, covariate-adjustment and matching will work
B. Monnery (EconomiX) Econometrics using Stata II 18 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
R EGRESSION ANATOMY
Under those assumptions, why exactly does covariate-adjustment
work, i.e. give a causal effect of T on Y ?
⇒ what do multiple regressions do ?
We know that a simple regression with OLS : Yi = β0 + β1 X1 + i
Cov (Y ,X1 )
... gives βˆ1 = Var (X1 )
And a multiple regression with OLS : Yi = β0 + β1 X1 + β2 X2 + ui
... gives β̂ = (X 0 X )−1 X 0 Y ... ?
To understand what it means, let’s turn to the regression anatomy
theorem
B. Monnery (EconomiX) Econometrics using Stata II 19 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
R EGRESSION ANATOMY
B. Monnery (EconomiX) Econometrics using Stata II 20 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
S ENSITIVITY TO CIA
We can estimate how sensitive the results are to potential
confounders
Simulation approach :
• Simulate a “fake” variable F that is correlated with both T and Y
• Look at the effect of including this new covariate F on β̂T
• By comparing the β̂T s under different constructions of F
(variance-covariance), document the sensitivity of your findings
with respect to a violation of CIA
⇒ If β̂T only disappears under “unrealistic” assumptions (super
large correlations (F , X ) and (F , Y )), then the effect is robust to
potential selection on unobservables
B. Monnery (EconomiX) Econometrics using Stata II 21 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
Matching
B. Monnery (EconomiX) Econometrics using Stata II 22 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
M ATCHING
Another popular method to deal with selection on observables is
matching
Matching = Appariemment
Idea : make many pairs of similar individuals (i, j), one treated & one
non-treated, and look at their average differences in outcomes
ˆ = 1
X
ATT (Yi − Yj (i))
N1
T =1
where Yj (i) is the outcome of j, the non-treated individual closest to
the treated i (i.e. the match for i)
B. Monnery (EconomiX) Econometrics using Stata II 23 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
Note that we can also recover ATU and ATE with matching :
ˆ = 1
X
ATU (Yi − Yj (i))
N0
T =0
ˆ = N1 ATT
ATE ˆ + N0 ATU
ˆ
N N
B. Monnery (EconomiX) Econometrics using Stata II 24 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
S IMPLE EXTENSIONS
Note that we can match
• on many dimensions, many X
that’s preferable to make CIA hold
• use several matches for a given i
that’s prefered to reduce variance
M
ˆ = 1 1 X
X
ATT ( Yi − Yjm (i) )
N1 M
T =1 m=1
For now, most simple 1x1 matching on one X
B. Monnery (EconomiX) Econometrics using Stata II 25 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
1 X 1 MATCHING ON ONE X
B. Monnery (EconomiX) Econometrics using Stata II 26 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
1 X 1 MATCHING ON ONE X
B. Monnery (EconomiX) Econometrics using Stata II 27 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
A NOTHER EXAMPLE : 1 X 1 MATCHING ON ONE X
B. Monnery (EconomiX) Econometrics using Stata II 28 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
A NOTHER EXAMPLE : 1 X 1 MATCHING ON ONE X
The estimated ATT after matching is 16426 − 13982 = 2444
whereas before matching : 16426 − 20724 = −4298
B. Monnery (EconomiX) Econometrics using Stata II 29 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
S EVERAL X
In practice, we usually need to match on many observable variables
⇒ difficult to find perfectly similar i and j on all X (exact matching)
Other methods :
• coarsened exact matching (“exact” matching within bins/ranges)
• distance-based matching
- Euclidian distance q
p PK
||xi − xj || = (xi − xj )0 (xi − xj ) = k =1 (xki − xkj )
2
- Normalized Euclidian distance, Mahalanobis distance
• propensity score matching
Distance-based and propensity score matching are most often used
B. Monnery (EconomiX) Econometrics using Stata II 30 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
S EVERAL MATCHES
In practice, we often want to increase precision by using several
matches for each i
• Single nearest neighbor matching
• k-nearest neighbors matching (e.g. k=5 or 10)
• Caliper (or raduis) matching (maximal distance i − j)
• Kernel matching (different weights by distance)
• etc.
Asymptotically, they are all similar ; but in practice, this choice can
matter
B. Monnery (EconomiX) Econometrics using Stata II 31 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
P ROPENSITY SCORE MATCHING
Like with distance-based matching, we want to aggregate all
differences in X in only one index, the propensity score p(x)
p(x) measures the probability that individuals are treated (T = 1)
based on their observables
• Among treated, some were very likely to be treated, some less so
• Among non-treated, some were very likely not to be treated,
some less so
common support in p(x) between the two groups
Propensity score matching matches individuals with similar p(x) (but
different actual treatment status)
⇒ need to estimate p(x)
B. Monnery (EconomiX) Econometrics using Stata II 32 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
P ROPENSITY SCORE MATCHING
To estimate p(x) for each individual (and then match neighbors), we
usually use a probit (or logit) model :
Pr (T = 1|X ) = Pr (T ∗ > 0)
= Pr (X 0 β + > 0)
= Pr ( > −X 0 β)
= 1 − CDF (−X 0 β)
= Phi(X 0 β)
⇒p̂i (xi ) ranges from 0 to 1 (if probit or logit is used)
X are pre-determined variables (and interactions, polynomials, etc.)
likely to explain T
and then predict the scores : p̂i (xi ) = Phi(Xi0 β̂)
⇒ Hopefully with common support and balance of x between the two
groups
B. Monnery (EconomiX) Econometrics using Stata II 33 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
M AIN PRACTICAL ISSUES
Check common support : compare the two distributions of p(x)
Check balance of covariates : use simple t-tests, proportional tests, or
the standarized bias :
if std bias > 20%, difference is still “large”
Be careful about inference : propensity score matching is a two-step
process, so you need to adjust your standard errors (using bootstrap)
Many other choices to make : type of matching (1-1, 1-5, caliper,
kernel, etc), replacement or not...
B. Monnery (EconomiX) Econometrics using Stata II 34 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
BONUS 3 : P RISON - BASED EDUCATION AND RECIDIVISM
Goal : make a 1-page critical review of the paper/chapter
• brief summary of the paper (topic, method, main points, results)
• discuss method, experimental design, interpretations,
conclusions
• relate it to the class
• criticisms, shortcomings ?
Send PDF by email before next monday (noon)
at bmonnery@[Link]
B. Monnery (EconomiX) Econometrics using Stata II 35 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
E XAMPLE : P RISON - BASED EDUCATION AND RECIDIVISM
Data on 31,000 prisoners released in New York State between 2005
and 2008
They follow recidivism within 3 years (rearrest)
Only 347 of them received a college degree in prison
Challenge : make those 347 graduates as comparable as possible to
other prisoners not getting a college degree
Method : match prisoners based on their propensity to get a degree
predicted for 47 covariates
⇒ 1-1 nearest neighbor matching with a caliper of 0.01
B. Monnery (EconomiX) Econometrics using Stata II 36 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
E XAMPLE : P RISON - BASED EDUCATION AND RECIDIVISM
B. Monnery (EconomiX) Econometrics using Stata II 37 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
E XAMPLE : P RISON - BASED EDUCATION AND RECIDIVISM
B. Monnery (EconomiX) Econometrics using Stata II 38 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
E XAMPLE : P RISON - BASED EDUCATION AND RECIDIVISM
B. Monnery (EconomiX) Econometrics using Stata II 39 / 41
F INDING DATA C OVARIATE - ADJUSTMENT M ATCHING
A PPLICATION ON S TATA
Let’s imagine we want to estimate the effect of halfway houses
(semi-liberté) instead of prison on recidivism in a sample of offenders
convicted to prison in France
• allows convicts to work, train, follow classes (probably good for
reentry)
• requires them to return in “custody” every night (probably ok to
monitor offenders)
• often perceived as less punitive (possibly bad for future
deterrence)
⇒ what’s the net causal effect on recidivism, after accounting for
selection ?
Main assumption : the Conditional Independence Assumption holds
after matching on propensity score
In Stata, we can simply use psmatch2
B. Monnery (EconomiX) Econometrics using Stata II 40 / 41
Econometrics using STATA :
Part 2
Benjamin Monnery
EconomiX, Univ Paris Nanterre
M1 Economie du Droit
2017-2018