Tutorial 7: Analysing a field experiment

Evaluation of Public Policies

Edoardo Alberto Viganò

In this lab we replicate key results from a field experiment on corruption. We practise balance
tests, regression with and without robust standard errors, heterogeneous treatment effects,
attrition checks, power analysis, and randomisation inference.

1 Setup

# Install once (run this, then comment it out)

# [Link](c("tidyverse", "sandwich", "lmtest", "marginaleffects", "pwr", "ri2"))

library(tidyverse)
library(sandwich)
library(lmtest)
library(marginaleffects)
library(pwr)
library(ri2)

2 The paper: Olken (2007)

This exercise is based on:

Olken, Benjamin A. 2007. “Monitoring Corruption: Evidence from a Field Experi-

ment in Indonesia.” Journal of Political Economy 115(2): 200–249.

Research question: can top-down audits or grassroots community monitoring reduce corrup-
tion in a public works programme?

1
Setting: 608 Indonesian villages building roads under the Kecamatan Development Project
(KDP), a World Bank-funded programme. Each village received roughly US$8,800 for road
construction.
Experimental design: two independent randomisations.

1. Audit treatment: some subdistricts were told their project would be audited by the
national audit agency (BPKP), raising audit probability from ~4% to ~100%.
2. Invitation treatment (our focus today): some villages received hundreds of invitations to
accountability meetings, encouraging broader community participation in monitoring.

Outcome: missing expenditures — the gap between what the village reported spending and an
independent engineers’ estimate of what the road actually cost. A higher value means more
corruption.

3 Data

3.1 Loading the data

olken <- read_csv("[Link]")

3.2 Variable descriptions

Variable Description
treat_invite 1 if village received invitation treatment, 0 if control
pct_missing % of missing expenditures (our corruption measure)
head_edu Years of education of the village head
mosques Number of mosques per 1,000 population
pct_poor % of village below the poverty line
total_budget Total project budget (millions Rp.)

3.3 Exploring the data

glimpse(olken) # compact overview

2
Rows: 567
Columns: 6
$ treat_invite <dbl> 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, ~
$ pct_missing <dbl> 0.38527447, -0.09574836, 0.14771932, -0.18259122, -
0.2930~
$ head_edu <dbl> 6, 14, 12, 9, 9, 9, 12, 16, 12, 12, 9, 9, 9, 6, 9, 12, 9,~
$ mosques <dbl> 0.9083831, 1.0666667, 0.7117438, 0.9489917, 1.6233766, 0.~
$ pct_poor <dbl> 0.4001222, 0.1856149, 0.4000000, 0.4379366, 0.3126954, 0.~
$ total_budget <dbl> 40.56500, 69.32150, 41.10650, 17.06200, 72.08600, 69.8330~

# How many villages are in each treatment arm?

olken |> count(treat_invite)

# A tibble: 2 x 2
treat_invite n
<dbl> <int>
1 0 191
2 1 376

# Summary statistics for the outcome variable

olken |>
group_by(treat_invite) |>
summarise(mean(pct_missing, [Link] = TRUE))

# A tibble: 2 x 2
treat_invite `mean(pct_missing, [Link] = TRUE)`
<dbl> <dbl>
1 0 0.252
2 1 0.229

The raw difference in means is already visible: invited villages have slightly lower average
missing expenditures.

4 Balance tests

Because treatment was randomised, treatment and control villages should be similar on pre-
treatment characteristics. We check this by regressing each covariate on the treatment indicator.
Significant coefficients would indicate an imbalance that could confound our estimates.

3
balance_edu <- lm(head_edu ~ treat_invite, data = olken)
balance_mosque <- lm(mosques ~ treat_invite, data = olken)
balance_poor <- lm(pct_poor ~ treat_invite, data = olken)
balance_budget <- lm(total_budget ~ treat_invite, data = olken)

summary(balance_edu)

Call:
lm(formula = head_edu ~ treat_invite, data = olken)

Residuals:
Min 1Q Median 3Q Max
-5.503 -2.434 0.566 0.566 8.566

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 11.50262 0.19585 58.730 <2e-16 ***
treat_invite -0.06866 0.24105 -0.285 0.776
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 2.707 on 560 degrees of freedom

(5 observations deleted due to missingness)
Multiple R-squared: 0.0001448, Adjusted R-squared: -0.001641
F-statistic: 0.08112 on 1 and 560 DF, p-value: 0.7759

summary(balance_mosque)

Call:
lm(formula = mosques ~ treat_invite, data = olken)

Residuals:
Min 1Q Median 3Q Max
-1.4121 -0.5770 -0.1739 0.4693 5.4769

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.47382 0.06029 24.447 <2e-16 ***
treat_invite -0.06170 0.07410 -0.833 0.405

4
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.8332 on 563 degrees of freedom

(2 observations deleted due to missingness)
Multiple R-squared: 0.00123, Adjusted R-squared: -0.0005441
F-statistic: 0.6933 on 1 and 563 DF, p-value: 0.4054

summary(balance_poor)

Call:
lm(formula = pct_poor ~ treat_invite, data = olken)

Residuals:
Min 1Q Median 3Q Max
-0.39524 -0.16789 -0.02181 0.16011 0.53945

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.405165 0.015311 26.462 <2e-16 ***
treat_invite 0.008747 0.018837 0.464 0.643
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.2111 on 558 degrees of freedom

(7 observations deleted due to missingness)
Multiple R-squared: 0.0003863, Adjusted R-squared: -0.001405
F-statistic: 0.2156 on 1 and 558 DF, p-value: 0.6426

summary(balance_budget)

Call:
lm(formula = total_budget ~ treat_invite, data = olken)

Residuals:
Min 1Q Median 3Q Max
-71.46 -27.97 -8.74 16.34 810.02

Coefficients:

5
 Estimate Std. Error t value Pr(>|t|)
(Intercept) 81.983 3.760 21.807 <2e-16 ***
treat_invite -1.760 4.621 -0.381 0.703
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 51.96 on 563 degrees of freedom

(2 observations deleted due to missingness)
Multiple R-squared: 0.0002577, Adjusted R-squared: -0.001518
F-statistic: 0.1451 on 1 and 563 DF, p-value: 0.7034

None of the covariates should be significantly related to treatment assignment. What do you
find? This is consistent with successful randomisation.

5 Main results

5.1 Bivariate estimate

We start with the simplest possible model: regress the outcome on the treatment indicator
only.

model1 <- lm(pct_missing ~ treat_invite, data = olken)

summary(model1)

Call:
lm(formula = pct_missing ~ treat_invite, data = olken)

Residuals:
Min 1Q Median 3Q Max
-1.33178 -0.20902 -0.01388 0.18182 1.42220

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.25211 0.02699 9.339 <2e-16 ***
treat_invite -0.02315 0.03322 -0.697 0.486
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.3436 on 475 degrees of freedom

6
 (90 observations deleted due to missingness)
Multiple R-squared: 0.001021, Adjusted R-squared: -0.001082
F-statistic: 0.4856 on 1 and 475 DF, p-value: 0.4862

The coefficient on treat_invite gives us the raw difference in means between treated and
control villages. What is the sign? Is it statistically significant?

5.2 Covariate-adjusted estimate

Adding pre-treatment covariates can reduce residual variance and give a more precise estimate.
Because treatment is randomised, this should not change the point estimate much.

model2 <- lm(pct_missing ~ treat_invite + head_edu + mosques + pct_poor + total_budget,

data = olken)
summary(model2)

Call:
lm(formula = pct_missing ~ treat_invite + head_edu + mosques +
pct_poor + total_budget, data = olken)

Residuals:
Min 1Q Median 3Q Max
-1.28605 -0.21411 -0.01291 0.18932 1.42530

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.3904455 0.0869471 4.491 8.96e-06 ***
treat_invite -0.0264183 0.0331558 -0.797 0.4260
head_edu -0.0055082 0.0058032 -0.949 0.3430
mosques -0.0481914 0.0189702 -2.540 0.0114 *
[ reached getOption("[Link]") -- omitted 2 rows ]
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.3412 on 466 degrees of freedom

(95 observations deleted due to missingness)
Multiple R-squared: 0.0294, Adjusted R-squared: 0.01898
F-statistic: 2.823 on 5 and 466 DF, p-value: 0.01594

How does the estimate on treat_invite compare to the bivariate result? Note that mosques
is significant — we will explore this further below.

7
5.3 Robust standard errors

In practice, residuals from cross-sectional regressions are rarely homoskedastic. We use

heteroskedasticity-consistent (HC0) standard errors from the sandwich package.

coeftest(model2, vcov = vcovHC(model2, type = "HC0"))

t test of coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) 0.39044550 0.09174690 4.2557 2.521e-05 ***
treat_invite -0.02641825 0.03240379 -0.8153 0.415327
head_edu -0.00550816 0.00609873 -0.9032 0.366905
mosques -0.04819141 0.01818008 -2.6508 0.008304 **
[ reached getOption("[Link]") -- omitted 2 rows ]
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Do any standard errors change notably compared to the OLS output above? Does
treat_invite remain insignificant?

6 Heterogeneous effects

The balance test hinted that mosques predicts the outcome independently of treatment. We
now check whether the effect of the invitation treatment varies with village-level characteristics
by including interaction terms.

# Does the effect of invitations depend on mosque density?

inter_mosque <- lm(pct_missing ~ treat_invite * mosques, data = olken)
summary(inter_mosque)

Call:
lm(formula = pct_missing ~ treat_invite * mosques, data = olken)

Residuals:
Min 1Q Median 3Q Max
-1.31756 -0.20871 -0.00924 0.18619 1.43061

8
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.36335 0.05517 6.586 1.21e-10 ***
treat_invite -0.08406 0.06693 -1.256 0.2097
mosques -0.07535 0.03265 -2.308 0.0215 *
treat_invite:mosques 0.04009 0.03986 1.006 0.3150
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.3415 on 473 degrees of freedom

(90 observations deleted due to missingness)
Multiple R-squared: 0.01703, Adjusted R-squared: 0.0108
F-statistic: 2.732 on 3 and 473 DF, p-value: 0.04333

# Does it depend on the village head's education?

inter_edu <- lm(pct_missing ~ treat_invite * head_edu, data = olken)
summary(inter_edu)

Call:
lm(formula = pct_missing ~ treat_invite * head_edu, data = olken)

Residuals:
Min 1Q Median 3Q Max
-1.32923 -0.21471 -0.01403 0.17160 1.45630

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.341106 0.118470 2.879 0.00417 **
treat_invite -0.081699 0.145928 -0.560 0.57584
head_edu -0.007694 0.009971 -0.772 0.44073
treat_invite:head_edu 0.004944 0.012286 0.402 0.68758
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.3442 on 471 degrees of freedom

(92 observations deleted due to missingness)
Multiple R-squared: 0.002714, Adjusted R-squared: -0.003638
F-statistic: 0.4273 on 3 and 471 DF, p-value: 0.7335

9
# Does it depend on poverty rates?
inter_poor <- lm(pct_missing ~ treat_invite * pct_poor, data = olken)
summary(inter_poor)

Call:
lm(formula = pct_missing ~ treat_invite * pct_poor, data = olken)

Residuals:
Min 1Q Median 3Q Max
-1.31350 -0.22960 -0.01545 0.17786 1.38512

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.30282 0.05802 5.219 2.71e-07 ***
treat_invite -0.02686 0.07180 -0.374 0.709
pct_poor -0.12502 0.12811 -0.976 0.330
treat_invite:pct_poor 0.01078 0.15751 0.068 0.945
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.3443 on 470 degrees of freedom

(93 observations deleted due to missingness)
Multiple R-squared: 0.006356, Adjusted R-squared: 1.367e-05
F-statistic: 1.002 on 3 and 470 DF, p-value: 0.3916

# Does it depend on the budget size?

inter_budget <- lm(pct_missing ~ treat_invite * total_budget, data = olken)
summary(inter_budget)

Call:
lm(formula = pct_missing ~ treat_invite * total_budget, data = olken)

Residuals:
Min 1Q Median 3Q Max
-1.32897 -0.21004 -0.00753 0.18179 1.43133

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.2017741 0.0587429 3.435 0.000645 ***

10
treat_invite -0.0126638 0.0673246 -0.188 0.850878
total_budget 0.0006064 0.0006288 0.964 0.335349
treat_invite:total_budget -0.0001245 0.0007063 -0.176 0.860159
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.3431 on 473 degrees of freedom

(90 observations deleted due to missingness)
Multiple R-squared: 0.007677, Adjusted R-squared: 0.001383
F-statistic: 1.22 on 3 and 473 DF, p-value: 0.3019

A significant interaction coefficient means the treatment effect is larger (or smaller) in villages
with higher values of that covariate. What do you find?

6.1 Interpreting interactions with marginaleffects

Interaction coefficients can be hard to interpret from raw regression output. The problem: when
you estimate pct_missing ~ treat_invite * mosques, the coefficient on treat_invite only
tells you the effect of treatment when mosques = 0 — which is not a realistic value.
The marginaleffects package solves this by computing the effect of treatment at realistic
values of the moderator.
Average marginal effect: what is the average effect of the invitation treatment across all villages
in the sample?

avg_slopes(inter_mosque, variables = "treat_invite")

Estimate Std. Error z Pr(>|z|) S 2.5 % 97.5 %

-0.0262 0.033 -0.792 0.428 1.2 -0.0909 0.0386

Term: treat_invite
Type: response
Comparison: 1 - 0

This gives us a single number: the treatment effect averaged across all observed values of
mosques. Compare it to the coefficient in model1 — they should be similar.
Marginal effect at specific values: how does the treatment effect change at low vs. high mosque
density?

11
slopes(inter_mosque, variables = "treat_invite",
newdata = datagrid(mosques = c(0.5, 1, 2, 3)))

mosques Estimate Std. Error z Pr(>|z|) S 2.5 % 97.5 %

0.5 -0.06401 0.0506 -1.2661 0.205 2.3 -0.1631 0.0351
1.0 -0.04397 0.0378 -1.1636 0.245 2.0 -0.1180 0.0301
[ reached 'max' / getOption("[Link]") -- omitted 2 rows ]

Term: treat_invite
Type: response
Comparison: 1 - 0

This shows the estimated treatment effect at four specific values of mosque density. If the
interaction is meaningful, you should see the effect change across rows.
Plot: visualise how the treatment effect varies with mosque density.

plot_slopes(inter_mosque, variables = "treat_invite", condition = "mosques") +

geom_hline(yintercept = 0, linetype = "dashed") +
labs(y = "Effect of invitation treatment on missing expenditures",
x = "Mosques per 1,000 population")
Effect of invitation treatment on missing expenditures

0.50

0.25

0.00

−0.25
0 2 4 6
Mosques per 1,000 population

12
The dashed line at zero represents “no effect.” Where the confidence band crosses that line,
the treatment effect is not statistically significant at that level of mosque density. This plot tells
us much more than the single interaction coefficient.

7 Attrition

Attrition occurs when some units in the sample are missing outcome data. If attrition is
differential — i.e., more common in one treatment arm — it can bias our estimates even in a
randomised experiment.

# Remove observations with missing outcome

olken_full <- olken |> filter()

# How many observations do we lose?

nrow(olken) - nrow(olken_full)

[1] 90

# Is attrition differential across treatment arms?

olken |>
group_by(treat_invite) |>
summarise(mean([Link](pct_missing)) * 100)

# A tibble: 2 x 2
treat_invite `mean([Link](pct_missing)) * 100`
<dbl> <dbl>
1 0 15.2
2 1 16.2

If attrition rates are similar across arms, we can be more confident that our estimates are not
biased by selective non-response. What do you find?

# Visualise the distribution of missing expenditures by treatment arm

boxplot(pct_missing ~ treat_invite, data = olken_full,
names = c("Control", "Invited"), ylab = "Missing expenditures (%)")

13
Missing expenditures (%)

1.0
0.0
−1.0

Control Invited

treat_invite
The distributions look broadly similar. This is consistent with a null average treatment effect.

8 Power analysis

We found no significant effect of invitations on corruption. But maybe we just didn’t have
enough data to detect a real effect? That’s what power analysis helps us check.
The idea: “power” is the probability that your study detects an effect if one truly exists. By
convention, we want power ≥ 0.80 (80%). Power depends on three things:

• Sample size (more data � more power)

• Effect size (bigger effects are easier to detect)
• Significance level (usually α = 0.05)

We use Cohen’s d to measure effect size: the difference in group means divided by the pooled
standard deviation. Rules of thumb: d = 0.2 is small, 0.5 is medium, 0.8 is large.

# How many villages per arm?

olken_full |> count(treat_invite)

# A tibble: 2 x 2
treat_invite n
<dbl> <int>
1 0 162
2 1 315

14
# With our sample sizes, could we detect a medium effect (d = 0.5)?
[Link](n1 = 162, n2 = 315, d = 0.5)

t test power calculation

n1 = 162
n2 = 315
d = 0.5
[Link] = 0.05
power = 0.9993156
alternative = [Link]

Power is very high (~0.99) for a medium effect. So a null finding is not due to lack of power for
anything medium or larger.

# How many villages per group would we need to detect a small effect (d = 0.2)?
[Link](d = 0.2, power = 0.8, type = "[Link]")

Two-sample t test power calculation

n = 393.4057
d = 0.2
[Link] = 0.05
power = 0.8
alternative = [Link]

NOTE: n is number in *each* group

We’d need ~394 per group. Our control group (162) is smaller than that, so we don’t have
enough power to detect very small effects. The takeaway: if the invitation treatment has any
effect at all, it is small.

9 Randomisation inference

Standard regression p-values rely on assumptions about the error distribution. Randomisa-
tion inference takes a different approach: it uses the actual randomisation as the basis for
inference.

15
The idea: imagine the treatment had zero effect on every village. Then it wouldn’t matter
which villages were labelled “treated” — any random assignment would give a similar result.
We can test this by:

1. Shuffling the treatment labels many times

2. Computing the treatment effect each time
3. Checking how often the shuffled effect is as large as the one we actually observed

If our observed effect is unusual compared to the shuffled effects, we reject the null. If not, we
conclude the treatment had no detectable effect.

# Tell R about the randomisation: 477 villages, 315 treated

declaration <- declare_ra(N = nrow(olken_full), m = 315)

# Prepare data
olken_ri <- olken_full |>
select(Z = treat_invite, Y = pct_missing)

# Shuffle labels and compute the effect 1000 times

[Link](43)
ri_result <- conduct_ri(
formula = Y ~ Z,
declaration = declaration,
sharp_hypothesis = 0,
data = olken_ri
)

ri2:::[Link](ri_result)

term estimate two_tailed_p_value

1 Z -0.02314737 0.475

Compare this p-value to the one from model1. They should be very similar — both tell us we
cannot reject the hypothesis that the invitation treatment had zero effect.

16