EXPERIMENTS
The simplest experimental design
One group; pre-test/post-test
Internal and external validity
Two groups: Use of a “control” group
Random assignment to groups
Field experiments vs. Natural Experiments
Experiment to critique
Identify and explain the internal validity threats present
EXPERIMENTS
Threats to internal validity
1. History
2. Maturation
3. Testing
4. Instrumentation
5. Mortality
6. Regression to the mean
7. Selection bias
A researcher wanted to assess various impacts of a restorative justice program for youths involved in minor violence. All
youths (N=100) who were charged with minor assaults and had their first court appearance scheduled (but had not yet
appeared) were identified from a large urban court. Before their first appearance, the youths were asked whether they
would go through a restorative justice program which would divert them from the court process. All 100 of the youths who
had scheduled first appearances wanted to go through the program, but there was only space for 50. Thus, the first 50 who
indicated that they wanted to be diverted were chosen and the rest (N=50) were used as the control group. The program
was delivered by a non-profit organization – there were five groups of 10 which met weekly for six weeks. One entire group
ended up dropping out, so in the end, there were only 40 youths. At the last session the researcher attended and held a
“graduation party” for the youths to celebrate the valuable experience they had and their increased empathy for victims. At
this time, the researcher asked the youths to rate how useful the experience was and whether their empathy for victims had
increased during the experience. The control group was asked the exact same questions after their last court appearance
(e.g. to rate how “useful” the experience was and whether their empathy for victims had increased during the experience).
Compared to the control group, the youths who went through the program were significantly more likely to rate the
experience as useful and to report that their empathy had increased substantially during the experience.
Six months later (after the program or after the last court appearance), the researcher contacted all the youths who had
been through the restorative justice program and reminded them about the program and how useful they reported the
program being. The researcher then asked how many times they committed any violent offences during the last six months.
The control group of youths were also contacted and asked the how many times they committed any offences during the
last six months. The researcher found that the program youths reported no offending during the six-month period, while
the control group reported an average of about 7 offences. The researcher concluded that the restorative justice program
not only substantially reduce re-offending compared to the regular court experience, but it also was perceived as more
useful and resulted in a greater increase in empathy. Identify and explain any threats to internal validity with this
experiment.
Selection bias – Identical Ns do not mean the groups are equivalent. The first 50 to sign up might be very
different from the ones who were slower to sign up. They may be more motived, etc so there’s no way to
know whether the differences observed at the end are due to the intervention (restorative justice
program) or due to pre-existing differences between who ended up in the control vs. experimental
groups.
Mortality – lost 10 from the experimental group. That might over-estimate the impact of the program
because those could be the ones who did not find the experience useful/had no empathy
increase/continued to offend but you’ve now lost them, making the group look “better” than it should.
Testing/Sensitization – At the last session the researcher had a “graduation party” for the youths in the
experimental group to celebrate the “valuable experience” they had and their “increased empathy” and
then gave them the questionnaire assessing how useful the program was and their level of empathy. This
might have made the youths in the experimental group aware of what the researcher was looking for. So
the results could be due to these kids conforming to the researchers’ expectations/hopes.
Testing/Sensitization – Again at the follow up re: offending (among the experimental group). The
researcher reminded the kids in the experimental group about the program and how useful they reported
the program being. That reminder might have led to kids in the experimental group to under-report
offending.
Instrumentation – the experimental group was asked about “violent” offending during the last 6 months
while the control group was asked about any offending during the past 6 months. The experimental
group might be reporting less than the control group because they were asked about a much more