Chapter- 5: Evaluation: Some Basics of impact evaluation
5. 1. What Do We Mean by “Impact Evaluation”?
Impact evaluations are a particular type of evaluation that seeks to answer cause-and- effect
questions.
NB. Cause-and- effect questions. The evaluation examines outcomes and tries
to assess what difference the intervention makes in outcomes.
Unlike general evaluations, which can answer many types of questions, impact evaluations are
structured around one particular type of question: What is the impact (or causal effect) of a
program on an outcome of interest?
This basic question incorporates an important causal dimension: we are interested only in the
impact of the program, that is, the effect on outcomes that the program directly causes. An
impact evaluation looks for the changes in outcome that are directly attributable to the program.
Impact evaluation
Investigates the changes brought about by an intervention.
can be undertaken on interventions at any scale: a small, local HIV-AIDS project; an
entire civil society strengthening program of an NGO; a sequence of natural resource
management projects undertaken in a geographic area; or a collection of concurrent
activities by different organizations aimed at improving a community’s capacity.
5. 2. Why Should We Do Impact Evaluation?
Some common reasons for doing impact evaluation include:
• To decide whether to fund an intervention – “ex-ante evaluation” is conducted before an
intervention is implemented, to estimate its likely impacts and inform funding decisions.
• To decide whether or not to continue or expand an intervention.
• To learn how to replicate or scale up a pilot.
• To learn how to successfully adapt a successful intervention to suit another context.
• To reassure funders, including donors and taxpayers (upward accountability), that money
is being wisely invested – including that the organization is learning what does and
doesn’t work, and is using this information to improve future implementation and
investment decisions.
• To inform intended beneficiaries and communities (downward accountability) about whether or
not, and in what ways, a program is benefiting the community.
5.3. Who Should Conduct Impact Evaluation?
Impact evaluation can be undertaken by:
an external evaluator or evaluation team;
an internal but separate unit of the implementing organization;
those involved in an intervention (including community members); or
a combined team of internal and external evaluators.
An external evaluator can bring a range of expertise and experience that might not be available
within the organization, and may have more independence and credibility than an internal
evaluator.
5.4. Methodologies in impact evaluation
5.4.1 Possible methods for examining the factual (the extent to which actual results match what
was expected):
Comparative case studies – did the intervention produce results only in cases when the other
necessary elements were in place?
Dose-response – were there better outcomes for participants who received more of the
intervention (for example, attended more of the workshops or received more support)?
Beneficiary/expert attribution – did participants/key informants believe the intervention had
made a difference, and could they provide a plausible explanation of why this was the case?
Predictions – did those participants or sites predicted to achieve the best impacts (because of
the quality of implementation and/or favorable context) do so? How can anomalies be
explained?
5.4.2 Possible methods for examining the counterfactual (an estimate of what would have
happened in the absence of the intervention) include:
Randomized controlled trial (RCT) – Potential participants (or communities, or households) are
randomly assigned to receive the intervention or be in a control group (either no intervention or
the usual intervention) and the average results of the different groups are compared.
Difference-in-difference – The before-and-after difference for the group receiving the
intervention (where they have not been randomly assigned) is compared to the before-after
difference for those who did not.
Logically constructed counterfactual – In some cases it is credible to use the baseline as an
estimate of the counterfactual. For example, where a water pump has been installed, it might be
reasonable to measure the impact by comparing time spent getting water from a distant pump
before and after the intervention, as there is no credible reason that the time taken would have
decreased without the intervention (White, 2007). Process tracing can support this analysis at
each step of the theory of change.
Matched comparisons – Participants (individuals, organizations or communities) are each
matched with a nonparticipant on variables that are thought to be relevant. It can be difficult to
adequately match on all relevant criteria.
Multiple baselines or rolling baselines – The implementation of an intervention is staggered
across time and intervention populations. Analysis looks for a repeated pattern in each
community of a change in the measured outcome after the intervention is implemented, along
with an absence of substantial fluctuations in the data at other time points. It is increasingly
used for population-level health interventions.