EVALUATION AND RESEARCH

© LOUIS COHEN, LAWRENCE MANION AND

KEITH MORRISON
 STRUCTURE OF THE CHAPTER
• Similarities between research and evaluation
• Differences between research and evaluation
• Connections between evaluation, research, politics
and policy making

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 DEFINING EVALUATION

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 COMPARING RESEARCH AND EVALUATION
• Origins: Research questions originate from scholars working in a field;
evaluation questions issue from stakeholders.
• Audiences: Evaluations are often commissioned and they become the
property of the sponsors and are not for the public domain; research is
disseminated widely and publicly.
• Purposes: Research contributes to knowledge in the field, regardless of its
practical application, and provides empirical information, i.e. ‘what is’;
evaluation is designed to use that information and those facts to judge the
worth, merit, value, efficacy, impact and effectiveness of something, i.e.
what is valuable. Research is conducted to gain, expand and extend
knowledge; evaluation is conducted to assess performance and to provide
feedback. Research is to generate theory; evaluation is to inform policy
making. Research is to discover; evaluation is to uncover. Research seeks to
predict what will happen; evaluation concerns what has happened or what
is happening.
• Stance: The evaluator is reactive (e.g. to a programme); the researcher is
active and proactive.
• Status: Evaluation is aLawrence
© 2018 Louis Cohen, means to and
Manion anKeith
end; research
Morrison; ischapters,
individual an end in itself.
the contributors
 COMPARING RESEARCH AND EVALUATION
• Focus: Evaluation is concerned with how well something works;
research is concerned with how something works.
• Outcome focus: Evaluation is concerned with the achievement of
intended outcomes; research may not prescribe or know its intended
outcomes in advance (science concerns the unknown).
• Participants: Evaluation focuses almost exclusively on stakeholders;
research has no such focus.
• Scope: Evaluations are concerned with the particular, e.g. a focus only
on specific programmes. They seek to ensure internal validity and
often have a more limited scope than research. Research often seeks
to generalize (external validity) and, indeed, may not include
evaluation.
• Setting of the agenda: The evaluator works within a given brief; the
researcher has greater control over what will be researched (though
often constrained by funding providers). Evaluators work within a set
of ‘givens’, e.g.
© 2018 Louisprogramme, field,
Cohen, Lawrence Manion participants,
and Keith Morrison; individual terms
chapters, theof reference
contributors

and agenda, variables; researchers create and construct the field.

COMPARING RESEARCH AND EVALUATION
• Relevance: Relevance to the programme or what is being evaluated is
a prime feature of evaluations; relevance for researchers has wider
boundaries (e.g. to generalize to a wider community). Research may
be prompted by interest rather than relevance. For the evaluator,
relevance must take account of timeliness and particularity.
• Timeframes: Evaluation begins at the start of the project and finishes
at its end; research is ongoing and less time-bound (though this may
not be the case with funded research).
• Uses of results: Evaluation is designed to improve; research is designed
to demonstrate or prove. Evaluation informs decision making; research
provides a basis for drawing conclusions. Evaluations might be used to
increase or withhold resources or to change practice; research
provides information on which others might or might not act, i.e. it
does not prescribe.
• Decision making: Evaluation is used for micro decision making;
research is used for macro decision making.
• Data sources and types: Evaluation has a wide field of coverage (e.g.
costs, benefits, feasibility, justifiability, needs, value for money), so
evaluators employ a wider and more eclectic range of evidence from
an array of disciplines and sources than researchers.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 COMPARING RESEARCH AND EVALUATION
• Ownership of data: The evaluator often cedes ownership to the
sponsor, upon completion; the researcher holds onto the intellectual
property.
• Politics of the situation: The evaluator may be unable to stand
outside the politics of the purposes and uses of, or participants in, an
evaluation; the researcher provides information for others to use.
• Use of theory: Researchers base their studies in social science
theory; this is not a necessary component of evaluation. Research is
theory-dependent; evaluation is ‘field-dependent’, i.e. not theory-
driven but derived from the participants, the project and
stakeholders. Researchers create the research findings; evaluators
may (or may not) use research findings.
• Reporting: Evaluators report to stakeholders/commissioners of
research; researchers may include these and may also report more
widely, e.g. in publications.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 COMPARING RESEARCH AND EVALUATION
• Standards for judging quality: Judgments of research quality
are made by peers; judgements of evaluation are made by
stakeholders. For researchers, standards for judging quality
include validity, reliability, accuracy, causality, generalizability,
rigour; for evaluators, to these are added utility, feasibility,
involvement of stakeholders, side-effects, efficacy, fitness for
purpose (though, increasingly, utility value and impact are seen
as elements for judging research).

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
SIMILARITIES BETWEEN EVALUATION
AND RESEARCH

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 DIFFERENCES BETWEEN EVALUATION AND
RESEARCH
(Smith, M. & Glass, G. (1987) Research and Evaluation in the Social Sciences. New Jersey: Prentice Hall)

• The intents and purposes of the investigation

• The scope of the investigation
• Values in the investigation
• The origins of the study
• The uses of the study
• The timeliness of the study
• Criteria for judging the study
• The agendas of the study

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
DIFFERENCES BETWEEN EVALUATION AND
RESEARCH
(Norris, N. (1990) Understanding Educational Evaluation. London: Kogan Page)

• The motivation of the enquirer

• The objectives of the research
• Laws versus description
• The role of explanation
• The autonomy of the enquiry

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
DIFFERENCES BETWEEN EVALUATION AND
RESEARCH
(Norris, N. (1990) Understanding Educational Evaluation. London: Kogan Page)

• Properties of the phenomena that are assessed

• Universality of the phenomena studied
• Salience of the value question
• Investigative techniques
• Criteria for assessing the activity
• Disciplinary base

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 CONFORMATIVE EVALUATION
(Stronach, I. and Morris, B. (1994) Polemical notes on educational evaluation in an age of
‘policy hysteria’. Evaluation and Research in Education, 8 (1-2), pp. 5-19)

• Short term, takes project goals as given.

• Ignores the evaluation of longer-term outcomes.
• Gives undue weight to the perceptions of
programme participants who are responsible for the
successful development and implementation of the
programme: ‘over-reports’ change.
• Neglects/‘under-reports’ the views of some
practitioners and critics.

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 CONFORMATIVE EVALUATION
• Adopts an atheoretical approach, and regards the
aggregation of opinion as the determination of
significance.
• Involves a tight contractual relationship with
programme sponsors that disbars public reporting or
encourages self-censorship to protect future funding.
• Risks implicit advocacy of the programme in its
reporting style.

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 MODELS OF EVALUATION

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 STAKE’S MODEL OF EVALUATION
Congruence between intentions & observations – what actually
happened
INTENTIONS Congruence OBSERVATIONS
Intended         Actual
antecedent antecedents
Intended         Actual
transactions transactions
Intended         Actual
outcomes outcomes
Antecedents = initial conditions
Transactions = processes,
© 2018 Louis what
Cohen, Lawrence Manion takes
and Keith Morrison; place during
individual chapters, the programme
the contributors
RESEARCH, POLITICS AND POLICY MAKING

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
RESEARCH, POLITICS AND POLICY
MAKING

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
RESEARCH, POLITICS AND POLICY
MAKING

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors