GUIDELINES AND CHECKLIST FOR

CONSTRUCTIVIST (a.k.a. FOURTH GENERATION) EVALUATION

1
Egon G. Guba & Yvonna S. Lincoln
November 2001

NOTE: The guidelines and checklists for constructivist evaluations and reports outlined herein are based
upon Egon G. Guba and Yvonna S. Lincoln, Fourth Generation Evaluation, Newbury Park, CA: Sage
Publications, 1989. Useful background information may be found in Yvonna S. Lincoln and Egon G.
Guba, Naturalistic Inquiry, Beverly Hills, CA: Sage Publications, 1985.

EVALUATION DEFINED

Evaluation is one of the three basic forms of disciplined inquiry, the others being research and policy
analysis. It is that form of inquiry whose focus is some evaluand (program, process, organization, person,
etc.) and which results in “merit” and/or “worth” constructions (judgments) about it. Merit constructions
converge on the intrinsic quality of an evaluand, irrespective of the setting in which it may find
applications. Worth constructions converge on the extrinsic usefulness or applicability of an evaluand in a
concrete local setting. Evaluation of a proposed or developing evaluand is termed “formative,” while
evaluation of some developed evaluand is termed “summative.”

CONSTRUCTIVIST EVALUATION DEFINED

Constructivist evaluation is that form of evaluation based on the propositions (basic assumptions)
undergirding the constructivist paradigm. The constructivist paradigm differs from other knowledge
paradigms commonly in use, including the scientific, the artistic, the religious, the legal, and others of
similar broad sweep. It is based on three fundamental assumptions, which are commonly termed the
ontological, epistemological, and methodological, viz:

The basic ontological assumption of constructivism is relativism, that is, that human (semiotic)
sense-making that organizes experience so as to render it into apparently comprehensible,
understandable, and explainable form, is an act of construal and is independent of any
foundational reality. Under relativism there can be no “objective” truth. This observation should
not be taken as an “anything goes” position; see the section on criteria below.

The basic epistemological assumption of constructivism is transactional subjectivism, that is, that
assertions about “reality” and “truth” depend solely on the meaning sets (information) and degree
of sophistication available to the individuals and audiences engaged in forming those assertions.

The basic methodological assumption of constructivism is hermeneutic-dialecticism, that is, a

process by which constructions entertained by the several involved individuals and groups
(stakeholders) are first uncovered and plumbed for meaning and then confronted, compared, and
contrasted in encounter situations. The first of these processes is the hermeneutic; the second is
the dialectic. See sections on “discovery” and “assimilation” below. Note that this methodological
assumption is silent on the subject of methods and, in particular, on the subject of “quantitative”
vs. “qualitative” methods. Both types of methods may be and often are appropriate in all forms of
evaluative inquiries.

It is not appropriate to “mix and match” paradigms in conducting an evaluation, for example, utilizing both
scientific (positivist) and constructivist propositions within the same study. This is not a call for “purity” nor

1
Egon G. Guba is Professor Emeritus, Indiana University; Yvonna S. Lincoln is Ruth Harrington
Chair of Educational Leadership and University Distinguished Professor of Higher Education,
Texas A&M University.
Evaluation Checklists Project
[Link]/evalctr/checklists
is it intended to be exclusionary. It is simply a caveat that mixing paradigms may well result in nonsense
approaches and conclusions.

THE TWO PHASES OF CONSTRUCTIVIST EVALUATION: DISCOVERY AND ASSIMILATION

The discovery phase of constructivist evaluation represents the evaluator’s effort to describe “what’s
going on here,” the “here” being the evaluand and its context. The discovery phase may not be needed
(or may be needed only minimally) if there is a preexisting construction or constructions relating to the
evaluand on which to build (e.g., from a prior evaluation or from a project proposal), that is, some
meanings (information) and some level of sophistication in their interpretation are already available.
There are many ways in which the discovery question can be answered, depending on what specific
relevant and preexisting constructions are brought to the inquiry by the evaluator and by local informants
and respondents. Discoveries are themselves semiotic organizations, i.e., mental constructions.
CAVEAT: If the preexisting constructions are drawn from sources external to the subject evaluation, and
in particular from the professional literature, care must be taken to assess their paradigmatic bases; if
those bases are not constructivism, serious disjunctions could easily be overlooked. So, for example,
drawing data from a study cast in positivist terms imbues those data with a truth value, a hard and fast
character, which in constructivist terms they do not deserve. Within a constructivist framework those
same data are seen as variable and transformable, depending on the view of the constructor. To use
those positivist data within a constructivist evaluation undermines the essence of the evaluation. Authors
of evaluation literature, including evaluation reports, that are based on constructivist principles will almost
certainly make their intent plain. In other cases the appearance of key concepts such as generalizability,
objectivity, proof, and the like, typical of positivism and other nonconstructivist approaches, may be key
signals as to the intent of the author.

The assimilation phase of constructivist evaluation represents the evaluator’ s effort to incorporate new
discoveries into the existing construction or constructions (or, if the new discovery is sufficiently different
from or in conflict with the existing construction or constructions, replacing them) so that the “new” (more
informed and sophisticated) construction will fit (subsume older and newer meanings, work (explain what
happens), demonstrate relevance (enable the core problems to be resolved, ameliorated, or better
defined), and exhibit modifiability (be itself open to change).

Discovery and assimilation are not necessarily sequential processes, but may overlap or be carried out in
parallel.

THE PROCESS OF CONSTRUCTIVIST EVALUATION: RESPONSIBILITIES OF THE

CONSTRUCTIVIST EVALUATOR

Constructivist evaluation is a process for doing evaluation that meets two conditions: It is organized by
the claims, concerns, and issues of stakeholding audiences, and it utilizes the methodology of the
constructivist paradigm. Given that mandate, it is possible to list the nine major responsibilities that the
constructivist evaluator must discharge. He or she must:

1. Identify the full array of stakeholders who are at risk by virtue of the stakes they hold in the
entity being evaluated. Such stakes may include but not be limited to money, status, power, face,
opportunity, or other coin; those stakes are determined by and defined by the stakeholders (in their own
terms) and not only by the evaluator or the client seeking the evaluation (although they too are
stakeholders and may identify their own stakes and definitions). Negative stakes may include possible
exploitation, disempowerment, and disenfranchisement. Stakeholders are entitled to receive and
evaluate in their own terms all information that the evaluation may disclose. In the subsequent
hermeneutic/dialectic process, the several stakes entering into the evaluation mix are assessed and
refined in the effort to come as close as possible to negotiated agreement. It is the evaluator’s
responsibility to seek out all stakeholders, including even those who may wish to maintain low visibility or
to absent themselves entirely.

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 2

 2. Elicit from the stakeholder groups their constructions about the form and process of the
evaluand and the range of claims, concerns, and issues they wish to raise in relation to it. The initial list
may be rearranged, deleted, or have additions made as the evaluation proceeds.

3. Provide a context and a methodology (the hermeneutic/dialectic) through which different

constructions of the evaluand, and different claims, concerns, and issues, can be understood, subjected
to critique, and taken into account. The process is first carried out within specific stakeholder groups;
then the products of these intragroup negotiations (defined constructions, claims, concerns, and issues)
are further negotiated in hermeneutic circles that cut across stakeholder groups, if necessary, in dialogic,
adversarial, or confrontational settings.

4. Generate consensus with respect to as many constructions, and their related claims,
concerns, and issues, as possible. Consensus should first be sought on an intragroup basis and then on
an intergroup basis. If consensus can be achieved with respect to an item, it can be eliminated from
further discussion, but retained for further action (and inclusion in the evaluation report) if there is
agreement on that action.

5. Prepare an agenda for negotiation on items about which there is no, or incomplete,
consensus. Failure to reach consensus implies the continuation of competing constructions, which
disjunction(s) can be ameliorated only through the introduction of new information or an increase in the
level of analytic sophistication. The evaluator’s task is to identify the information needed. Because more
information may be required than it is possible to obtain, given time and/or resource constraints, the
evaluator must devise some means (preferably also through a hermeneutic/dialectic process) for
prioritizing the unresolved items. Stakeholder inputs are essential in this determination, lest this need be
taken as an opportunity to disempower selected stakeholders.

6. Collect and provide the information called for in the agenda for negotiation. The provision of
needed information cannot be guaranteed, but the evaluator must make every good faith effort to do so.
Further, if stakeholders lack the sophistication to deal with obtained information, training must be
provided, arranged by the evaluator.

7. Establish and mediate a forum of stakeholder representatives in which negotiation can take
place. Unresolved differences in constructions, as well as unresolved claims, concerns, and issues, are
reviewed in light of the new information and/or level of sophistication, in the hope that their number can
be reduced. It is likely that some items will remain unresolved, thereby setting the stage for another, later
round of evaluation activity. Outcomes of this forum must include action steps if the negotiation is to be
regarded as successful.

8. Develop a report, probably several targeted reports, that communicate to each stakeholder
group any consensus on constructions and any resolutions regarding the claims, concerns, and issues
that they have raised (as well as those raised by other groups that appear relevant to that group). The
most useful form for such report(s) is the case study, which may provide the vicarious experience needed
to influence stakeholder constructions. (See below for additional observations on the reporting process.)

9. Recycle the evaluation to take up still unresolved constructions and their attendant claims,
concerns, and issues. New aspects may be explored that have emerged on the basis of the first-round
evaluation. Constructivist evaluations are never completed; they pause until a further need or opportunity
for review and reassessment emerges.

CONTRACTING FOR A CONSTRUCTIVIST EVALUATION

It is prudent to initiate a contract for an evaluation of any sort, but especially so for a constructivist
evaluation, since this form is neither widely known nor commonly understood. A contract should be
drawn that protects both the client from evaluator misrepresentation or malpractice and the evaluator from
client misunderstanding or misexpectation. Such a contract should cover the following points at a
minimum (other stipulations may be added as seems appropriate in the actual situation).

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 3

 1. Identification of the client or sponsor of the evaluation. Stakeholders in an evaluation are
entitled to know who the client or sponsor is since that position clearly represents a priori power and
vested interests.

2. Identification of the evaluand (the entity to be evaluated). The construction held by the client
or sponsor is at best an initial form that is likely to evolve into a series or progression of constructions that
emerge.

3. A statement of purpose for the evaluation: formative/merit, formative/worth, summative/merit,

summative/worth, or some combination of these forms.

4. A statement of agreement from the client or sponsor particularly, but also from stakeholder
groups, to adhere to the “Conditions for a Productive Hermeneutic/Dialectic.” These include a
commitment from all parties to work from a position of integrity (always recalling that false commitments
can be and sometimes are made), minimal competence on the part of all parties to communicate (a
condition that may exclude children, the mentally handicapped, and psychotic or self-deluded
personalities), a willingness to share power, a willingness to consider change, a willingness to reconsider
one’s own value positions, and a willingness to commit the time and energy needed.

5. A statement of intent from the evaluator with respect to stakeholding audiences. If stakeholder
claims, concerns, and issues (including those of the client or sponsor) are to serve as the focus for the
evaluation, it is clearly necessary to identify, seek out, and involve all relevant stakeholders. The search
for stakeholders should continue throughout the period of the evaluation, since it is highly unlikely that all
stakeholders will be known at the onset of the evaluation. Some stakeholders may refuse to participate,
for example, because they fear to make their positions known because of retaliation from more powerful
groups. In those cases the evaluator must make every effort to construct their likely positions from
whatever sources may be available, e.g., other knowledgeable informants, existing documents, residues
from earlier actions such as legal cases, and the like.

6. A brief description of the methodology to be used. The contract should also note and make
provision for an emergent evaluation design, particularly since clients are likely to assume that an a priori
and thereafter fixed design can be devised. The hermeneutic/dialectic methodology employed in
constructivist evaluations clearly militates against that possibility.

7. A guarantee of access to records, documents, and respondents. This guarantee must take
account of legal protections where they exist and must provide for procedures to be followed in the event
that access becomes blocked.

8. A statement of the evaluator’s intent to guarantee confidentiality and anonymity of information

sources insofar as that can be legally accomplished. It should be noted that an evaluator does not enjoy
special privilege as does, for example, an attorney, clergyman, or physician.

9. A description of the reporting modes to be utilized. The case report is the preferred mode.
The purposes of the case report include providing thick description, giving vicarious experience, serving
as a metaphoric springboard (a form of naturalistic generalization), and challenging constructions of
various stakeholders in ways that lead to reassessment and reconstruction. Reports must be freely
available to all stakeholder groups, and the evaluator must take the responsibility to explicate reports for
those stakeholders who may lack the background, information, or sophistication to deal with them.

10. A listing of technical specifications, including the names and background of the agents who
will carry out the evaluation (allowing for additions and deletions as may be required), a tentative
schedule (not a design), a budget (at all stages a good faith “best estimate”), and a listing of likely
products.

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 4

CONDUCTING THE CONSTRUCTIVIST EVALUATION; THE USE OF THE HERMENEUTIC/DIALETIC
METHODOLOGY

The constructivist evaluation is carried out through a series of steps which, while listed here in serial form,
may well be iterative and reiterative in practice as constructions evolve and as particular claims,
concerns, and issues are dealt with. The serial form below is used as a matter of convenience. The
listing begins at the point at which a contract satisfactory to all parties has been agreed upon.

1. Organizing the evaluation: Selecting the initial team of evaluators, making entree
arrangements, making logistical arrangements, and assessing local political/cultural factors.

2. Identifying stakeholders: Identifying agents commissioning and carrying out the evaluand,
identifying “beneficiaries” as well as “victims” of the evaluand’s action, mounting continuing search
strategies for other stakeholders, assessing trade-offs and sanctions, and formalizing agreements with
and among them.

3. Developing intrastakeholder group constructions: forming multiple hermeneutic circles of 10-

12 members each representing one stakeholder audience; soliciting descriptions (constructions) of the
evaluand and identifying and probing claims, concerns, and issues that emerge, culminating so far as
possible in negotiated agreements on all identified.

4. Enlarging joint intrastakeholder group constructions utilizing the evaluator’s prior construction
(but allotting it no special privilege), existing documentary information, interplay of in-group interview data
with observational data, literature analects, and other sources found to be relevant.

5. Sorting out constructions, claims, concerns, and issues resolved by consensus, setting these
aside as possible case report components.

6. Prioritizing unresolved items via a negotiated prioritizing process determined by and involving
the stakeholder group members.

7. Collecting additional information and adding sophistication in its use by training negotiators,
seeking new information, performing special studies as needed.

8. Preparing the agenda for negotiation by defining and elucidating competing constructions;
working at illuminating, supporting, or refuting items (providing additional training as needed); and testing
the agenda derived.

9. Developing intergroup constructions. Step 8 will have resulted in a negotiated agenda for
each of the several stakeholder groups. This step 9 effectively recapitulates steps 3-8 for a newly formed
hermeneutic circle consisting of persons selected by the individual circles as their representatives. The
result is a composite construction that includes all forms of the evaluand constructions as well as their
relevant claims, concerns, and issues. It is virtually certain that some items will not have been negotiated
to the satisfaction of all stakeholder groups; these are set aside for later reconsideration in a subsequent
recycling .

10. Reporting on the results for Step 9. There may be several reports tailored to the claims,
concerns, and issues of specific stakeholder groups. Agreements on elements of these reports may lead
to proposed action steps. The report should be aimed particularly at the stipulated purpose(s) of the
evaluation, that is, formative/merit, formative/worth, summative/merit, and/or summative worth.

11. Recycle the entire process to take particular account of elements set aside in step 9 that
were irresolvable at that time.

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 5

CONSTRUCTIVIST EVALUATION REPORTS

The end product of a constructivist evaluation (but never a final product, since it is subject to successive
iterations) is the case report. In a sense, a case study is never finished, it is merely due. There may be
multiple reports, targeted to specific stakeholder audiences; and they may take many forms, possibly not
including what might normally be termed a “technical” report, if such a report is beyond the competence
of a stakeholding audience to deal with. The report does not culminate in judgments, conclusions, or
recommendations except insofar as these are concurred on by relevant respondent stakeholders.

Instead, the case report is the joint construction that emerges as the result of the hermeneutic/dialectic
process. Throughout this process the stakeholders—individually, in similar groups, and across groups—
are chosen to uncover widely variable viewpoints. They are exposed to new information and new, more
sophisticated ways of analysis and interpretation until some level of consensus is reached.

The case report helps the reader realize (in the sense of making real), not only the states of affairs that
are believed by stakeholders to exist, but also the underlying motives, feelings, and rationales leading to
those beliefs. The case report is characterized by a thick description that not only clarifies the all-
important context but that makes it possible for the reader to experience it vicariously.

The case report must, finally, contain an appendix that describes in detail the methodology followed and
makes it possible to judge the extent to which quality criteria (those listed in the following section) are
met.

CRITERIA FOR ASSESSING THE QUALITY OF CONSTRUCTIVIST EVALUATIONS AND REPORTS

Standards normally applied in making quality judgments of evaluations, for example, the Joint Committee
Standards or the Guiding Principles for Evaluators of the American Evaluation Association, are
inappropriate for constructivist evaluations precisely because they are based upon a fundamentally
different theoretical paradigm (as explained in the opening paragraphs of this statement). Two different
approaches have been generated to deal with this dilemma; both are useful during the evaluation process
as procedural checklists and afterward in assessing the completed evaluation report (product) for quality:

1. The “parallel” criteria (sometimes called “trustworthiness” or “foundational” criteria). These

evolved from an effort to produce criteria more or less parallel to those conventionally used, i.e., internal
and external validity, reliability, and objectivity. They are probably most useful, first, in guiding
methodological decisions during the evaluation and later in auditing the overall evaluation process (see c
and d below). However, their very “parallelism” to positivist tenets renders them less than fully adequate
for determining the quality of a constructivist approach. These parallel criteria are (full definitions can be
found in Fourth Generation Evaluation, pp. 233-43):

a. Credibility, roughly parallel to internal validity, established by prolonged engagement

at the site, persistent observation, peer debriefing (a kind of external critic), negative case
analysis (a process of reworking postulated hypotheses), progressive subjectivity (continuous
checking of developing constructions against records of constructions that were expected prior to
data collection), and (most important) member checks, continuous testing of hypotheses, data,
preliminary categories, and interpretations with members of stakeholding audiences.

b. Transferability, roughly parallel to external validity, established not by the evaluator

but by receivers of evaluation reports who make personal judgments of the degree to which
findings are sufficiently similar to their own situations (judged from the thick description) to
warrant testing for the viability of local application (testing for localization rather than the more
usual generalization).

c. Dependability, roughly parallel to reliability, established through the use of the

dependability audit with the assistance of an external auditor, who examines the record of the

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 6

 inquiry in the way a fiscal auditor examines fiscal records, to determine the methodological
decisions made and to understand the reasons for them.

d. Confirmability, roughly parallel to objectivity, which determines the extent to which

constructions, assertions, facts, and data can be traced to their sources, the inspection being
done by an external auditor (who may be the same or different from the dependability auditor).
The “raw products” and the “processes used to compress them” are inspected and confirmed as
appropriate.

2. The authenticity criteria. Whereas the parallel criteria are embedded in the assumptions of
positivism, the authenticity criteria are based directly on the assumptions of constructivism and are
responsive to the hermeneutic/dialectic aspects of that paradigm. These criteria are (full definitions can
be found in Fourth Generation Evaluation, pp. 245-250):

a. Fairness, determined by an assessment of the extent to which all competing

constructions have been accessed, exposed, and taken into account in the evaluation report, that
is, in the negotiated emergent construction.

b. Ontological authenticity, determined by an assessment of the extent to which

individual constructions (including those of the evaluator) have become more informed and
sophisticated.

c. Educative authenticity, determined by an assessment of the extent to which

individuals (including the evaluator) have become more understanding (even if not more tolerant)
of the constructions of others.

d. Catalytic authenticity, determined by an assessment of the extent to which action

(clarifying the focus at issue, moving to eliminate or ameliorate problems, sharpening values) is
stimulated and facilitated by the evaluation.

e. Tactical authenticity, determined by an assessment of the extent to which individuals

are empowered to take the action that the evaluation implies or proposes.

Two other observations are appropriate in respect to the quality question. First, the ability of the
hermeneutic/dialectic process to act as a powerful source of quality control should not be overlooked. In
this process, data inputs are analyzed immediately on receipt. They are “fed back” for comment,
elaboration, correction, revision, expansion, or emendation to the very respondents who provided them
just a moment before. Those inputs will, furthermore, be incorporated into the joint, collaborative
reconstruction that emerges as the process continues. Opportunities for errors to go undetected and/or
challenged are very small indeed under those circumstances. It is the immediate and continuing interplay
of information that militates against the possibility of noncredible outcomes. It is difficult to maintain false
fronts or support deliberate deception when information is subject to continuous and multiple challenges
from a variety of stakeholders. The publicly inspectable and inspected nature of the hermeneutic/dialectic
process itself prevents much of the kinds of secrecy and information poverty that have characterized
client-focused evaluations. And finally, any intent on the part of the evaluator to favor particular
stakeholders is at least equally detectable.

Second, for a quality evaluation to result, it is necessary for the evaluator to play a dual (and sometimes
conflicting) role: advocate and educator. In virtually every situation the stakeholding audiences will differ
greatly in the amount of information they bring to the table; the degree to which they can articulate their
existing constructions of the evaluand and the claims, concerns, and issues they experience; and the
degree of sophistication they possess in processing new information that emerges, some of which may
be highly technical. Furthermore, the hermeneutic/dialectic process itself is not one in which they are well
versed; thus, it is incumbent on the evaluator to provide the training (and if necessary, the representation)
they need. The required balance between these roles is delicate, and the evaluator will need to exercise
great care to avoid bias and favoritism.

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 7

CODA

Constructivist evaluation differs in fundamental ways from other forms of evaluation, of which there are
many. In Fourth Generation Evaluation we described the historical evolution of evaluation practice: a
first generation focused on measurement, a second generation focused on description, a third generation
focused on judgment, and a fourth generation focused on negotiation (the hermeneutic/dialectic). It is this
fourth generation form of evaluation that is the subject of this checklist and set of guidelines, now dubbed
constructivist evaluation. We believe this form obviates the major problems of the first three generations:
a tendency toward managerialism, that is, an evaluation approach that favors the point of view of the
client or funder, that inappropriately saves the manager harmless, and that is disempowering, unfair, and
disenfranchising to selected stakeholders; a failure to accommodate value-pluralism; and an
overcommitment to the scientific (positivist) paradigm of inquiry.

Constructivist evaluation is a difficult model to adopt. It is highly labor intensive. It is ever-recursive and
requires frequent recapitulations. If is often adversarial and confrontational. It is a diffuse process
impossible to specify in detail (in design form); hence, its personnel and resource commitments can at
best be “guesstimated.” It requires the evaluator to play multiple roles which at times may appear to be
in conflict. It denies the possibility of reliable generalizations and of determining solutions “that work”
everywhere. Yet from a value-oriented view, it is, we think, the best way to evolve viable and acceptable
solutions to claims, concerns, and issues widely felt and to the formulation of constructions widely seen to
fit, work, demonstrate relevance, and exhibit continuing modifiability. It is one of the more realistic and
socially—and politically—sensitive approaches to performing useful—and utilized—evaluations.

EVALUATION DEFINED

Evaluation is one of the three basic forms of disciplined inquiry, the others being research and policy
analysis. It is that form of inquiry whose focus is some evaluand (program, process, organization, person,
etc.) and which results in “merit” and/or “worth” constructions (judgments) about it. Merit constructions
converge on the intrinsic quality of an evaluand, irrespective of the setting in which it may find
applications. Worth constructions converge on the extrinsic usefulness or applicability of an evaluand in a
concrete local setting. Evaluation of a proposed or developing evaluand is termed “formative,” while
evaluation of some developed evaluand is termed “summative.”

CONSTRUCTIVIST EVALUATION DEFINED

Constructivist evaluation is that form of evaluation based on the propositions (basic assumptions)
undergirding the constructivist paradigm. The constructivist paradigm differs from other knowledge
paradigms commonly in use, including the scientific, the artistic, the religious, the legal, and others of
similar broad sweep. It is based on three fundamental assumptions, which are commonly termed the
ontological, epistemological, and methodological, viz:

The basic ontological assumption of constructivism is relativism, that is, that human (semiotic)
sense-making that organizes experience so as to render it into apparently comprehensible,
understandable, and explainable form, is an act of construal and is independent of any
foundational reality. Under relativism there can be no “objective” truth. This observation should
not be taken as an “anything goes” position; see the section on criteria below.

The basic epistemological assumption of constructivism is transactional subjectivism, that is, that
assertions about “reality” and “truth” depend solely on the meaning sets (information) and degree
of sophistication available to the individuals and audiences engaged in forming those assertions.

The basic methodological assumption of constructivism is hermeneutic-dialecticism, that is, a

process by which constructions entertained by the several involved individuals and groups
(stakeholders) are first uncovered and plumbed for meaning and then confronted, compared, and
contrasted in encounter situations. The first of these processes is the hermeneutic; the second is

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 8

 the dialectic. See sections on “discovery” and “assimilation” below. Note that this methodological
assumption is silent on the subject of methods and, in particular, on the subject of “quantitative”
vs. “qualitative” methods. Both types of methods may be and often are appropriate in all forms of
evaluative inquiries.

It is not appropriate to “mix and match” paradigms in conducting an evaluation, for example, utilizing both
scientific (positivist) and constructivist propositions within the same study. This is not a call for “purity” nor
is it intended to be exclusionary. It is simply a caveat that mixing paradigms may well result in nonsense
approaches and conclusions.

THE TWO PHASES OF CONSTRUCTIVIST EVALUATION: DISCOVERY AND ASSIMILATION

The discovery phase of constructivist evaluation represents the evaluator’s effort to describe “what’s
going on here,” the “here” being the evaluand and its context. The discovery phase may not be needed
(or may be needed only minimally) if there is a preexisting construction or constructions relating to the
evaluand on which to build (e.g., from a prior evaluation or from a project proposal), that is, some
meanings (information) and some level of sophistication in their interpretation are already available.
There are many ways in which the discovery question can be answered, depending on what specific
relevant and preexisting constructions are brought to the inquiry by the evaluator and by local informants
and respondents. Discoveries are themselves semiotic organizations, i.e., mental constructions.
CAVEAT: If the preexisting constructions are drawn from sources external to the subject evaluation, and
in particular from the professional literature, care must be taken to assess their paradigmatic bases; if
those bases are not constructivism, serious disjunctions could easily be overlooked. So, for example,
drawing data from a study cast in positivist terms imbues those data with a truth value, a hard and fast
character, which in constructivist terms they do not deserve. Within a constructivist framework those
same data are seen as variable and transformable, depending on the view of the constructor. To use
those positivist data within a constructivist evaluation undermines the essence of the evaluation. Authors
of evaluation literature, including evaluation reports, that are based on constructivist principles will almost
certainly make their intent plain. In other cases the appearance of key concepts such as generalizability,
objectivity, proof, and the like, typical of positivism and other nonconstructivist approaches, may be key
signals as to the intent of the author.

The assimilation phase of constructivist evaluation represents the evaluator’ s effort to incorporate new
discoveries into the existing construction or constructions (or, if the new discovery is sufficiently different
from or in conflict with the existing construction or constructions, replacing them) so that the “new” (more
informed and sophisticated) construction will fit (subsume older and newer meanings, work (explain what
happens), demonstrate relevance (enable the core problems to be resolved, ameliorated, or better
defined), and exhibit modifiability (be itself open to change).

Discovery and assimilation are not necessarily sequential processes, but may overlap or be carried out in
parallel.

THE PROCESS OF CONSTRUCTIVIST EVALUATION: RESPONSIBILITIES OF THE

CONSTRUCTIVIST EVALUATOR

Constructivist evaluation is a process for doing evaluation that meets two conditions: It is organized by
the claims, concerns, and issues of stakeholding audiences, and it utilizes the methodology of the
constructivist paradigm. Given that mandate, it is possible to list the nine major responsibilities that the
constructivist evaluator must discharge. He or she must:

1. Identify the full array of stakeholders who are at risk by virtue of the stakes they hold in the
entity being evaluated. Such stakes may include but not be limited to money, status, power, face,
opportunity, or other coin; those stakes are determined by and defined by the stakeholders (in their own
terms) and not only by the evaluator or the client seeking the evaluation (although they too are
stakeholders and may identify their own stakes and definitions). Negative stakes may include possible
exploitation, disempowerment, and disenfranchisement. Stakeholders are entitled to receive and

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 9

evaluate in their own terms all information that the evaluation may disclose. In the subsequent
hermeneutic/dialectic process, the several stakes entering into the evaluation mix are assessed and
refined in the effort to come as close as possible to negotiated agreement. It is the evaluator’s
responsibility to seek out all stakeholders, including even those who may wish to maintain low visibility or
to absent themselves entirely.

2. Elicit from the stakeholder groups their constructions about the form and process of the
evaluand and the range of claims, concerns, and issues they wish to raise in relation to it. The initial list
may be rearranged, deleted, or have additions made as the evaluation proceeds.

3. Provide a context and a methodology (the hermeneutic/dialectic) through which different

constructions of the evaluand, and different claims, concerns, and issues, can be understood, subjected
to critique, and taken into account. The process is first carried out within specific stakeholder groups;
then the products of these intragroup negotiations (defined constructions, claims, concerns, and issues)
are further negotiated in hermeneutic circles that cut across stakeholder groups, if necessary, in dialogic,
adversarial, or confrontational settings.

4. Generate consensus with respect to as many constructions, and their related claims,
concerns, and issues, as possible. Consensus should first be sought on an intragroup basis and then on
an intergroup basis. If consensus can be achieved with respect to an item, it can be eliminated from
further discussion, but retained for further action (and inclusion in the evaluation report) if there is
agreement on that action.

5. Prepare an agenda for negotiation on items about which there is no, or incomplete,
consensus. Failure to reach consensus implies the continuation of competing constructions, which
disjunction(s) can be ameliorated only through the introduction of new information or an increase in the
level of analytic sophistication. The evaluator’s task is to identify the information needed. Because more
information may be required than it is possible to obtain, given time and/or resource constraints, the
evaluator must devise some means (preferably also through a hermeneutic/dialectic process) for
prioritizing the unresolved items. Stakeholder inputs are essential in this determination, lest this need be
taken as an opportunity to disempower selected stakeholders.

6. Collect and provide the information called for in the agenda for negotiation. The provision of
needed information cannot be guaranteed, but the evaluator must make every good faith effort to do so.
Further, if stakeholders lack the sophistication to deal with obtained information, training must be
provided, arranged by the evaluator.

7. Establish and mediate a forum of stakeholder representatives in which negotiation can take
place. Unresolved differences in constructions, as well as unresolved claims, concerns, and issues, are
reviewed in light of the new information and/or level of sophistication, in the hope that their number can
be reduced. It is likely that some items will remain unresolved, thereby setting the stage for another, later
round of evaluation activity. Outcomes of this forum must include action steps if the negotiation is to be
regarded as successful.

8. Develop a report, probably several targeted reports, that communicate to each stakeholder
group any consensus on constructions and any resolutions regarding the claims, concerns, and issues
that they have raised (as well as those raised by other groups that appear relevant to that group). The
most useful form for such report(s) is the case study, which may provide the vicarious experience needed
to influence stakeholder constructions. (See below for additional observations on the reporting process.)

9. Recycle the evaluation to take up still unresolved constructions and their attendant claims,
concerns, and issues. New aspects may be explored that have emerged on the basis of the first-round
evaluation. Constructivist evaluations are never completed; they pause until a further need or opportunity
for review and reassessment emerges.

CONTRACTING FOR A CONSTRUCTIVIST EVALUATION

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 10

It is prudent to initiate a contract for an evaluation of any sort, but especially so for a constructivist
evaluation, since this form is neither widely known nor commonly understood. A contract should be
drawn that protects both the client from evaluator misrepresentation or malpractice and the evaluator from
client misunderstanding or misexpectation. Such a contract should cover the following points at a
minimum (other stipulations may be added as seems appropriate in the actual situation).

1. Identification of the client or sponsor of the evaluation. Stakeholders in an evaluation are

entitled to know who the client or sponsor is since that position clearly represents a priori power and
vested interests.

2. Identification of the evaluand (the entity to be evaluated). The construction held by the client
or sponsor is at best an initial form that is likely to evolve into a series or progression of constructions that
emerge.

3. A statement of purpose for the evaluation: formative/merit, formative/worth, summative/merit,

summative/worth, or some combination of these forms.

4. A statement of agreement from the client or sponsor particularly, but also from stakeholder
groups, to adhere to the “Conditions for a Productive Hermeneutic/Dialectic.” These include a
commitment from all parties to work from a position of integrity (always recalling that false commitments
can be and sometimes are made), minimal competence on the part of all parties to communicate (a
condition that may exclude children, the mentally handicapped, and psychotic or self-deluded
personalities), a willingness to share power, a willingness to consider change, a willingness to reconsider
one’s own value positions, and a willingness to commit the time and energy needed.

5. A statement of intent from the evaluator with respect to stakeholding audiences. If stakeholder
claims, concerns, and issues (including those of the client or sponsor) are to serve as the focus for the
evaluation, it is clearly necessary to identify, seek out, and involve all relevant stakeholders. The search
for stakeholders should continue throughout the period of the evaluation, since it is highly unlikely that all
stakeholders will be known at the onset of the evaluation. Some stakeholders may refuse to participate,
for example, because they fear to make their positions known because of retaliation from more powerful
groups. In those cases the evaluator must make every effort to construct their likely positions from
whatever sources may be available, e.g., other knowledgeable informants, existing documents, residues
from earlier actions such as legal cases, and the like.

6. A brief description of the methodology to be used. The contract should also note and make
provision for an emergent evaluation design, particularly since clients are likely to assume that an a priori
and thereafter fixed design can be devised. The hermeneutic/dialectic methodology employed in
constructivist evaluations clearly militates against that possibility.

7. A guarantee of access to records, documents, and respondents. This guarantee must take
account of legal protections where they exist and must provide for procedures to be followed in the event
that access becomes blocked.

8. A statement of the evaluator’s intent to guarantee confidentiality and anonymity of information

sources insofar as that can be legally accomplished. It should be noted that an evaluator does not enjoy
special privilege as does, for example, an attorney, clergyman, or physician.

9. A description of the reporting modes to be utilized. The case report is the preferred mode.
The purposes of the case report include providing thick description, giving vicarious experience, serving
as a metaphoric springboard (a form of naturalistic generalization), and challenging constructions of
various stakeholders in ways that lead to reassessment and reconstruction. Reports must be freely
available to all stakeholder groups, and the evaluator must take the responsibility to explicate reports for
those stakeholders who may lack the background, information, or sophistication to deal with them.

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 11

 10. A listing of technical specifications, including the names and background of the agents who
will carry out the evaluation (allowing for additions and deletions as may be required), a tentative
schedule (not a design), a budget (at all stages a good faith “best estimate”), and a listing of likely
products.

CONDUCTING THE CONSTRUCTIVIST EVALUATION; THE USE OF THE HERMENEUTIC/DIALETIC

METHODOLOGY

The constructivist evaluation is carried out through a series of steps which, while listed here in serial form,
may well be iterative and reiterative in practice as constructions evolve and as particular claims,
concerns, and issues are dealt with. The serial form below is used as a matter of convenience. The
listing begins at the point at which a contract satisfactory to all parties has been agreed upon.

1. Organizing the evaluation: Selecting the initial team of evaluators, making entree
arrangements, making logistical arrangements, and assessing local political/cultural factors.

2. Identifying stakeholders: Identifying agents commissioning and carrying out the evaluand,
identifying “beneficiaries” as well as “victims” of the evaluand’s action, mounting continuing search
strategies for other stakeholders, assessing trade-offs and sanctions, and formalizing agreements with
and among them.

3. Developing intrastakeholder group constructions: forming multiple hermeneutic circles of 10-

12 members each representing one stakeholder audience; soliciting descriptions (constructions) of the
evaluand and identifying and probing claims, concerns, and issues that emerge, culminating so far as
possible in negotiated agreements on all identified.

4. Enlarging joint intrastakeholder group constructions utilizing the evaluator’s prior construction
(but allotting it no special privilege), existing documentary information, interplay of in-group interview data
with observational data, literature analects, and other sources found to be relevant.

5. Sorting out constructions, claims, concerns, and issues resolved by consensus, setting these
aside as possible case report components.

6. Prioritizing unresolved items via a negotiated prioritizing process determined by and involving
the stakeholder group members.

7. Collecting additional information and adding sophistication in its use by training negotiators,
seeking new information, performing special studies as needed.

8. Preparing the agenda for negotiation by defining and elucidating competing constructions;
working at illuminating, supporting, or refuting items (providing additional training as needed); and testing
the agenda derived.

9. Developing intergroup constructions. Step 8 will have resulted in a negotiated agenda for
each of the several stakeholder groups. This step 9 effectively recapitulates steps 3-8 for a newly formed
hermeneutic circle consisting of persons selected by the individual circles as their representatives. The
result is a composite construction that includes all forms of the evaluand constructions as well as their
relevant claims, concerns, and issues. It is virtually certain that some items will not have been negotiated
to the satisfaction of all stakeholder groups; these are set aside for later reconsideration in a subsequent
recycling .

10. Reporting on the results for Step 9. There may be several reports tailored to the claims,
concerns, and issues of specific stakeholder groups. Agreements on elements of these reports may lead
to proposed action steps. The report should be aimed particularly at the stipulated purpose(s) of the
evaluation, that is, formative/merit, formative/worth, summative/merit, and/or summative worth.

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 12

 11. Recycle the entire process to take particular account of elements set aside in step 9 that
were irresolvable at that time.

CONSTRUCTIVIST EVALUATION REPORTS

The end product of a constructivist evaluation (but never a final product, since it is subject to successive
iterations) is the case report. In a sense, a case study is never finished, it is merely due. There may be
multiple reports, targeted to specific stakeholder audiences; and they may take many forms, possibly not
including what might normally be termed a “technical” report, if such a report is beyond the competence
of a stakeholding audience to deal with. The report does not culminate in judgments, conclusions, or
recommendations except insofar as these are concurred on by relevant respondent stakeholders.

Instead, the case report is the joint construction that emerges as the result of the hermeneutic/dialectic
process. Throughout this process the stakeholders—individually, in similar groups, and across groups—
are chosen to uncover widely variable viewpoints. They are exposed to new information and new, more
sophisticated ways of analysis and interpretation until some level of consensus is reached.

The case report helps the reader realize (in the sense of making real), not only the states of affairs that
are believed by stakeholders to exist, but also the underlying motives, feelings, and rationales leading to
those beliefs. The case report is characterized by a thick description that not only clarifies the all-
important context but that makes it possible for the reader to experience it vicariously.

The case report must, finally, contain an appendix that describes in detail the methodology followed and
makes it possible to judge the extent to which quality criteria (those listed in the following section) are
met.

CRITERIA FOR ASSESSING THE QUALITY OF CONSTRUCTIVIST EVALUATIONS AND REPORTS

Standards normally applied in making quality judgments of evaluations, for example, the Joint Committee
Standards or the Guiding Principles for Evaluators of the American Evaluation Association, are
inappropriate for constructivist evaluations precisely because they are based upon a fundamentally
different theoretical paradigm (as explained in the opening paragraphs of this statement). Two different
approaches have been generated to deal with this dilemma; both are useful during the evaluation process
as procedural checklists and afterward in assessing the completed evaluation report (product) for quality:

1. The “parallel” criteria (sometimes called “trustworthiness” or “foundational” criteria). These

evolved from an effort to produce criteria more or less parallel to those conventionally used, i.e., internal
and external validity, reliability, and objectivity. They are probably most useful, first, in guiding
methodological decisions during the evaluation and later in auditing the overall evaluation process (see c
and d below). However, their very “parallelism” to positivist tenets renders them less than fully adequate
for determining the quality of a constructivist approach. These parallel criteria are (full definitions can be
found in Fourth Generation Evaluation, pp. 233-43):

a. Credibility, roughly parallel to internal validity, established by prolonged engagement

at the site, persistent observation, peer debriefing (a kind of external critic), negative case
analysis (a process of reworking postulated hypotheses), progressive subjectivity (continuous
checking of developing constructions against records of constructions that were expected prior to
data collection), and (most important) member checks, continuous testing of hypotheses, data,
preliminary categories, and interpretations with members of stakeholding audiences.

b. Transferability, roughly parallel to external validity, established not by the evaluator

but by receivers of evaluation reports who make personal judgments of the degree to which
findings are sufficiently similar to their own situations (judged from the thick description) to

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 13

 warrant testing for the viability of local application (testing for localization rather than the more
usual generalization).

c. Dependability, roughly parallel to reliability, established through the use of the

dependability audit with the assistance of an external auditor, who examines the record of the
inquiry in the way a fiscal auditor examines fiscal records, to determine the methodological
decisions made and to understand the reasons for them.

d. Confirmability, roughly parallel to objectivity, which determines the extent to which

constructions, assertions, facts, and data can be traced to their sources, the inspection being
done by an external auditor (who may be the same or different from the dependability auditor).
The “raw products” and the “processes used to compress them” are inspected and confirmed as
appropriate.

2. The authenticity criteria. Whereas the parallel criteria are embedded in the assumptions of
positivism, the authenticity criteria are based directly on the assumptions of constructivism and are
responsive to the hermeneutic/dialectic aspects of that paradigm. These criteria are (full definitions can
be found in Fourth Generation Evaluation, pp. 245-250):

a. Fairness, determined by an assessment of the extent to which all competing

constructions have been accessed, exposed, and taken into account in the evaluation report, that
is, in the negotiated emergent construction.

b. Ontological authenticity, determined by an assessment of the extent to which

individual constructions (including those of the evaluator) have become more informed and
sophisticated.

c. Educative authenticity, determined by an assessment of the extent to which

individuals (including the evaluator) have become more understanding (even if not more tolerant)
of the constructions of others.

d. Catalytic authenticity, determined by an assessment of the extent to which action

(clarifying the focus at issue, moving to eliminate or ameliorate problems, sharpening values) is
stimulated and facilitated by the evaluation.

e. Tactical authenticity, determined by an assessment of the extent to which individuals

are empowered to take the action that the evaluation implies or proposes.

Two other observations are appropriate in respect to the quality question. First, the ability of the
hermeneutic/dialectic process to act as a powerful source of quality control should not be overlooked. In
this process, data inputs are analyzed immediately on receipt. They are “fed back” for comment,
elaboration, correction, revision, expansion, or emendation to the very respondents who provided them
just a moment before. Those inputs will, furthermore, be incorporated into the joint, collaborative
reconstruction that emerges as the process continues. Opportunities for errors to go undetected and/or
challenged are very small indeed under those circumstances. It is the immediate and continuing interplay
of information that militates against the possibility of noncredible outcomes. It is difficult to maintain false
fronts or support deliberate deception when information is subject to continuous and multiple challenges
from a variety of stakeholders. The publicly inspectable and inspected nature of the hermeneutic/dialectic
process itself prevents much of the kinds of secrecy and information poverty that have characterized
client-focused evaluations. And finally, any intent on the part of the evaluator to favor particular
stakeholders is at least equally detectable.

Second, for a quality evaluation to result, it is necessary for the evaluator to play a dual (and sometimes
conflicting) role: advocate and educator. In virtually every situation the stakeholding audiences will differ
greatly in the amount of information they bring to the table; the degree to which they can articulate their
existing constructions of the evaluand and the claims, concerns, and issues they experience; and the

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 14

degree of sophistication they possess in processing new information that emerges, some of which may
be highly technical. Furthermore, the hermeneutic/dialectic process itself is not one in which they are well
versed; thus, it is incumbent on the evaluator to provide the training (and if necessary, the representation)
they need. The required balance between these roles is delicate, and the evaluator will need to exercise
great care to avoid bias and favoritism.
CODA

Constructivist evaluation differs in fundamental ways from other forms of evaluation, of which there are
many. In Fourth Generation Evaluation we described the historical evolution of evaluation practice: a
first generation focused on measurement, a second generation focused on description, a third generation
focused on judgment, and a fourth generation focused on negotiation (the hermeneutic/dialectic). It is this
fourth generation form of evaluation that is the subject of this checklist and set of guidelines, now dubbed
constructivist evaluation. We believe this form obviates the major problems of the first three generations:
a tendency toward managerialism, that is, an evaluation approach that favors the point of view of the
client or funder, that inappropriately saves the manager harmless, and that is disempowering, unfair, and
disenfranchising to selected stakeholders; a failure to accommodate value-pluralism; and an
overcommitment to the scientific (positivist) paradigm of inquiry.

Constructivist evaluation is a difficult model to adopt. It is highly labor intensive. It is ever-recursive and
requires frequent recapitulations. If is often adversarial and confrontational. It is a diffuse process
impossible to specify in detail (in design form); hence, its personnel and resource commitments can at
best be “guesstimated.” It requires the evaluator to play multiple roles which at times may appear to be
in conflict. It denies the possibility of reliable generalizations and of determining solutions “that work”
everywhere. Yet from a value-oriented view, it is, we think, the best way to evolve viable and acceptable
solutions to claims, concerns, and issues widely felt and to the formulation of constructions widely seen to
fit, work, demonstrate relevance, and exhibit continuing modifiability. It is one of the more realistic and
socially—and politically—sensitive approaches to performing useful—and utilized—evaluations.

This checklist is being provided as a free service to the user. The provider of the checklist has not
modified or adapted the checklist to fit the specific needs of the user and the user is executing his or her
own discretion and judgment in using the checklist. The provider of the checklist makes no
representations or warranties that this checklist is fit for the particular purpose contemplated by user and
specifically disclaims any such warranties or representations.

Guidelines and Checklist for Constructivist (a.k.a Fourth Generation) Evaluation 15