Sage Research Methods

Mixed Methods Design in Evaluation

For the most optimal reading experience we recommend using our website.
A free-to-view version of this content is available by clicking on this link, which
includes an easy-to-navigate-and-search-entry, and may also include videos,
embedded datasets, downloadable datasets, interactive questions, audio
content, and downloadable tables and resources.

Author: Donna M. Mertens

Pub. Date: 2018
Product: Sage Research Methods
DOI: [Link]
Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

Methods: Mixed methods, Evaluation, Case study research

Keywords: healthy eating
Disciplines: Education, Health, Social Policy and Public Policy, Social Work, Sociology
Access Date: August 16, 2025
Publisher: SAGE Publications, Inc
City: Thousand Oaks
Online ISBN: 9781506330631

© 2018 SAGE Publications, Inc All Rights Reserved.

Page 2 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

Mixed Methods Evaluation Designs for Systematic

Reviews

Systematic reviews aim to identify, evaluate and summarise the findings of all relevant individual
studies, thereby making the available evidence more accessible to decision makers. When appro-
priate, combining the results of several studies gives a more reliable and precise estimate of an in-
tervention’s effectiveness than one study alone. (Centre for Reviews and Dissemination, 2009, p. v)

Being more explicit about the personal and political in research and increasing the potential for the
increased involvement of different sections of society nationally and internationally is an important
goal for systematic reviews. (Gough, Oliver, & Thomas, 2012, p. 13)

In This Chapter

• Overview of mixed methods and systematic reviews

• Mixed methods designs for systematic reviews from the Methods branch
• Mixed methods designs for systematic reviews from the Values branch
• Mixed methods designs for systematic reviews from the Use branch
• Mixed methods designs for systematic reviews from the Social Justice branch
• Mixed methods designs for systematic reviews from dialectical pluralism

Mixed Methods and Systematic Literature Review

Systematic literature reviews are powerful ways to bring together the learnings from a large number of studies
of a particular type of intervention or policy. Gough, Oliver, and Thomas (2012) provide the following definition
of a systematic review: “a review of research literature using systematic and explicit, accountable methods”
(p. 2). Systematic reviews can be used by evaluators to answer questions such as What does the available
evidence say about the effects of a particular intervention? Or, What is the impact of a particular policy? The
results of systematic reviews do not provide uncomplicated answers to these questions because of the limi-
tations of studies included in the review in terms of methodological approaches, reporting of results, and pop-

Page 3 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

ulations included or excluded from the studies. The general process for systematic reviews includes these
elements: identify a research question, develop a protocol that specifies the search strategy and establishes
criteria for inclusion and exclusion of studies, identify and describe the relevant research, critically appraise
research reports in a systematic manner, and synthesize findings into a coherent statement.

Systematic reviews can take many forms; the decision about how to proceed with a systematic review is influ-
enced at least in part by the philosophical stance of the reviewers. Some systematic literature reviews focus
on RCTs or quasi-experimental studies that yield quantitative statistics for purposes of compiling the results,
such as the Cochrane Collaboration ([Link]), which focuses on health care; the Campbell Collab-
oration ([Link]), which focuses on education, social welfare, and crime and justice;
and the U.S. Department of Education’s What Works Clearinghouse ([Link] However,
systematic reviews that only use quantitative measures result in a loss of information if the studies on the
topic used mixed methods or qualitative only designs. Mixing results of research that come from different tra-
ditions raises many complex issues. Caracelli and Cooksy (2013) described a systematic review process that
incorporates quantitative, qualitative, and mixed methods studies that emphasize the importance of both the
intellectual quality of the original studies and the context, culture, and values relevant in evaluation studies.

Before jumping into the sample studies, I want to clarify two terms often associated with systematic reviews:
scoping reviews and meta-analysis. Scoping reviews are generally conducted when a topic “has not yet
been extensively reviewed or is of a complex or heterogeneous nature. . . . They are commonly undertak-
en to examine the extent, range, and nature of research activity in a topic area; determine the value and
potential scope and cost of undertaking a full systematic review; summarize and disseminate research find-
ings; and identify research gaps in the existing literature” (Pham et al., 2014, p. 371). They can be done as a
stand-alone study or as a preliminary step in a systematic review. The intent is to map the knowledge domain
of available literature, rather than to provide a summary of the best available research on that topic. Meta-
analysis is a statistical approach to synthesizing findings based on the effect sizes to standardize outcomes
and facilitate comparisons across studies (Mertens, 2015b). Meta-analysis is used with studies that attempt
to establish causal claims between interventions and outcomes. It is commonly used with studies that use
randomized controlled trials (RCTs) or quasi-experimental designs. It can be part of a systematic review if the
types of studies included in the review meet the criteria for inclusion and have the appropriate statistical data
available in the article.

Page 4 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

Methods Branch: Mixed Methods Systematic Reviews

The Cochrane Collaboration has a 25-year history of quantitative systematic reviews of studies that use ran-
domized or quasi-experimental designs. More recently, it has also begun to explore ways to include qualita-
tive data in systematic reviews. Noyes (2010) explains how this collaboration has included qualitative studies
in systematic reviews in the Cochrane Handbook for Systematic Reviews of Interventions.

Qualitative research can contribute to Cochrane intervention reviews in four ways:

1. Informing reviews by using evidence from qualitative research to help define and refine the question,
and to ensure the review includes appropriate studies and addresses important outcomes;
2. Enhancing reviews by synthesising evidence from qualitative research identified whilst looking for ev-
idence of effectiveness;
3. Extending reviews by undertaking a search to specifically seek out evidence from qualitative studies
to address questions directly related to the effectiveness review; and
4. Supplementing reviews by synthesising qualitative evidence within a stand-alone, but complemen-
tary, qualitative review to address questions on aspects other than effectiveness. (Noyes, Popay,
Pearson, Hannes, & Booth, 2008, cited in Noyes, 2010, p. 532)

The Cochrane Collaboration has made progress on using qualitative research for Items 1, 2, and 3 but not
so much progress on the final item. It has produced a guide for mixed methods systematic reviews (Pluye &
Hong, 2014) and one for the conduct of reviews of qualitative data (Noyes & Lewin, 2011).

Thomas et al. (2004) conducted a systematic review related to healthy eating in children in the United King-
dom that used the Cochrane systematic review process as well as a systematic review of qualitative studies
on the topic.

SAMPLE STUDY 5.1: Methods Branch

Mixed methods systematic review: Effectiveness of healthy eating programs for children (Thomas et al.,
2004).

Problem: Many programs have been developed to improve healthy eating in children, but findings have been
conflicting and little is known about the barriers and facilitators for healthy eating among children. Their over-

Page 5 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

all question was “What is known about the barriers to, and facilitators of, healthy eating among children aged
4–10 years?” (p. 1010).

Evaluand: Studies that evaluated the effectiveness of programs that focused on increased consumption of
fruits and vegetables.

Design: MM systematic review was achieved by combining a quantitative review using the Cochrane system
with a qualitative review of the studies that focused on children’s perceptions of and experience with eating
fruits and vegetables, as well as on healthy eating in general.

Sample: Twenty-one quantitative studies were found that met the criteria needed for conducting a meta-analy-
sis. Eight qualitative studies were found that met the criteria of quality of reporting, sufficiency of strategies for
establishing reliability and validity, and rooted in children’s perspectives. The quantitative studies were cho-
sen if they provided data on outcomes before and after the intervention and data on all outcomes measured
and if they used an equivalent control or comparison group.

Data collection: Data were collected using a two-stage process. The evaluators started with a mapping (scop-
ing) exercise to determine the range of studies available and to develop inclusion and exclusion criteria. They
searched multiple databases (e.g., Medline) using the terms children, barriers and facilitators of health pro-
motion, and healthy eating. They then hand searched journals by searching through their reference lists; they
also contacted authors of the included studies and other key organizations in the United Kingdom that pro-
mote healthy eating. They identified studies that fell into the specified scope and analyzed those studies to
determine if they met the criteria for inclusion. The evaluators searched for two types of studies: “controlled
trials (randomized or non-randomized) that examined interventions to promote healthy eating and studies
that examined children’s perspectives and understandings (views studies), often by using qualitative research
methods—for example, in-depth interviews and focus groups” (p. 1010).

Data analysis: An in-depth analysis was conducted for each study that included data extraction and assess-
ment of study quality. The quantitative analysis used meta-analysis for studies with sufficient data. Qualitative
analysis techniques were used to synthesize the findings of the studies of children’s views; they entered the
data into a computer-assisted data analysis program and looked for similarities and differences for barriers,
facilitators, and implied recommendations. The evaluators then integrated the findings from the quantitative
and qualitative results through a cross-study synthesis. The evaluators combined the findings using a matrix

Page 6 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

that “juxtaposed the barriers, facilitators, and implied recommendations against the actual interventions that
had been implemented and evaluated. . . . We had to go back to the original interventions evaluated in the
trials to identify those that built on the barriers and facilitators suggested by the children” (p. 1011).

Results: The results of the meta-analysis indicated that intervention programs have a small but significant
effect. The effect size from the pooled studies suggest that, on average, children will increase their fruit and
vegetable intake by a half a portion a day. The qualitative analysis revealed that children do not have a high
opinion of healthy foods, hence it might be advisable to encourage children to eat tasty fruits and vegetables,
rather than “healthy” ones. The crossover analysis revealed that studies that did not emphasize health mes-
sages were more likely to yield an increase in consumption of fruits and vegetables.

Benefits of using MM: The main benefit of using mixed methods “is that the conclusions of reviews may be
substantially altered by the inclusion of qualitative data, which are more likely to reflect the experiences of
the target groups for intervention” (p. 1012). Greater insights from the target population may lead to more
appropriate recommendations for modified interventions. “The method allows the integration of quantitative
estimates of benefit and harm with qualitative understanding from people’s lives. The insights gained from the
synthesis of qualitative studies allow exploration of statistical heterogeneity in ways that it would be difficult to
imagine in advance” (p. 1012).

Guidance for Designing a Mixed Methods Systematic Review Study in the Methods Branch

1. Systematic reviews can begin with scoping reviews to get an understanding of the range of published
and unpublished literature on the specified topic.
2. Systematic reviews must be carried out according to predetermined standards, including the search
process, the criteria for inclusion and exclusion, and processes for data analysis.
3. Evaluators can start by identifying evaluation questions to frame the study. What is the problem, and
what kind of reviews have already been conducted on this problem? The understanding of the prob-
lem can evolve as the reviewers move through the process of reading articles.
4. Protocols need to be developed for identifying studies. This usually includes deciding on the types of
study designs (e.g., RCTs, quasi-experimental, case studies), the search terms, and the databases
to be searched. It can also include searching gray literature, following up on reference lists, and ask-
ing experts in the field.

Page 7 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

5. Establish or find criteria that exist to assess the quality of the identified studies. Rate the studies ac-
cording to quality.
6. Extract data from the studies, including both quantitative and qualitative data, and record it in a sys-
tematic manner.
7. Synthesize the findings using a statistical procedure where possible, such as meta-analysis, or using
narrative or matrix analysis to look at effects across the studies (adapted from Caracelli & Cooksy,
2013).

Extending Your Thinking

Mixed Methods Systematic Review in the Methods Branch

1. What are your thoughts about combining quantitative and qualitative and mixed methods studies in a
systematic review? Should these be kept separate? How can the data from quantitative, qualitative,
and mixed methods studies best be integrated in a review?
2. Choose a topic of interest to you. Search scholarly databases and web-based resources to see if
there is a systematic review on that topic already. If there is, compare the methods used to those
recommended in this chapter. What are the strengths and weaknesses of their methods?
3. Design a study for a systematic review on a topic of interest to you, following the guidelines found in
this chapter.

Values Branch: Qualitatively Dominant Systematic Re-

views

The process of conducting a systematic review of literature at a surface level looks similar in the Methods
and Values branches, that is, research questions are developed, criteria are set for inclusion and exclusion,
searches of literature are conducted, and data from the selected articles are coded. However, in the Values
branch, systematic literature reviews begin with an acknowledgment of the evaluator/researcher as instru-
ment, that is, what are the characteristics of the evaluators framing and conducting the study, and how do

Page 8 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

these characteristics come into play in the review process? Inclusion of quantitative, qualitative, and mixed
methods studies in the literature sample means that strategies need to be formulated for coding both quanti-
tative and qualitative processes and findings. Archibald et al. (2015) conducted a qualitatively dominant sys-
tematic review of literature on the use of mixed methods in qualitative research journals (see Sample Study
5.2).

SAMPLE STUDY 5.2: Values Branch

Qualitatively dominant mixed methods systematic review: Mixed methods in qualitative journals (Archibald et
al., 2015).

Problem: Mixed methods use is increasing in the research and evaluation communities, yet there remains a
great deal that is unknown about the concept of how mixing actually occurs, especially in qualitatively domi-
nant studies.

Evaluand: Articles published in six leading qualitative research journals from 2003 through 2014 that were
mixed methods in nature.

Design: Systematic review with coding of quantitative and qualitative data on each article simultaneously. In-
tegration of the quantitative and qualitative data at the analysis stage.

Sample: Forty-four theoretically oriented articles and 50 empirical articles made up the sample. The articles
were chosen through team discussion to identify the six qualitative research journals based on their aware-
ness of their importance and influence in the field. Selection of the articles for inclusion was based on their
including both quantitative and qualitative components or by self-identifying as mixed methods studies. If ar-
ticles used several qualitative methods but no quantitative methods or strategies, then they were excluded.

Data collection: The evaluators first discussed the reviewer as the instrument, giving information about them-
selves that was relevant to the choice of journals, criteria for inclusion and exclusion, and coding and inter-
pretation. The evaluators hand searched the six journals for the prescribed date range using keywords such
as mixed and quantitative and qualitative. Each article was read by two reviewers independently to ensure it
met the inclusion criteria.

Data analysis: Two reviewers independently coded each article; for empirical articles, they coded the “re-
search topic/problem, purpose/rationale/philosophy, research questions, design, method fidelity, and imple-

Page 9 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

mentation, as well as overall mixing, interpretive rigor, and rhetoric/terminology” (p. 10). Theoretical articles
were coded for “focus, conceptual position taken, and conclusions drawn/practice recommendations” (p. 10).
Quantitative data were analyzed using descriptive statistics and nonparametric statistics.

Results: Quantitative data revealed that less than 2% of the articles published in the qualitative journals in-
cluded mixed methods, with an increasing trend in the number of articles from 2003 through 2012 and a de-
crease in the subsequent years of the study; 52 percent of the articles specifically identified as mixed meth-
ods. Qualitative data revealed that the definition of mixed methods tended to be rather simplistic, mentioning
the combination of quantitative and qualitative techniques. Almost half the studies reported that the article
was based on a larger study, which typically meant that the quantitative portion was published in another jour-
nal. Hence, it increased the challenge of the reviewers to identify the point of integration for the qualitative
and quantitative portions of the study.

Benefits of using MM: The combination of qualitative and quantitative methods of coding and analysis allowed
the evaluators to address the breadth and quality of mixed methods research published in the preeminent
qualitative research journals. They were able to describe the extent of integration as well as gain insight into
the rationales and methods used for integration of the two types of data. Integration was most commonly ex-
plained in articles that involved the development of instruments. The second most common type of integration
was for purposes of triangulation. The predominant use of sequential designs probably contributed to the sep-
arate reporting of quantitative and qualitative portions of studies and may be a legacy of paradigmatic silos in
professional journals. Nevertheless, this study’s findings highlight the impact of such publishing practices as
not providing sufficient examples of truly mixed designs.

Guidance for Designing a Mixed Methods Systematic Review Study in the Values Branch

1. Practice critical reflexivity to identify characteristics and values relevant to your evaluation work on
the chosen topic.
2. If possible, work with a team to identify the pool of possible literature sources for the study. Conduct
a literature review using standard procedures for database searching, make use of hand searches of
reference lists, and make contact with authors and key organizations.
3. Develop research questions to guide decisions about what to review and what criteria to use for in-
clusion and exclusion. Allow these decisions to be flexible in the early stages of the review process

Page 10 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

so that modifications can be made to accommodate what is learned about the topic.
4. Develop coding processes that allow for both quantitative and qualitative entry of data so that the
combination of data can be used to prepare results and to inform interpretations. Engage in discus-
sions with team members to reach consensus about the coding and the ratings of quality in the arti-
cles.
5. Develop and implement strategies to integrate the data from the quantitative and qualitative codes
into the results and conclusions.

Extending Your Thinking

Mixed Methods Systematic Review in the Values Branch

1. Identify another study that used a qualitatively dominant systematic review design. How did that de-
sign differ from the one used by Archibald et al. (2015)? What do you identify as the strengths and
weaknesses of each approach?
2. Using your own work in evaluation, select a topic for which you can design a qualitatively dominant
systematic review. Design the review process. What kind of criteria will you use for inclusion and ex-
clusion? How will you include diverse perspectives in the study design and implementation?

Use Branch: Mixed Methods Systematic Reviews

Systematic reviews under the Use branch can use the same strategies used in the Methods branch (i.e.,
meta-analysis of experimental studies), but they can also use other strategies to be more inclusive of studies
that are quantitative but not experimental in design, as well as include mixed methods and qualitative studies.
The Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) (Boaz, Ashby, &
Young, 2002) developed a method for systematic review based on the pragmatic position that what works is
not all that matters. Rather, systematic reviews can be used to examine other questions such as what the
problem is, what causes it, and what is being done about it.

Page 11 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

SAMPLE STUDY 5.3: Use Branch

Use branch systematic review: Children and young people with diabetes in educational settings (Edwards,
Noyes, Lowes, Spencer, and Gregory, 2014).

Problem: Type 1 diabetes frequently occurs in children and youth in schools, colleges, and universities.
Whereas students in college and university can generally self-manage the disease, evidence is needed on
how to optimally support children’s diet, exercise, blood glucose monitoring, and insulin regime when they are
in school.

Evaluand: Studies of the effectiveness of intervention-based studies and children, parent, and professional
views of barriers and facilitators to optimal support.

Design: Mixed methods systematic review from the Use branch. Edwards et al. adapted the mixed methods
synthesis guidelines developed by the EPPI-Centre.

Sample: Eleven intervention studies and 55 views studies (i.e., studies that reported the views of children,
parents, or professionals). MM triangulation was used as a guiding framework for building the sample of stud-
ies included in the review. They identified very few randomized controlled studies and so they continued to
search for quantitative studies that provided evidence on attitudes and experiences of participants, as well as
nonintervention survey and qualitative studies.

Data collection: The search strategy included databases such as Medline, Scopus, Cochrane Library, and
PsychINFO. Reference lists and published reviews were also searched. The primary criteria for inclusion was
that the studies focused on young people with type 1 diabetes in an educational setting or on parents, peers,
educational setting personnel, and health professionals who worked with this age group. Quantitative studies
were excluded if they did not include a before-and-after measure, and surveys and qualitative studies were
excluded if they did not report the views of the children, parents, or professionals. Independent reviewers rat-
ed the quality of the studies based on criteria appropriate for quantitative, survey, and qualitative methods.

Data analysis: Meta-analysis was not possible because of the heterogeneity of the populations, interventions,
and outcomes. Quantitative outcome data were entered into a narrative summary table. Qualitative analysis
was performed using [Link] with a coding framework derived from the study’s conceptual framework and
other codes that emerged. Evidence from quantitative studies on attitudes and experiences was mapped onto

Page 12 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

evidence from surveys and qualitative studies. They then conducted an overall narrative synthesis that inte-
grated the findings from both types of studies.

A final overarching synthesis of intervention and non-intervention studies was conducted. For this
final synthesis a matrix was constructed that mapped best practice guidance against the age-relat-
ed barriers and facilitators identified by children and young people, parents, school personnel and
school health professionals and age-related interventions and outcomes in [quantitative studies]. .
. . We were particularly interested to see the extent to which interventions were effective and ad-
dressed the barriers identified by children, parents and teachers/health professionals, and built upon
the facilitators to providing optimal care and management of children and young people with T1D
[type 1 diabetes] in educational settings. (p. 6)

Results: Interventions that were effective included health plans and school nurse support. Short-term effec-
tiveness was noted for telemedicine in school for individual cases, but there was no evidence that this was
linked to longer-term follow-up. The views studies revealed that children, parents, and professionals experi-
enced many barriers, including school structure, organizational policies, educational practices, and nonsup-
portive attitudes. Even though schools had intervention plans, the plans were not implemented in a compre-
hensive way. When staff were supportive, children and parents were very appreciative.

Benefits of using MM: “This novel mixed-method systematic review is the first to integrate intervention effec-
tiveness with views of children/parents/professionals mapped against school diabetes guidelines. Diabetes
management could be generally improved by fully implementing and auditing guideline impact. Evidence is
limited by quality and there are gaps in knowledge of what works” (p. 3).

Guidance for Designing a Mixed Methods Systematic Review Study in the Use Branch

The steps for conducting a Use branch systematic review are adapted from EPPI’s recommendations (Boaz
et al., 2002):

1. Identify a specific question on which to focus the review. The questions can emerge from knowledge
of extant literature and be combined with perspectives from professionals and participants in the pro-
grams.
2. Identify as much research as possible, including published studies that use multiple methods, extant

Page 13 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

reviews, Internet searches, and personal and professional networks. Examples can be found in most
published reviews, including those on the EPPI-Centre website ([Link]
3. Develop exclusion and inclusion criteria for the studies based on the focus questions.
4. Evaluate the quality of the evidence in each study using a protocol that addresses methodological
rigor. Use criteria that are appropriate to the methodology in the reviewed study, that is, different cri-
teria apply to quantitative, qualitative, and mixed methods studies. Examples of protocols can be
found at the Centre for Reviews and Dissemination (2009) website, which also provides detailed guid-
ance on how to put together a protocol (see [Link]/inst/crd/[Link]).
5. Synthesize the studies in different ways, using both meta-analysis and narrative synthesis. Recog-
nize that meta-analysis might not be possible because of the different methodologies and outcomes
included in your pool of studies. Use narrative synthesis to compare studies on similarities and differ-
ences in terms of methods and outcomes to support claims made about the effectiveness of interven-
tions.
6. Integrate quantitative and qualitative data from systematic reviews by creating tables that display the
narrative data from the studies. The use of narrative analysis allows for discussion with stakeholders
of complex interventions that have multiple outcomes.

Extending Your Thinking

Mixed Methods Systematic Review in the Use Branch

1. You have been hired by a firm to conduct a systematic review using Use branch framing on the topic
of reducing recidivism for youthful offenders. How would you begin the study? What question would
you draft as a beginning point for discussions with stakeholders for the review? How would you de-
sign the study using the guidelines provided in the preceding section?
2. What is already known about settling refugees who come from conflict zones? What literature is al-
ready available? How would you design a systematic review using Use branch framing to evaluate
the current state of knowledge about resettling refugees?

Page 14 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

Social Justice Branch: Mixed Methods Systematic Re-

views

Systematic reviews in the Social Justice branch follow a similar pathway as systematic reviews in other
branches in that they use systematic processes for identifying studies and coding the findings of the studies.
However, these reviews are different in that there is a focus on a marginalized population (which can also be
the case under other branches but is essential in the Social Justice branch) and because there is a focus on
the systemic barriers and facilitators that impact the effect of the intervention under study. This means there is
greater interest in identifying qualitative and mixed methods studies that are inclusive of the systemic barriers
and facilitators. The intended use of the systematic review is to inform practice or policy in ways that over-
come barriers of specific relevance in marginalized communities.

Everson-Hock et al. (2013) conducted a Social Justice branch systematic review of the effect of interventions
designed to impact changes in diet and exercise to prevent type 2 diabetes in poor communities in the United
Kingdom. Their study provides an example of how the inclusion of systemic barriers and facilitators can be
achieved for the purpose of informing policy.

SAMPLE STUDY 5.4: Social Justice Branch

Transformative mixed methods systematic review (Everson-Hock et al., 2013).

Problem: Type 2 diabetes is increasing in the United Kingdom, and the increase is especially large in low-in-
come populations. People in low-income populations are less likely to live in conditions that support having a
healthy diet or that permit regular exercise regimes.

Evaluand: Thirty-five studies on diet and exercise to prevent diabetes in the United Kingdom: 9 quantitative,
23 qualitative, and 3 mixed methods. The focus was on studies that used public health campaigns like mass
media and mailings to change behavior.

Design: MM social justice systematic review.

Sample: The evaluators searched multiple databases (e.g., Medline, PsychINFO, and the Cochrane Library)
using search terms such as low socioeconomic status, diabetes, exercise, community physical activities, and

Page 15 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

diet. They also searched the gray literature through the Internet and hand searched relevant literature. They
established criteria for the inclusion of studies. Quantitative studies had to include community-based physical
activity and dietary interventions for a low-SES group, with a comparison to usual care or a placebo, or no
comparison group. The qualitative evaluations and mixed methods studies were included if they assessed
beliefs and perceptions of physical activity and diet for low-SES adults or health care providers. The studies
all had to be from the United Kingdom because the purpose was to influence national policy. Thirty-five stud-
ies were included in the final sample.

Data collection: Two reviewers read each study and extracted the following data: sampling, aims, intervention,
measured outcomes, themes, barriers, and facilitators.

Data analysis: The quality of the studies was ascertained using the National Institute for Health and Clinical
Excellence (NICE) quality assessment checklists (NICE, 2009). The interventions and populations in the
quantitative studies were very heterogeneous; therefore the authors decided not to do a meta-analysis, opting
instead to do narrative synthesis of these studies. For the qualitative data, they conducted thematic analysis,
deriving the themes from the data. To integrate the quantitative and qualitative studies, the authors developed
a matrix that allowed them to assess the extent to which “the interventions incorporated the barriers and fa-
cilitators identified in the qualitative synthesis” (p. 266).

Results: The results of the analysis of the quantitative studies revealed mixed effectiveness in terms of
changes in diet and exercise: “While no single intervention demonstrated a clear positive effect on all out-
come measures considered, some studies showed positive impacts on some outcomes and no intervention
had a negative impact on any outcome” (p. 270). The qualitative analysis yielded 48 barriers and 32 facili-
tators of healthy eating and physical activity. The juxtaposition of quantitative and qualitative findings in the
matrix showed that about half the barriers and facilitators were included in the intervention studies and about
half were not. None of the interventions addressed barriers mentioned by participants such as fear of crime,
unsafe neighborhoods, cost of healthy eating, and family attitudes about health and weight. Thus, interven-
tions could be improved by increased responsiveness to the needs of the low-SES population.

Benefits of using MM: A purely quantitative systematic review of the impact of programs designed to decrease
type 2 diabetes in poor communities would have resulted in an inability to draw a definitive conclusion about
their effectiveness because of the limited number of studies, their heterogeneity, and the exclusion of con-
textual factors that explain the results. By using a social justice mixed methods approach to the systematic

Page 16 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

review, the evaluators were able to include studies that did examine the barriers and facilitators within each
context. In addition, the qualitative analysis part of the study provided insights into variables that could be
introduced into national policies that would potentially increase the effectiveness of interventions. They con-
cluded:

A strength of this review was the inclusion of many types of evidence, which allowed us to explore
effectiveness findings in contextual detail and create explicit links between quantitative and qualita-
tive evidence, using methods appropriate for the data. . . . This enabled us to identify gaps in the
intervention evidence base and thus directions for future research. (p. 270)

Guidance for Designing a Mixed Methods Systematic Review Study in the Social Justice Branch

1. Define the questions to be investigated in a way that includes a focus on marginalized populations
and takes barriers and facilitators into account.
2. Design the search protocol to reflect issues of relevance to the marginalized populations. For exam-
ple, Everson-Hock et al. (2013) included topics such as crime and safety in their search for barriers
and facilitators, concepts that may or may not have surfaced in a systematic review of interventions
to address type 2 diabetes under other branches of evaluation. Also make note of the barriers and
facilitators not addressed in the studies reviewed. For example, Everson-Hock et al. noted that none
of the reviewed studies had included the use of qualitative data to inform the content or implementa-
tion of the interventions themselves, thus revealing a gap in the responsiveness of culture and con-
text that could have implications for program effectiveness.
3. Expand the search to include consultation with community members to determine if there is informa-
tion of relevance that has not been published in scholarly journals but would provide insights into the
variables that influence program effectiveness, particularly of a cultural or contextual nature.
4. Expand the analysis of the extant data using strategies other than or in addition to quantitative meta-
analysis. After critically evaluating each study (often by more than one reviewer), determine what an-
alytic strategies make the most sense given the nature and quality of the quantitative and qualitative
data. Be aware of the heterogeneity of the interventions and the conditions under which the interven-
tions are implemented; code for these characteristics. For example, Everson-Hock et al. (2013) noted
that the goal of the interventions in deprived communities focused on different outcomes, such as
increasing fruit and vegetable intake or choosing and cooking healthy food. They also found that the
interventions were quite heterogeneous, ranging from introducing a new retail outlet for selling food

Page 17 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

in a deprived community to an environment-focused community awareness campaign designed to in-

crease exercise. The differences in the chosen interventions and variables resulted in differences in
outcomes. For this reason, the evaluators felt it was not appropriate to synthesize across the stud-
ies using meta-analysis. Quantitative data may be better analyzed using a narrative synthesis than a
meta-analysis if the heterogeneity among the studies is high. Thematic analysis can be conducted of
the qualitative data that are presented in quantitative, qualitative, and mixed methods studies.
5. The use of a matrix is one method for integrating quantitative and qualitative data. This strategy is
useful, particularly when the quantitative studies have been analyzed using a narrative synthesis.
This allows for the juxtaposition of concepts and can provide greater insights into barriers and facili-
tators that influence impact.
6. Develop a mechanism for disseminating the results of the study to practitioners, policymakers, and
members of the evaluation community so that the results can be used for transformative purposes.
Unfortunately, this is a weak link in the chain of many evaluations and needs to be consciously ad-
dressed if it is to be a reality. Everson-Hock et al. (2013) made recommendations for practitioners,
policymakers, and evaluators regarding how to improve programs, evaluation designs, and policies
by taking cultural and contextual factors into account, especially the role of family, attitudes and per-
ceptions relating to behavior and weight, and fear of crime that are not typically addressed in the cur-
rent work on improving health.

Extending Your Thinking

Mixed Methods Systematic Review in the Social Justice Branch

1. Pick up the newspaper or consult a website. Look for a story about a social problem that is difficult to
solve (e.g., climate change and environmental justice, police-community interactions, health issues).
Design a Social Justice branch systematic review study using the guidelines presented in this chap-
ter.
2. Find a systematic review that does not use a social justice lens as its framing. Critically analyze the
study to identify aspects of the topic that may be overlooked because social justice was not a focal
point in designing or carrying out the study. How would you redesign the study to bring a social justice
lens to the topic?

Page 18 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

Dialectical Pluralism: Mixed Methods Systematic Reviews

DP systematic reviews include adherence to the principles used in the separate evaluation branches that are
brought together under the DP approach. Therefore, if the intent is to conduct a DP study that adheres to the
Methods and Use branches, the evaluator would use a strategy similar to that available from the Campbell
Collaboration or the Cochrane Institute and explained earlier in this chapter under the Methods branch sec-
tion, and the qualitative design would reflect the Use branch strategies explained earlier in this chapter. The
key to a DP approach is to bring the results of the two approaches together to identify issues that arise from
that juxtaposition.

Petrosino and colleagues (2013) used only experimental or quasi-experimental studies in the quantitative por-
tion of their study, following the recommendations of the Campbell Collaboration in that regard. They also
conducted a narrative analysis of the same studies for the qualitative portion of the review and compared the
results of the two approaches.

SAMPLE STUDY 5.5: Dialectical Pluralism

Dialectical pluralism and MM systematic review: Scared Straight (Petrosino et al., 2013).

Problem: Strategies used to deter juveniles from becoming delinquent may do more harm than good. Scared
Straight is one strategy that attempts to scare youth by giving them firsthand observation of prison life and
interaction with incarcerated adults. It might have negative effects.

Evaluand: Studies published up until December 2011 that used either an experimental or quasi-experimental
design and involved programs that brought youth to penal institutions in order to scare them into avoiding
criminal behavior.

Design: Systematic review using quantitative methods and narrative review.

Sample: Literature covered the years 1945 to 2011. Of over 500 citations found through the search process,
only 9 studies met the criteria for design and outcomes; the sample of these studies all together was in excess
of 1,000 students. These spanned the time frame of 1967 to 1992. No more current studies met the criteria
for inclusion. They used the same nine studies for both the quantitative and qualitative portions of the study.

Page 19 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

Data collection: Studies were collected by means of hand searches of 29 leading criminology and social sci-
ence journals; review of prior systematic reviews’ bibliographies; electronic search of relevant databases such
as PsycINFO, ERIC, the Campbell Collaboration Register, the Cochrane Central database, and the Criminal
Justice Abstracts; outreach to researchers and research centers; and in association newsletters. Quantitative
data were extracted from the reports in the form of rates of new offenses or violation of a probation order.
Qualitative data were extracted in the form of the description of the interventions and the methods used in the
studies.

Data analysis: Meta-analysis of postintervention offending rates using official data and narrative analysis of
qualitative data in nine studies. Data were extracted from each study using a data collection instrument devel-
oped by Petrosino. Quantitative and qualitative results of each individual study are included in Petrosino et al.
(2013). The quantitative data were analyzed as the proportion of each group that reoffended. The qualitative
data were analyzed using a narrative approach.

Results: The meta-analysis and narrative analysis indicated that the programs actually do more harm than
doing nothing. Scared Straight actually increases delinquency. The narrative analysis revealed that the de-
scriptions of treatment were quite varied, sometimes not providing much detail, and that the description of
methods used to randomize individuals to groups was not well reported. In one study, 43% of the experimen-
tal group and 17% of the control group reoffended; this quantitative finding was remarkable in that the study’s
authors did not discuss it as a troubling negative outcome in their narrative. Narratives of other studies also
reported negative or nonsignificant outcomes concerning contact with the police, yet they also reported posi-
tive attitudes of parents, teachers, and inmates about the program.

Benefits of using MM: The quantitative analysis based on a meta-analysis of quantitative experimental and
quasi-experimental studies clearly shows a negative effect of the Scared Straight program in terms of in-
creased juvenile delinquent behaviors. The narrative data provides insights into problems with prior studies
in terms of the amount of detail about the intervention and methods used but also in terms of ignoring no
differences and negative outcomes for students and inclusion of positive outcomes in the form of parents,
teachers, and inmates’ perceptions. The conundrum that continues about Scared Straight is why policymak-
ers continue to fund the program when the evidence supports that it is at best ineffective and at worst harmful.
It is ironic that no randomized trials have been conducted on the program since 1992.

Page 20 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

Guidance for Designing a Mixed Methods Systematic Review Study for Dialectical Pluralism

1. If DP is planned to include Methods and Values branch approaches to systematic review, then evalu-
ators can choose the approaches to quantitative and qualitative design from those two approaches.
For example, Petrosino and colleagues (2013) used the specific procedures for a Methods branch
systematic review as prescribed by the Campbell Collaboration. Their approach to the qualitative
parts of the studies might be characterized more as a pragmatic, Use branch design because they
did not seek out additional qualitative studies as would have been done under the Values branch.
2. If the qualitative portion of the study reflects the Use branch, then the narrative analysis follows pro-
cedures as outlined for this approach with a specific focus on finding information that would be useful
to policymakers and practitioners. This is what Petrosino and colleagues did; they wanted to impact
federal policy in the United States by demonstrating the lack of positive outcomes on young people
of the Scared Straight program.
3. Combine the quantitative and qualitative results so that implications can be drawn from both aspects
of the study. Petrosino et al. (2013) did this by noting that the meta-analysis indicated either no or a
negative effect of the Scared Straight program on youth. The qualitative portion of their study revealed
that the authors of the studies had downplayed the negative outcomes for youth and emphasized the
positive perceptions of teachers, parents, and inmates about the program. They noted that this way
of presenting the findings may have led to the justification to continue funding for a program that is
not having the intended effect on youth and might actually be harmful.

Extending Your Thinking

Mixed Methods Systematic Review for Dialectical Pluralism

1. Identify a Campbell Collaboration or a Cochrane Institute meta-analysis on a topic of interest to you.

Use that as a basis for a DP systematic review by conducting a qualitative review on the same topic.
Bring the results of the two studies together. What insights do you gain by adding the qualitative re-
view? What challenges did you encounter in this exercise?
2. Design a DP systematic review on a topic of interest to you. Be explicit about the two branches of
evaluation you are using for the quantitative and qualitative parts of the study. Also be explicit about
how you would bring the results of the two parts of the study into conversation with each other.

Page 21 of 22 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

Summary and Moving Forward

Systematic reviews provide a potentially powerful tool for informing policymakers and other stakeholders
about the effectiveness of a program by bringing together the results of many studies conducted on a par-
ticular topic. The outcomes of systematic reviews clearly depend on what type of and how many studies are
used in the reviews, determined by the evaluator’s search strategies and inclusion and exclusion criteria. In
the Methods branch, the emphasis has been on experimental and quasi-experimental design studies and the
calculation of effect sizes. Qualitative strategies have only recently been making inroads into Methods branch
systematic reviews and take the form of noting contextual information available in the experimental studies.
In the Values branch, more emphasis is placed on the qualitative analysis of articles and development of
themes to understand outcomes expressed in quantitative form. In the Use branch, the quantitative portion
of the study can use the criteria for systematic reviews found in the Methods branch. This can be combined
with qualitative data that can provide insights into different stakeholders’ perspectives. In the Social Justice
branch, literature is consciously chosen to include the experiences of members of marginalized communities,
and the qualitative data are analyzed to reveal power inequities and systemic discrimination at the root of in-
transigent social problems. The focus is also on resilience in communities and identification of strategies that
work. Less emphasis is placed on aggregating quantitative data across studies, and more attention is given
to the contextual factors that support or impede effectiveness. In the DP approach, evaluators may use the
quantitative strategy of meta-analysis combined with qualitative analysis of information about the interven-
tions in the studies.

In the next chapter, we look at mixed methods designs in evaluation that are used for particular purposes,
such as needs assessment, gender analysis, Indigenous settings, evaluations with people with disabilities,
arts-based evaluations, evaluations in conflict zones, evaluations that use visual spatial analysis, and eval-
uations that focus on ongoing development in an organization rather than a specific program. I chose these
specific variations in mixed methods design because the issues that arise are unique and require different
mixed methods approaches.

[Link]

Page 22 of 22 Mixed Methods Design in Evaluation