Sage Research Methods

Mixed Methods Design in Evaluation

For the most optimal reading experience we recommend using our website.
A free-to-view version of this content is available by clicking on this link, which
includes an easy-to-navigate-and-search-entry, and may also include videos,
embedded datasets, downloadable datasets, interactive questions, audio
content, and downloadable tables and resources.

Author: Donna M. Mertens

Pub. Date: 2018
Product: Sage Research Methods
DOI: [Link]
Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

Methods: Mixed methods, Evaluation, Case study research

Keywords: decision making, discipline
Disciplines: Education, Health, Social Policy and Public Policy, Social Work, Sociology
Access Date: August 16, 2025
Publisher: SAGE Publications, Inc
City: Thousand Oaks
Online ISBN: 9781506330631

© 2018 SAGE Publications, Inc All Rights Reserved.

Page 2 of 10 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

Trends, Challenges, and Advances in Mixed Methods

Evaluation

In This Chapter

• Identifying trends and challenges for the future

• Improving evaluation theory and practice as illuminated by the multiple mixed methods frameworks
used by evaluators
• Integration and dissemination in mixed methods evaluation

Trends and Challenges for the Future of Mixed Methods

These are exciting times for evaluators expanding their repertoires to include more sophisticated mixed meth-
ods designs. As noted in Chapter 1, we are in the early stages of what it means to use mixed methods in a
systematic and critically conscious way. This is a time for creativity and divergence of thought, of considera-
tion of possibilities that have not yet been established as standard practice, while at the same time being able
to build on what is known about good practice from quantitative, qualitative, and mixed methodologists. So,
what are the trends and challenges in mixed methods that have potential to improve evaluation theory and
practice?

In this section, I build on the work of a task force of the Mixed Methods International Research Association
to identify those trends and challenges over the next 5 years (Mertens et al., 2016). The issues that arose
through that activity yielded these topics of interest: advances in philosophy and methodology, innovative de-
signs, technological advancements and big data, preparation of mixed methods researchers, and responsive-
ness to complex social problems. These topics have relevance for mixed methods evaluators when they are
looked at in the context of evaluation instead of research.

Page 3 of 10 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

Philosophy and Methodology

As is evident through the multiple philosophical and theoretical frames described in this text, the evaluation
world is in the midst of growing awareness of the importance of assumptions that guides its work and im-
plications for methodological choices. Evaluators have many options in their philosophical orientations to
guide thinking about mixed methods designs, including those that represent quantitatively dominated (Meth-
ods branch), qualitatively dominated (Values branch), pragmatically oriented (Use branch), transformative ori-
ented (Social Justice branch), and designs that bring paradigms into conversation with each other (dialectical
pluralism). As we move into the future, evaluators will find fertile ground in sharing their understandings of the
meanings of the philosophical assumptions and how they inform their work, as well as through efforts to cross
over between the branches (as is suggested in the water metaphor presented in Chapter 1).

What will it mean to bring evaluators who were trained in different branches together to engage in conversa-
tions about the influence of philosophical assumptions and their positive and limiting influences? What mech-
anisms are available and appropriate for sustaining conversations across branches in evaluation in respectful
ways? What kinds of evaluation questions might emerge that would be different and more sophisticated than
“Does it work?” or “Does it work better than another approach?” Will mixed methods designs developed from
a deeper understanding of philosophical assumptions lead to improved methodologies that have increased
potential to contribute to societal improvements? How will mixed methods designs from different evaluation
branches inform understandings about context, the nature of the problems, the determination of appropriate
solutions, and the use of evaluation for decision making?

Innovative Designs

As is also evident throughout this text, the concept of mixed methods designs for evaluations has expanded
beyond the idea that a survey can include both open- and closed-ended items or that of having a survey and
a focus group. The mixed methods designs in evaluation in this text illustrate the possibility for sophisticated
evaluation designs capable of informing decision making throughout the life span of an initiative, whether that
is a project, a program, a policy, or an organization. The mixed methods designs for program evaluation in
many of the chapters are of a cyclical nature, meaning that early evaluation activities inform next steps in the
program and in the evaluation. Such designs might call for a change of the culture of thinking about evaluation

Page 4 of 10 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

as something that happens at the end of a program, instead seeing it as a way to support ongoing learning
and innovation.

Advancements in design research (Philip & De Bruyn, 2013) and developmental evaluation (Patton, 2011)
provide a glimpse into how mixed methods designs can be developed that are responsive to changing con-
ditions in a context or organization. Mixed methods designs have the potential to be responsive in a dynamic
way that is reflective of the complexity found in most evaluation contexts. When design research principles
and complexity theory are brought into mixed methods designs, it opens up possibilities for informing the un-
derstanding of the context as it changes and applying those understandings to needed adjustments in terms
of interventions, implementation strategies, and revisions of these same elements. Challenges arise related
to how to integrate improved understandings about the evaluation process and findings from earlier stages
of an evaluation into later stages and how to use this process to influence decision and policy making. How
will the value of using more sophisticated designs be established with funders due to these requiring more
time and resources? What arguments will be persuasive to funders that mixed methods designs have greater
potential to address complex problems and deliver solutions that have greater potential to be effective? What
will be gained and/or lost by using more creative cyclical mixed methods designs for evaluators and their
stakeholders?

Technological Advances and Big Data

If there is one thing we can say for certain, it is that the use of technology in the field of evaluation is ubiquitous
and dynamic. The implications of technology in the design of mixed methods studies is mind-boggling. Tech-
nology comes in so many forms and can be applied in so many ways. Mixed methods designs in evaluation
can include data collection strategies that are technologically based, such as video capture, audio capture,
web-based data collection, blogs, tweets, and so forth. How can evaluators integrate technological advance-
ments into mixed methods designs in ways that are productive and ethical? This question comes to the fore
especially when consideration is given to integrating the use of data collected through and made available
through technology that was not necessarily collected for the specific evaluation.

Such data can take the form of “big data” that is receiving so much attention (Hesse, Moser, & Riley, 2015).
Big data is a term used to describe data that are extremely large in volume, come in a variety of forms (struc-
tured, unstructured, narrative, numeric), and come from multiple sources (e.g., health surveys, social media,

Page 5 of 10 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

real-time wearable devices such as Fitbits, electronic record keeping, and other government and nongovern-
ment databases). Big data might allow for access to data on an entire population, rather than a selected
sample. How will mixed methods designs be changed if there is access to this level of data? What questions
need to be explored related to the quality of the data, the merger of data across data sets, and ethical and
confidentiality issues?

Data that is not big data can also be collected via technology, sometimes incidentally, sometimes purposefully.
With mobile phones equipped with cameras, data can be captured surreptitiously (or serendipitously). What
are the ethical implications of including such data in evaluations? How can evaluators maximize the use of
technology in appropriate and ethical ways to enhance mixed methods designs by administering surveys on-
line or in real time, using mapping applications, or using social networking applications? How can technology
be used to increase the use of visual and spatial data collection and presentation of findings? How can it
be used to engage with a wider range of stakeholders? What are the implications of these issues for mixed
methods design?

Preparation of Mixed Methods Evaluators

When I was in graduate school, my professors taught me quantitative methods of evaluation design because
that was their expertise. As I moved into the world of evaluation, I realized that I was only getting part of the
story with quantitative methods, and so I engaged in extensive professional development activities to add
qualitative methods to my repertoire. Like many evaluators, I intuitively came to the conclusion that evalua-
tions in complex social contexts benefited by using both quantitative and qualitative approaches. As I tried to
figure out how to bring the different perspectives about methods together, I was fortunate to be in the right
places at the right times to witness and participate in the birth and early growth of the mixed methods com-
munity. I tell you this story because my experience is not unlike many of those who are preparing evaluators
in universities and other professional development settings. Evaluation is a multidisciplinary profession, and
hence evaluators come with a legacy of the methodologies dominant in their disciplines.

According to Scriven (2008), a transdiscipline is a discipline that can stand alone but can also be used as a
tool in several other disciplines. Evaluators might gain insights into how to integrate methods by considering
evaluation as a transdiscipline. Scriven (2008) used this term to describe evaluation because it is not only

Page 6 of 10 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

a discipline unto itself, it is also used in so many other disciplines. Evaluation shares this transdisciplinary
nature with other disciplines such as ethics and statistics; each has a unique way of approaching issues but
is also used in other areas of inquiry such as education, health, criminal justice, social work, and economic
development.

Coryn and Hattie (2006) suggest that consideration of evaluation as a transdiscipline can serve as a model
for the preparation of evaluators. Jacob (2008) strengthens the argument for transdiscipline training for eval-
uators because of the increased acknowledgment of complexity in the evaluation of social phenomena and
public programs and policies. Training alone may not be sufficient to address this challenge; it may be nec-
essary to rethink the composition of evaluation teams to be more transdisciplinary. A parallel strand of ac-
tion about preparing evaluators arises from the attention being given to defining competencies for evaluators
and the process of credentialing evaluators (King & Stevahn, 2015). Whereas it is beyond the scope of this
text to examine those competencies and processes, considerable activity has been and is occurring on this
front, including the specification of Essential Skills Series by the Canadian Evaluation Society (2017) in 1999,
followed by a voluntary credentialing process in 2003, as well as the creation of competency statements in
South Africa, Aotearoa New Zealand, Japan, and Thailand. The American Evaluation Association is also tak-
ing action on this front. How will the awareness of the importance of mixed methods in evaluation influence
discussions about competencies? This is a question that I do not have an answer for, but I suggest it is an
important question for the evaluation community to consider.

This brief contextual analysis of the preparation of evaluators serves to highlight the challenges present in
the evaluation world. What are the best ways to prepare faculty who can teach about mixed methods, new
evaluators, and practicing professionals? What are the strategies that support a transdisciplinary approach
to mixed methods design in evaluation? What can be done to facilitate respectful and productive conversa-
tions across disciplines to enhance mixed methods designs in evaluation? How can course sequences be
developed that lead to an understanding of how to include the methodological rigor developed within spe-
cific branches of evaluation with an integrated approach to yield mixed methods designs? What should be
included in undergraduate, graduate, and professional training in methods to support development of mixed
methods skills?

Page 7 of 10 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

Responsiveness to Complex Social Problems

Many of the evaluation examples in this text serve to illustrate the application of complex mixed methods de-
signs to address wicked problems such as poverty, violence, and climate change. Mixed methods designs
in evaluation provide a sophisticated and creative means for the evaluation community to contribute to the
solutions to what have heretofore been considered intransigent conditions that impact people’s lives and the
health of the planet. This characterization of the role of the evaluator in these terms is not without controversy.
Evaluators who situate themselves on one or another of the evaluation branches (or currents) might reject or
endorse this role depending on their philosophical assumptions. Should the evaluator be engaged in strate-
gies designed to support social and environmental transformation? What are the methodological implications
of explicitly addressing issues of human rights, power inequities, and oppression? What are the ethical issues
of adopting (or not adopting) such a stance, and how will this be viewed by ethics review boards?

The use of mixed methods designs can lead to asking different kinds of questions reflective of the systemic
structures that allow for an oppressive status quo to continue or to stimulate the kind of systemic change
needed to transform society. According to The Program Evaluation Standards: A Guide for Evaluators and
Evaluation Users, evaluators have a professional responsibility to ensure the rigor of their work and the use-
fulness of their processes and results (Yarbrough, Shulha, Hopson, & Caruthers, 2011). The American Eval-
uation Association’s (AEA; 2004) Guiding Principles for Evaluators reinforces the importance of the need for
rigorous evaluations conducted in a culturally competent manner with inclusion of stakeholders on the basis
of race/ethnicity, sex/gender, economics, and other relevant factors. Mixed methods designs allow evaluators
the flexibility they need to be inclusive in culturally responsive ways by modifying approaches to support the
appropriate needs of diverse stakeholder groups. AEA’s guidelines also instruct evaluators to work for the
general and public welfare, meaning they are to go beyond the interests of any particular stakeholder to con-
sider the welfare of society as a whole. Mixed methods designs provide the needed sophistication to include
not only diverse stakeholders but also phases of data collection about context, history, legacy, legislations,
and other conditions that can influence the effectiveness of interventions.

Page 8 of 10 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

Integration and Dissemination in Mixed Methods Evalua-

tion

Mixed methods designs also bring the possibility of creative avenues for sharing and disseminating findings
from evaluations by building in dissemination as part of the design process. Evaluators can find advice about
how to publish in mixed methods journals (Archibald et al., 2015; Fetters & Freshwater, 2015; Mertens, 2011;
O’Cathain, Murphy, & Nicholl, 2007). It is important to share our methodological advances. Evaluators can
provide leadership by publishing their work in evaluation journals and in mixed methods journals, as well as
in methodological journals and discipline-specific journals that might have traditionally only published quanti-
tative or qualitative studies.

The power of mixed methods lies in the added value of having both quantitative and qualitative data and the
insights that can be gained when these data are integrated in a single study or a series of studies. When multi-
phased, cyclical mixed methods designs are used; these present opportunities to integrate the data at several
points during the evaluation. Strategies for integration and dissemination can be changed to be responsive to
the phase of the evaluation as well as to the stakeholder’s needs. As seen in the examples in this text, mixed
methods designs have included multiple strategies for integration and dissemination of findings such as use
of focus groups, web-based presentations, visual spatial depictions, and videos. Use of such methods built
into the design of the evaluation allows for greater engagement with a wider range of stakeholders and has
the potential to increase the usefulness of the findings at each stage of the evaluations.

Conclusions and Moving Forward

Evaluators interested in exploring designs that are innovative and responsive to the complexity in their eval-
uation contexts are standing on fertile ground. A journey into the critically conscious development of mixed
methods designs has begun, and it offers a myriad of opportunities to increase the impact that evaluation
can have on important societal and environmental problems and solutions. The complexity involved in under-
standing contexts and cultures is enhanced by the use of mixed methods designs that focus on collection of
data about history and diversity. Much remains to be done; the members of the evaluation community who
are willing to take risks and share their experiences can lead to changes of great importance for the current

Page 9 of 10 Mixed Methods Design in Evaluation

Sage Sage Research Methods
© 2018 by SAGE Publications, Inc.

population as well as for generations to come.

[Link]

Page 10 of 10 Mixed Methods Design in Evaluation