QUALITATIVE ME THODS IN

IMPLEMENTATION SCIENCE

Q UA L I TAT I V E M E T H O DU.S.
S I NDEPARTMENT OFI OHEALTH
I M PL EM EN TAT N S C I ENAND
C E HUMAN SERVICES | NATIONAL INSTITUTES OF HEALTH 1
 PREFACE

In recent years, the field of implementation science has significantly grown, both in the theories and frameworks
used to explain it and in the available findings used to understand and influence the adoption, implementation
and sustainment of evidence-based practices (and the de-implementation of ineffective or harmful practices).
Capacity has greatly increased for our investigators to conduct studies that reflect the complexity and dynamism of
improving care within clinical, community and public health settings. Improvements have accrued in the range of
research designs used, the methods to collect data, and analytical strategies used to make sense of implementation
processes and outcomes. From its beginnings, implementation science has recognized the importance of both
qualitative and quantitative data and throughout its development, leading scholars have called for mixed methods
approaches to ensure maximal return from each investigation.

Despite clear recognition of the value of qualitative research methods within implementation studies, there remains
confusion as to which methods to use, how to best use them, and what guidance investigators should follow in
their application. The Qualitative Research in Implementation Science (QualRIS) group was formed to reduce this
confusion. Our investigators have moved from viewing qualitative research methods as an add-on to the major
components of a study (i.e. addition of focus groups to address exploratory work on provider factors affecting
implementation outcomes) to a necessary but often underspecified core of a study. In this paper, the QualRIS
group explains multiple ways in which qualitative methods can be effectively used to answer a range of high priority
implementation science questions, and describes resources that are available to support the community. While
the field will continue to progress in its methodologies, this report offers a snapshot of how qualitative methods in
implementation science have advanced and where they can go from here. We hope this serves as a helpful resource
to investigators and that the report’s recommendations are taken on so that the next generation of implementation
science studies include the best qualitative research approaches to fit each question.

We are grateful to the QualRIS team for their substantial effort to review progress and offer guidance to the field,
and to all those who seek to integrate qualitative research methods to strengthen their implementation science
investigations.

Sincerely,

David A Chambers
Deputy Director for Implementation Science
Division of Cancer Control and Population Sciences
National Cancer Institute

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E
 TABLE OF CONTENT S

I. Introduction 01

II. Approach 02

III. Traditional Qualitative Research Methods: An Overview 03

IV. Qualitative Research Methods in Implementation Science

in Health and Health Care: An Overview 04

V. Value of Qualitative Methods in Implementation Science 05

A. Elicit Stakeholder-Centered Perspectives

B. Inform Design and Implementation
C. Understand Contexts Across Diverse Settings
D. Provide Documentation and Encourage Reflection
on Implementation Processes
E. Gain Insight Into Implementation Effectiveness
F. Understand Mechanisms of Change
G. Contribute to Theoretical Development

VI. Methodological Considerations for the Use of

Qualitative Methods in Implementation Science 09

A. Sampling
B. Timing of Data Collection
C. Roles of Qualitative Research Teams in Implementation Science
D. Data Collection
E. Data Analysis
F. Presentation of Qualitative Findings

VII. Collecting and Analyzing Qualitative Data with Rigor 14

VIII. Need and Opportunities for Innovation in

Use of Qualitative Methods in Implementation Science 16

IX. Conclusion 18

X. Acknowledgements 19

References 20

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E
 I . INTRODUC TION

Implementation science is a rapidly developing field dedicated to putting evidence-based practices to use in
real-world settings (Brownson et al., 2018; Glasgow et al., 2003; Kessler et al., 2012). Implementation science has
been defined as, the scientific study of methods to promote the integration of research findings and evidence-based
interventions into healthcare practice and policy. (See NIH PAR-16-237, PAR-18-007, and PAR-18-017 at
[Link] The field extends beyond healthcare into health behavior and
promotion, including public and population health (Leeman et al., 2017), as well as education, policy, and other fields.
The evidence-based interventions that are the focus of implementation science in healthcare include programs,
practices, guidelines, and public and organizational policies that are intended to improve the health of individuals and
populations (Rabin et al., 2008). To meet its overarching aim, implementation science seeks to identify factors and
strategies that influence or predict adoption, adaptation, implementation (or de-implementation), sustained use, and
scale-up and spread of evidence-based interventions.

Implementation science relies on qualitative and quantitative methods—or mixed methods—to understand the
strategies and processes needed to implement an evidence-based change, the outcomes the change produces, the
barriers and facilitators to implementing the change, and how all of these may vary across different types of contexts.
Qualitative research methods are widely used in implementation science (Palinkas, 2014; Weiner et al., 2011). The
power of mixed methods in yielding broader responses to research questions has been demonstrated in numerous
areas of inquiry, and its advantages and challenges have been widely discussed (Zhang & Watanabe-Galloway, 2014;
Hadi & Closs, 2016; Bishop, 2015). The use of mixed methods in implementation science has gained much recent
attention (Tonkin-Crine et al., 2016; Palinkas et al., 2015; Palinkas et al., 2011; Beidas et al., 2014).

The uses of qualitative methods in mixed methods research vary according to the study design and can be quite
complex. The discussion of design in mixed methods studies is beyond the scope of this paper. We will not discuss
mixed methods specifically, but, rather, refer the reader to some notable resources on the subject (Creswell, 2015;
Creswell et al., 2011; Creswell & Plano-Clark, 2011).

While there is a great deal written on standards for qualitative research in general, there are aspects of doing
qualitative research in the context of studying implementation that may make the norms and guidance for best
practices in this use quite different. For example, demand for timely or rapid turnaround of results, need for multiple
stakeholder perspectives, diversity of contexts across implementation sites, and contextual changes over time all pose
unique challenges to the agility and rigor of qualitative methods. This paper is an initial step to provide guidance to the
field about the use of qualitative methods in implementation science. Our target audiences include implementation
researchers with limited to no experience with qualitative research methods, as well as experienced qualitative
researchers who are learning about implementation science and potentially partnering with implementation scientists.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 1
 II . APPROACH

In 2015, the National Cancer Institute’s (NCI’s) Implementation Science Team convened a group of experts in
implementation science and/or qualitative research, to develop guidance for the use of qualitative methods in
implementation science. The Qualitative Research in Implementation Science (QUALRIS) group conducted its work
via teleconference, email, and as an NCI online learning community ([Link] The group
consulted literature on best practices in qualitative methods and drew upon members’ own extensive experience
in adapting these methods for implementation science. Following this iterative process, the group determined focal
areas to examine, presented preliminary work to scientific audiences (Hamilton et al., 2015), and then drafted this white
paper. After several revisions, the draft white paper was reviewed by the group’s advisory panel, consisting of five
leaders in implementation science and qualitative methods: Ross Brownson, Cam Escoffery, Russell E. Glasgow, Anne
Sales, and Bryan J. Weiner.

After briefly describing traditional qualitative research methods, we (1) describe the role of qualitative research in
health-related implementation science; (2) offer guidelines for conducting qualitative research in implementation
science; (3) identify areas of need and opportunities for the development of innovative qualitative methods; and (4)
provide resources for those wishing to employ or learn more about qualitative methods in implementation science.

Below is a list of QUALRIS group members and their affiliations.

Deborah Cohen, Ph.D. Jennifer Leeman, DrPH, MDIV

Department of Family Medicine School of Nursing
Oregon Health and Science University University of North Carolina, Chapel Hill

Benjamin F. Crabtree, Ph.D. Deborah K. Padgett, Ph.D., M.P.H.

Department of Family Medicine and Community Health Silver School of Social Work
Rutgers Robert Wood Johnson Medical School New York University

Laura Damschroder, M.S., M.P.H. Lawrence Palinkas, Ph.D.

VA Ann Arbor Center for Clinical Management Research School of Social Work
PeRsonalizing Options through Veteran Engagement University of Southern California
(PrOVE) QUERI, Ann Arbor, Michigan
Borsika Rabin, Ph.D., M.P.H., PharmD
Alison B. Hamilton, Ph.D., M.P.H. Center of Excellence for Stress and Mental Health, VA
Center for the Study of Healthcare Innovation, San Diego and Department of Family Medicine and
Implementation and Policy Public Health, School of Medicine
VA Greater Los Angeles Healthcare System and University of California San Diego and Triple Aim QUERI,
Department of Psychiatry and Biobehavioral Sciences Eastern Colorado Healthcare System, Denver, Colorado
University of California, Los Angeles
Heather Schacht Reisinger, Ph.D.
Suzanne Heurtin-Roberts, Ph.D., M.A., M.S.W. Iowa City VA Health Care System and
Implementation Science Team, Division of Cancer Carver College of Medicine, University of Iowa
Control and Population Sciences
National Institutes of Health/National Cancer Institute

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 2
 III . TR ADITIONAL QUALITATIVE RES E ARCH
ME THODS : AN OVERVIEW

Qualitative methods generally emerged from the social sciences, specifically anthropology and sociology, and tend
to focus on approaches for studying human behavior and experience. These attributes are particularly salient for
implementation science because of its focus on understanding how implementation processes influence and are
influenced by dynamic contextual factors, which requires study of human behaviors and experiences, individually
and within collectives (e.g., a healthcare organization or setting). Qualitative methods do not depend upon numerical
manipulation of data, although basic frequencies and summations may be used to present data, provide contextual
information, and transpose data for use in mixed methods analysis.

The diverse forms of qualitative methods do, for the most part, share common elements and perspectives (Crabtree &
Miller, 1999; Padgett, 2008; Pope & Mays, 2006; Southam-Gerow & Dorsey, 2014). Traditionally, such methods seek
in-depth understanding, and may be used to generate hypotheses. Qualitative methods are typically inductive and
naturalistic; in other words, data are drawn from people, often in their natural settings. Qualitative methods, to varying
degrees, seek interpretation of social phenomena, rather than their objective representation. Qualitative methods intend
to give a holistic, full representation of complex, dynamic social circumstances reality (Padgett, 2016; Pope, Mays, &
Popay, 2006; Damschroder et al., 2009; Kitson et al., 2017). Traditional qualitative research approaches have typically
involved deep engagement of researchers in naturalistic settings and with participants over significant periods of time.
The immersive and iterative nature of qualitative research allows for the comprehension of the dynamism and complexity
of aspects of everyday life.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 3
 IV. QUALITATIVE RES E ARCH ME THODS IN
IMPLEMENTATION SCIENCE IN HE ALTH
AND HE ALTH C ARE: AN OVERVIEW

Implementation science is action- and change-oriented, with a focus on understanding what is changing in relation
to the intervention and implementation strategies; how, when, where, and why changes happen (or not); and who is
involved in the changes. Studies in implementation science usually occur over a relatively short period of time, with
a clear before, during, and after. (See Brear et al., 2018; Ivankova, 2015; and Meyer, 2000, for discussions of action
research and qualitative methods.) With this backdrop in mind, our team has identified nine key features of the use of
qualitative methods within implementation science:

1. In contrast to a more constructivist, exploratory, inductive paradigm, qualitative methods in implementation

science tend to be more positivist and deductive in nature, with their use increasingly guided by theories and
organized by one or more implementation models or frameworks.

2. Qualitative methods in implementation science tend to be practical and targeted, geared more toward
explaining specific implementation-related phenomena.

3. Implementation science involving qualitative methods is inherently multidisciplinary rather than

discipline-specific.

4. Implementation science typically integrates qualitative and quantitative methods to strengthen findings, often
in the context of complex mixed methods designs rather than using qualitative methods alone.

5. Implementation science using qualitative methods typically involve qualitative methods teams rather than
solo researchers.

6. Qualitative data collection in implementation science is often time-limited, with data collection occurring at
single or punctuated time points (e.g., pre-, mid-, and post-implementation), often in rapid fashion rather than
continuous data collection over an extended period, a hallmark of traditional qualitative methods.

7. Engagement in implementation science tends to be targeted toward multiple stakeholders (e.g., providers,
administrators) in multiple, diverse settings (e.g., several health clinics) rather than traditional long-term, in-
depth engagement within a community.

8. Qualitative data analysis in implementation science tends to be targeted to specific a priori research questions,
is usually more rapid, and is often integrated with quantitative analyses. This contrasts with
more traditional qualitative analysis, which tends to be phenomenologically oriented with an emphasis
on reflexivity.

9. In addition to contributing to traditional outlets such as the scientific literature, implementation science seeks to
disseminate both qualitative and quantitative findings through “grey” literature, including interim reports, policy
and operations briefs, workshops and trainings, online resources, site profiles, and implementation metrics
that may be more readily accessed by practitioners.

Implementation science is a young, evolving, and complex field of research. Its researchers are confronted
with several challenges, including facing and embracing the complexity and dynamic nature of interventions,
implementation processes, and intersecting implementation contexts and their interactive effects on outcomes
(Abbott et al., 2014; Azar et al., 2015; Brives et al., 2016; Brownson et al., 2012; Damschroder et al., 2009;
Glasgow, 2008; Glasgow et al., 2012).

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 4
 V. VALUE OF QUALITATIVE ME THODS IN
IMPLEMENTATION SCIENCE

Here we review several ways in which qualitative methods are valuable in implementation research
conducted to date.

A. Elicit Stakeholder-Centered Perspectives

Qualitative methods are used to elicit the perspectives of those who have a “stake” in implementation and who are
responsible for “healthcare-related decisions that can be informed by research evidence,” i.e., stakeholders, such as
administrators, providers, staff, patients, clients, families, and community members (Martinez et al., 2016; Concannon
et al., 2012). Understanding the perspectives of those involved in and/or affected by implementation of an intervention
is essential to ensuring that interventions address problems recognized as high priority and use approaches that are
feasible and acceptable within real-world practice contexts (Chalmers & Glasziou, 2009; Glasgow et al., 2012; Tunis et
al., 2003). For example, Leeman and colleagues (2014) conducted in-depth interviews with public health practitioners
working in obesity prevention to understand factors that influenced the adoption and implementation of evidence-based
policy, systems, environmental change interventions at the state and local levels. Kane and colleagues (2014) also
conducted interviews with providers to assess perspectives on feasibility and acceptability of new guidelines. Qualitative
researchers often bring a participatory perspective, especially when focused on implementing practices to serve
marginalized populations, relying on partnerships with a wide range of stakeholders to achieve study goals.

B. Inform Design and Implementation

Qualitative studies have been used to inform the design and implementation of interventions. For example, Proctor and
colleagues (2007) conducted a qualitative pilot study to document the perspectives of agency directors on the challenge
of implementing evidence-based practices in community mental health agencies prior to the development and testing of
a specific implementation intervention. Using a participatory research approach (see Reason & Bradbury, 2008), Elsey
and colleagues (2016) conducted qualitative semi-structured interviews and focus groups in Nepal to understand patient
and health worker knowledge of tobacco and patients’ motivation to quit. Findings were used to inform the design and
implementation of a smoking cessation intervention. A Norwegian research team used a series of individual and group
interviews with various stakeholders to inform the implementation of tailored interventions to improve the treatment of
elderly patients with depression (Aakhus et al., 2015). The stakeholders, including health professionals, researchers, and
relatives of elderly patients with depression, were asked to brainstorm interventions to improve adherence to treatment
guidelines. These qualitative findings were used to test targeted versus non-targeted interventions to improve adherence
in an implementation trial. Semi-structured interviews with mental health practitioners were used to examine the barriers
to using a smartphone application to enhance patient compliance to evidence-based psychotherapy for post-traumatic
stress disorder (PTSD) (Reger et al., 2017). Based on these interviews, a targeted intervention was developed to
increase use of the smartphone application among PTSD therapists.

C. Understand Contexts Across Diverse Settings

One of the most frequently cited reasons for using qualitative methods in implementation science is to understand
contextual factors that influence implementation, such as variations in uptake (Bekelman et al., 2016; Cadogan et
al., 2016; Cohen et al., 2016; Elsey et al., 2016; Lord et al., 2016; Marshall et al., 2008; Marty et al., 2008; Rapp et
al., 2010; Sommerbakk et al., 2016; Varsi et al., 2015; Damschroder et al., 2017). Since contexts can differ markedly
across implementation sites, understanding contexts and their relation to implementation is of critical importance.
See Section VI/B below. Qualitative methods are well suited to explore how context, actions, and people contribute
to successful (or failed) implementation. Researchers have examined factors that influence implementation of the

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 5
 life of study or initiative, from pre-implementation (Koenig et al., 2016), to implementation (Moeckli et al., 2013), to
sustainability (Colon-Emeric et al., 2016). For example, researchers (Swain et al., 2009) studied implementation of an
illness management and recovery program for people with severe mental illness in community mental health settings.
Qualitative methods were used to assess perceived barriers and facilitators of implementation such as leadership,
organizational culture, training, and staff and supervision. The analysis showed that several cross-site factors worked
synergistically to influence implementation (Whitley et al., 2009). Freedman and colleagues (2012) supplemented survey
data with in-depth interviews to identify factors critical to a health center’s readiness to implement a farmers’ market.
They found several factors to guide researchers and health center staff in development of a farmer’s market to improve
access to healthy foods. Damschroder and colleagues (2009) have used qualitative assessments, guided by the
Consolidated Framework for Implementation Research (CFIR), to understand variations in implementation success and
provide recommendations for scaling up implementation of programs (Damschroder & Lowery, 2013; Damschroder,
Reardon, AuYoung et al., 2017; Damschroder, Reardon, Sperber et al., 2017). Internationally, researchers have used
qualitative methods to encapsulate findings in conceptual models of system transformations (Bate et al., 2008; Lukas et
al. 2007), and have highlighted context as an active agent influencing the implementation and not just a backdrop to it
(Dopson & Fitzgerald, 2006).

D. Provide Documentation and Encourage Reflection on Implementation Processes

Another value of qualitative methods is to document and reflect on implementation processes. For example, Aarons and
Palinkas (Aarons & Palinkas, 2007; Palinkas & Aarons, 2009) collected qualitative data through annual interviews and
focus groups to assess the process of implementation of Safe Care®, an intervention designed to reduce child neglect
and out-of-home placements of neglected children into foster care. Hoagwood and colleagues (2007) used a case
study of an individual child to describe the process of implementation of an evidence-based, trauma-focused, cognitive-
behavioral therapy for treatment of symptoms of PTSD in children living in New York City in the aftermath of the World
Trade Center attack on September 11, 2001.

Qualitative methods such as ethnography are well-suited to capture implementation micro-processes, that is, at
the level of individual interactions (Nilsson et al., 2018). Furthermore, fidelity to intervention use and implementation
strategies and their adaptations can be documented and better understood by qualitative inquiry, as can adoption of
improved practices (Bunger et al., 2017; Cohen et al., 2008). Such inquiry can inform development of and contribution
to the adaptome, an aggregate body of knowledge documenting “adaptations to interventions and their impacts
on implementation, service, and health outcomes” as proposed by Chambers and Norton (2016). This also raises
considerations of flexibility and fidelity during adaptation (Cohen & Crabtree, 2008).

E. Gain Insight Into Implementation Effectiveness

Qualitative methods are used to study the effectiveness of implementation. Proctor and colleagues (2011) created a
taxonomy of implementation outcomes to consider when assessing implementation effectiveness. Many implementation
outcomes are well-suited for evaluation via qualitative methods. Furthermore, qualitative methods are often used
when valid measures are not available. (See Rabin et al., 2016, for measurement resources in implementation). The
effectiveness of implementation strategies, (i.e., approaches employed to integrate an intervention into a new practice
setting; Powell et al., 2012; Powell et al., 2017; Powell et al., 2015), also requires evaluation, often best accomplished with
qualitative methods. Wozniak and colleagues (2015) evaluated the implementation of an efficacious collaborative care
model or patients with diabetes and depression in Canada. Researchers used qualitative interviews among staff and
systematic documentation of the implementation to assess effectiveness. The qualitative data analysis identified factors
such as lack of training in team operations, physician-nurse relations, and local physician culture as barriers to optimal
collaborative care that the model stipulated.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 6
 Rapp and co-authors (2008) used a mix of qualitative data collection techniques to identify and evaluate strategies that
contributed to successful implementation in six community mental health centers (CMHCs). Data collection techniques
included text analysis (e.g., notes, memos, reports), formal and informal interviews, and direct observation. Data
was coded according to 26 a priori themes and included in comprehensive site reports for each of the six CMHCs.
Through an iterative analytic process, investigators and key stakeholders blinded to the original a priori categories,
created a provisional coding guide. This was used to develop a final coding scheme applied to all six implementation
sites. Eight strategies were found that contributed to successful implementation in the six CMHCs.

F. Understand Mechanisms of Change

Qualitative methods can be employed to understand mechanisms of change and the relationship(s) between the
theorized and actual mechanisms of change (Kazdin, 2007) by identifying and explaining which aspects of the
intervention or its implementation result in change. Now, “mechanism of change” can be defined variously and one
study’s mechanism may be another study’s moderator or intermediate outcome. A discussion of this variation is
beyond our scope, but in this section, we will describe some uses of qualitative methods to understand change. (For
further perspectives on change, see Lewis et al., 2018; Dalkin et al., 2015; May, 2013; Rycroft-Malone et al., 2012).

Following a continuous quality improvement (CQI) effort for care of Indigenous Australians, Schierhout and colleagues
(2013) explored the patterns of change in service delivery in 36 widely-varying health centers that had participated
in the CQI project. Using a mixed-methods “realist” approach (see Rycroft-Malone, 2012; Dalkin et al., 2015),
implementation researchers conducted key informant interviews to elicit interpretations about cross-site variation’s
role in producing service-delivery variation. Inductive analysis led to fine grained hypotheses of change and further
data were collected though clinical audits, project monitoring reports and workshops with key informants involved in
implementation. The team abstracted potential patterns of context, mechanism, and outcomes to develop a working
theory of change for program strengthening.

Qualitative methods can help us to understand how interventions (change) can be sustained (Cohen et al., 2016;
Palinkas et al., 2016; Colon-Emeric et al., 2016). For example, in a study examining how to sustain complex
interventions in long-term care, 15 focus groups were conducted with 83 staff who had participated in the intervention
trial (Colon-Emeric et al., 2016). Participants gave perceptions of intervention salience and sustainability. Analysis
produced insights about intervention features, organizational context, and external support offering concrete
suggestions on how to maintain changes brought about by interventions.

Damschroder, Reardon, AuYoung, and colleagues (2017) mapped qualitatively-identified barriers and facilitators,
to specific types of outcomes defined by RE-AIM to identify factors that may moderate the intervention’s effects
on outcomes. This work highlights the key role qualitative methods can play in explicating multiple potential causal
pathways for further testing (Forman et al., 2017).

G. Contribute to Theoretical Development

There is ample description, cataloguing, and discussion of conceptualization, that is, theory, frameworks, and
models in implementation science (Nilsen, 2015; Chambers, 2014; Tabak et al., 2012; Davies et al., 2010; May et al.,
2009; Eccles et al., 2005) but less agreement on terminology. According to Tabak and colleagues (2018), theory and
frameworks can both be considered to belong to the category of models. Davidoff and colleagues (2015) recognize
multiple levels of theory, grand, mid-level, and lower. They propose that lower or “program” level theories are perhaps
the most useful in improvement work. Similarly, Chambers (2014) points out that rather than “classical” theory, these
more immediate-level models or frameworks are useful in formulating specific research questions and explaining
dissemination and implementation processes across diverse stakeholders and settings.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 7
 Researchers have explicitly challenged the implementation science community to contribute to advancing existing
theoretical frameworks to stimulate new theoretical developments (Damschroder et al., 2009; Kitson et al., 2008).
Theory provides an efficient way to build the knowledge base through shared language and concepts that are
foundational for guiding data collection, measurement, analyses, and interpretation, and creates a natural bridge of
comparison to other studies using the same theories (Colquhoun et al., 2014; Foy et al., 2011). Qualitative approaches
can advance conceptual development in multiple ways.

Qualitative methods can inform the development of conceptual models or frameworks through grounded methods
(Corbin & Strauss, 2008; Glaser & Strauss, 1967). For example, Zazzali and colleagues (2008) developed a conceptual
model of implementation of Functional Family Therapy using qualitative data elicited through semi-structured interviews
with 15 program administrators. Their model could then be tested using quantitative methods. Blasinsky et al. (2006)
developed a rating scale to construct predictors of program outcomes and sustainability of a collaborative care
intervention to assist older adults suffering from major depression or dysthymia based on qualitative data obtained
from semi-structured interviews. Woltmann et al. (2008) created categories of turnover and designations of positive,
negative, and mixed influence of turnover on outcomes using qualitative data obtained through interviews with staff,
clinic directors, and consultant trainers in their study of staff turnover during implementation of evidence-based
practices in mental healthcare.

Testable theories of implementation within organizations have been evaluated using qualitative methods. For
example, Klein and colleagues’ model of implementation (Klein & Sorra, 1996; Klein et al., 2001) was first tested in a
manufacturing setting. This model was then adapted for clinical settings and assessed for applicability and usefulness
in two different studies using qualitative data (Helfrich et al., 2007; Damschroder et al., 2011) collected through semi-
structured interviews of key stakeholders. Both studies recommended refinements to the model based on their
respective qualitative findings. The adapted model can be further tested in more settings and verified using a full range
of methods including qualitative, quantitative, and/or mixed methods.

Highly cited implementation frameworks have been developed based on foundational, empirical qualitative data
including the Promoting Action on Research Implementation in Health Services (PARIHS) (Kitson, 2008) and
Normalization Process Theory (NPT) (May, 2006; May et al., 2009). PARIHS developers further developed their
framework into i-PARIHS, based on a growing knowledge base (Harvey & Kitson, 2015), and May (2013) further
extended NPT to create a generalized, mid-range theory of implementation based, in part, on accumulating qualitative
findings from several researchers using NPT. Because of the complex and dynamic interplay between implementation
processes, places, and people, qualitative methods are often used in conjunction with frameworks to guide evaluation
or execution of implementation. For example, a systematic review of published studies using the CFIR, revealed that 23
of 26 eligible studies used qualitative methods (Kirk et al. 2016).

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 8
 VI . ME THODOLO GIC AL CON SIDER ATION S
FOR THE US E OF QUALITATIVE ME THODS
IN IMPLEMENTATION SCIENCE

When using qualitative methods in implementation science, several considerations should be kept
in mind, including sampling procedures, the timing of data collection, the role of the qualitative
researcher, procedures for collection and analysis of qualitative data, and presentation of findings.
Each of these considerations is examined briefly below.

A. Sampling
Sampling of participants in qualitative studies is generally purposive in nature (Palinkas et al., 2015). Random sampling
is occasionally used if participants are involved in quantitative studies as well or if there is a need to generalize study
results in a manner that cannot be accomplished with maximum variation or other forms of purposive sampling. In
implementation science, participants are often sampled based on their level of participation in the implementation
process and role in the organization to maximize diversity of perspectives (Colon-Emeric et al., 2016). Bekelman and
colleagues (2016) conducted stratified purposeful sampling to capture the range of perspectives from providers of
different disciplines who provide or have patients enrolled in outpatient palliative care. They also recruited leaders who
were likely to decide whether and how to adopt and sustain outpatient palliative care in their settings.

Implementation scientists may also select the entire population of those implementing or receiving the intervention,
particularly when the population is small, such as in a feasibility pilot study of an implementation strategy. Sample size
may also be limited by geographic setting, such as population defined by a census tract. To be sure, size is not the
most important sampling issue, depth or data saturation is the goal. The state of saturation is reached by sampling to
the point of redundancy or repetition, when data collection in a particular sample yields no new information (Crabtree &
Miller, 1999; Francis et al., 2010).

Samples may be selected from any population of interest, not only individual persons but of care delivery sites,
political or economic institutions, geographic entities, and so on. Focusing on case study analyses, Sax et al. (2013)
purposefully sampled six hospitals from a larger study of 15 hospitals involved in an infection control implementation
project across Europe. The investigators used criteria sampling to choose three “high potential for change” hospitals
and three “low potential for change” hospitals to examine the process of uptake of the intervention designed to reduce
hospital infections in intensive care units. This sampling strategy was based upon preliminary qualitative results.
Researchers then used key informant interviews, ethnographic observation, written protocol collection, guideline
collection, and photography to collect data relevant to infection control. Preliminary analysis of data found 15 themes
relevant to successful implementation of infection control practices (Casillas et al., 2013) across the hospitals and factors
influencing leadership dynamics in infection control (Clack et al., 2013).

B. Timing of Data Collection

Due to implementation science’s orientation toward change, qualitative researchers must pay particular attention to
the dynamic nature of the contexts in which they are working (Chambers et al., 2013). See Section V/C above. Time is
a central factor to consider when conducting qualitative research in implementation science. Early in implementation
studies, qualitative researchers are often called upon to assess the organizational or community context in which
the intervention will be implemented. This may require semi-structured interviews with the key stakeholders who will
be involved or impacted by the intervention to examine the “baseline” context of a setting, asking questions such
as, what is the stakeholders’ current practice in relation to the target outcome? (Stewart et al., 2010; Nutting et al.,

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 9
 2010). Observations of the community or clinical setting are frequently conducted to assess the interactions of the
stakeholders and whether their behavioral practices are consistent with the perceptions they articulated in interviews.
Observations can be especially helpful in documenting current processes (Nilsson et al., 2018).

The baseline assessment of context is then used to make key decisions in the next stage of the study. Qualitative
findings are often used to choose the most appropriate implementation strategies or, at the very least, to tailor strategies
to settings and actors. The timing of when the intervention must be implemented, therefore, impacts how the baseline
data will be analyzed. With a typically shorter turnaround time, qualitative researchers in implementation science have
turned toward rapid analysis techniques, including template analysis, in which key concepts are identified in advance to
structure and focus the inquiry (Reger et al., 2017; Hamilton, 2013; Fox et al., 2016).

Qualitative researchers are often asked to conduct formative evaluations of implementation processes, which may entail
documentation of implementation strategies as they are used in real time. Qualitative researchers may also be asked to
provide feedback as the implementation process is occurring to aid in the change process, asking questions such as,
what are the major facilitators and barriers to implementation, and what could be changed to improve implementation
and sustainability?

Finally, qualitative research may be conducted at the end of an implementation study to understand and explain both
implementation and effectiveness outcomes. It may also be used to assess and consider the reactions of adopters of
the intervention as well as stakeholders who resisted adoption.

An issue related to timing is the iterative nature of qualitative and mixed methods research in implementation science.
Understanding and facilitating implementation and sustainment of innovative and evidence-based practices is a
continuous learning process. With multiple time points of data collection and attention towards change, this can
make harnessing the iterative nature of qualitative research more difficult. This also raises the issue of when qualitative
methods “cross over” and become part of the implementation strategies, e.g., when key stakeholder interviews become
mechanisms of education about and engagement in an intervention (Hamilton et al., 2017).

C. Roles of Qualitative Research Teams in Implementation Science

Implementation science is a truly interdisciplinary field. The health-related implementation science team generally
includes clinical experts, social-behavioral scientists, biostatisticians, economists, intervention developers, and
organizational scientists (Fernald & Duclos, 2005; Guest & MacQueen, 2008). Qualitative researchers need to consider
this team as they develop interview and observational guides as well as when they decide who will conduct the
interviews, and who will participate in the analysis process (Damschroder & Lowery, 2013). The qualitative expert
will need to lead these activities while managing the interdisciplinary team’s involvement. Leading a multidisciplinary
team can be a challenging undertaking (see Vogel et al., 2014; and Hesse-Biber, 2016; Crabtree et al., 1994, on team
science). The challenge can be amplified when the qualitative expert is not the principal investigator. See Section VIII.

D. Data Collection
Several different data collection techniques have been used in implementation science, such as one-on-one interviews,
focus groups, observation, and archival or document analysis

Interviews can be conducted either in-person, by telephone, or electronic media (e.g., Skype) and are ideally digitally
recorded and transcribed verbatim for analysis. (Mealer & Jones, 2014; Kothari et al., 2012; DiCiccio-Bloom & Crabtree,
2006), Also used in implementation science are semi-structured interviews or elicitation methods techniques that utilize
a quasi-statistical perspective. For example, Palinkas and colleagues (2016) reported using a free list exercise to elicit

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 10
 participants’ conceptions of the term sustainment, the elements of their programs they wished to sustain, and the
requirements for sustaining these elements. Responses were weighted based on order of presentation and compared
by source of program funding.

Focus groups, also called group interviews, are widely used in health research and in implementation science
(Reichstadt et al. 2007; Tausch & Menold, 2016; Kitzinger, 2006; Krueger & Casey, 2000; Thomas et al., 1992). The
advantage of the group interview approach is that it capitalizes on within group interaction to stimulate thought resulting
in rich detailed accounts of the focal topic (Asbury 1995). Focus groups are usually held on-site, gathering participants
in a common locality (Howard et al., 2017) but they can also be held online. DuBois and colleagues (2015) conducted
online focus groups about decision-making among gay and bisexual adolescent males. Focus groups tend to be
popular with researchers not only because of the results they generate, but also because they frequently offer a more
economic use of financial and spatial resources as well as time. However, there are drawbacks and limitations to focus
groups and the decision to use focus groups or not really depends upon the information being sought. Carey and Smith
(1994) noted the major pitfall of the focus group technique is the potential impact of censoring and conforming behavior
in group interaction. Participants can be strongly influenced by each other, potentially narrowing the range of ideas
obtained. Also, participants may feel uncomfortable to express a sincerely held idea or opinion, which calls into question
the accuracy of results (Smithson, 2000).

Observation often occurs during site visits, which are conducted for a variety of reasons, including assessing local
contexts and observing the nature and intensity of how interventions are being implemented (Cohen et al., 2016; Fox
et al., 2016; Palinkas et al., 2016). Data gleaned from observation is typically in the form of field notes, which may be
more structured in implementation science than in traditional ethnographic research, in order to facilitate more rapid
or targeted analysis. When conducted over a period of time, such observations capture the interactions and behaviors
of participants in their natural settings (e.g., a hospital or community-based organization). For example, Escoffery and
colleagues (2009) made annual observations of 40 swimming pools during the five-year, mixed-methods implementation
study of an evidence based skin cancer prevention program along with qualitative interviews and quantitative surveys.

Archival analysis entails the collection of documents, including meeting minutes, logs, policies, guidelines, and health
records (Turner et al., 2016; Lessard et al., 2016). Pelcastre-Villafuerte and colleagues (2014) evaluated the processes
and performance of Project “Casas” to deliver culturally and linguistically appropriate services to indigenous women
in four Mexican states. Data were collected through extraction from archived regulatory and program documents as
well as semi-structured interviews. The study not only evaluated the past performance of the project, it also provided
information for the improvement of the existing program sites, as well as more cost-effective implementation of the
model in new sites. Other archival data include diaries and videotapes used to document implementation experiences
(Cohen et al., 2016; Elsey et al., 2016), videotaping, and photography (e.g., photo voice) (Kowitt et al., 2015).

Implementation science typically includes multiple data sources to aid in triangulation. For example, Kennedy and
colleagues (2016) employed a longitudinal case study design in which ethnographic methods comprising video, non-
participant observation of intervention delivery, and qualitative interviews at baseline, six and 12 months were used
to identify processes and dynamics of delivering a social network intervention and to capture individual outcomes of
the use of a web-based tool. Shea et al. (2014) studied the feasibility of implementing a virtual tumor board program
that connected community-based clinicians to academic tumor boards. Methods included observation of virtual
tumor board cases using a structured observation checklist and semi-structured interviews with participants. In an
implementation study of behavioral health counseling in primary care practices, Cohen and colleagues (2011) used visit
notes, documents, interviews, and online implementation diaries in a comparative analysis of nine different interventions
in nine practice-based research networks. They found practice health risk assessment and brief counseling, coupled
with referral and outreach to a valued and known counseling resource, to be the best way to consistently coordinate
and encourage follow-through for health behavior counseling.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 11
 E. Data Analysis
A commonly used analytic approach to qualitative data in implementation science is thematic analysis (Hsieh & Shannon,
2009). There are several variations of thematic analysis, ranging from techniques that are largely deductive to techniques
that are largely inductive, with substantial variation in between. Reflecting more of a deductive approach to qualitative
data analysis, several studies (Bajunirwe et al., 2016; Elsey et al., 2016) have employed a framework or template
approach (Gale et al., 2013; Parkinson et al., 2016; Srivristava & Thomson, 2009) to data analysis. See Section VIII. Such
approaches are theory- or framework-driven and involve mapping data or emergent themes onto a priori domains,
themes, or categories. Cadogan and colleagues (2016) used a framework analysis approach to map emergent themes
onto the Theoretical Domains Framework (Michie et al., 2014). Several studies (Elsey et al., 2016; Kennedy et al., 2016;
Nordmark et al., 2016) have used NPT (May & Finch, 2009) to provide conceptual orientation for exploratory data analysis
of complex interventions.

One of the better known examples of a framework approach is the use of the CFIR (Damschroder et al., 2009) to
guide the analysis of data collected through semi-structured interviews. For example, Bekelman and colleagues (2016)
conducted semi-structured interviews, using the CFIR domains to assess barriers and facilitators to the adoption and
scale-up of outpatient palliative care. Varsi and colleagues (2015) used the CFIR to examine barriers and facilitators for
the implementation of an internet-based patient-provider communication service (Forman et al., 2017). Hamilton and
colleagues (2018) used CFIR post-hoc to examine the impact of external context on implementation of an evidence-
based prevention intervention (Hamilton et al., 2018).

Content analysis typically involves identifying pre-selected incidents or references in the data (Padgett, 2016). Frequently
used with archival materials, content analysis does not entail a search for deeper meanings and interpretations.
Padwa and colleagues (2016) conducted a directed content analysis of qualitative data using the Conceptual
Model of Evidence-Based Practice Implementation in Public Service Sectors (Aarons et al., 2011) to understand the
implementation of integrated behavioral health protocols in primary care settings.

Since theoretical frameworks are generally expected to drive implementation science (Tabak et al., 2012), highly
inductive approaches such as grounded theory (Charmaz, 2006) may not always be a good fit for analyzing qualitative
data gathered in implementation science (Davidoff et al., 2015). In grounded theory, preconceived conceptual categories
must “earn their way” into the findings, rather than drive the data collection effort (as in implementation science).
However, principles of grounded theory can still be useful in informing implementation-related qualitative analysis (e.g.,
being data-driven).

Perhaps the most common strategy for data analysis is to combine inductive and deductive approaches. For instance,
Mozaffar et al. (2016) conducted a thematic inductive and deductive analysis, using a biography of artifact perspective
to move beyond a single timeframe and a “localist” perspective and to examine both technical and nontechnical
aspects of computerized hospital physician order entry and clinical decision-support systems. Colon-Emeric et al.
(2016) analyzed interview transcripts using framework analysis of a priori concepts, combined with inductive analyses.
Sommerbakk and colleagues (2016) used a combination of thematic analysis using an inductive approach and
theoretical thematic approach, applying codes to Grol & Wensing’s (2004) multilevel model of barriers and facilitators.
Using a combination of inductive and deductive approaches, analysts can draw on ideas emerging from the empirical
data as well as frameworks and existing literature (Bradley, Curry, & Devers, 2007).

Whether inductive, deductive, or a combination of the two approaches, thematic analysis usually involves the process
of constant comparison (Strauss & Corbin, 1998) to construct themes based on a taxonomy of codes. Themes may
be compared across organizations and stakeholder groups (Kane et al., 2016). In the rapid analytic approach used by
Hamilton (2013), main topics (domains) are drawn from interview and focus group guides, and a summary template is

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 12
 developed and used to summarize transcripts (e.g., Fox et al., 2016). Summaries are analyzed using matrix analysis and
key actionable findings (e.g., the variable use of implementation strategies) are shared with the implementation team to
guide further implementation in real time, particularly during the course of phased designs such as in a Hybrid Type II
Study (Curran et al., 2012). This rapid approach can inform more in-depth hybrid analytic approaches, described above.

In terms of assessing interrater reliability during data analyses, coding independently then meeting as a group to reach
consensus appears to be the most common strategy (Bekelman et al., 2016; Cadogan et al., 2016; Lessard et al., 2016;
Lord et al., 2016; Martinez et al., 2016). During these meetings, investigators compare how the same texts are coded by
different coders and resolve discrepancies in terminology and assignment of codes (Fox et al., 2016). Some qualitative
studies (Damschroder & Lowery, 2013; Lord et al., 2016) make explicit use of a consensus-based directed content
analysis approach (Hill, 2012). In a few instances, this comparison is quantified by calculating percent agreement or
kappa statistics in assignment of codes (Palinkas et al., 2013; Saldana, 2014). Occasionally, a qualitative expert not
integrally involved in the study is employed to review or “audit” the process as an effort to maximize validity of findings
(Damschroder & Lowery, 2013). (For a thorough discussion of team-based qualitative analysis see Guest & McQueen,
2008, and consult section VI/c, above.)

F. Presentation of Qualitative Findings

It is a challenge to present depth of qualitative analysis in the limited space allowed for journal articles. Most
implementation science presents qualitative results organized by themes illustrated with direct quotations from study
participants, an approach that is accessible to readers (Sandelowski & Leeman, 2012). Results should not merely be
comprised of a list of themes, but should specify which themes were emergent and which were a priori, and when
relevant, how themes are related to one another. Although verbatim quotations are commonly used to document
the existence or character of a theme, they should also provide a level of eloquence and insight not possible with
a summary of the theme. Usually, one quote is sufficient to make a specific point, unless the point itself contains
important nuances or contrasts that require more than one quotation to illustrate.

When using quotes, it is advisable to link them to a participant identification number so that readers can be assured
that different participants are being quoted. When multiple stakeholders are involved, identifying their role (e.g., hospital
administrator, nurse, physician), if permissible, helps to contextualize what is being said and by whom. In addition, for
multi-site studies, it is useful to use identifiers to distinguish between sites and demonstrate contextual patterns within
and across sites and/or experimental conditions, if relevant.

Although themes and associated quotes typically form the narrative structure of the findings, tables have also been
used to organize the presentation of themes with illustrative quotes. Tables have the advantage of presenting the
results in a condensed form, enabling authors to publish in journals that have more stringent word limitations. Elsey and
colleagues (2016) used qualitative data to construct a table of barriers and facilitators to implementation using NPT (May
et al., 2009). Also, Guetterman, Fetters, and Creswell (2015) investigated mixed methods displays and discussed what
each display type communicated and represented.

A third form of presentation of study results is the use of figures and diagrams that illustrate relationships among themes
identified from analyses of qualitative data. Palinkas and colleagues (2008) used a figure to illustrate relationships among
themes and subthemes representing predictors and outcomes of short-term implementation of three evidence-based
treatments for depression, anxiety, and behavioral problems in 8- to 13-year-old youths. Lessard and colleagues
(2016) used figures to depict a conceptual framework of facilitation categories and themes, the relationship between
implementation-oriented facilitation roles and support-oriented facilitation roles, and the dynamic interrelations between
groups of actors and/or between actors in the same group of facilitators. Elsey and colleagues (2016) used qualitative
data to construct a figure outlining patient flow in a proposed smoking cessation intervention.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 13
 VII . COLLEC TING AND ANALY Z ING
QUALITATIVE DATA WITH RIGOR

Qualitative methods encompass diverse approaches, yet they share a need to foster standards of quality and
methodological rigor (Lincoln & Guba, 1985; Tracy, 2010). Given differences in paradigmatic or epistemological
orientations, qualitative researchers may adhere to a non-prescriptive interpretive approach or they may prefer a
somewhat more structured or formalized approach (post-positivist) to determining quality. Most qualitative researchers
in health services and implementation science opt for explicit standards without sacrificing the flexibility that is the
hallmark of qualitative inquiry.

It is helpful to distinguish between criteria applied to completed qualitative studies (evaluative standards) and specific
strategies or actions undertaken as part of the study’s design and procedures. This distinction, beginning with a
discussion of evaluative standards and moving on to recommending specific strategies for rigor, will comprise this
section. Evaluation standards are an imperative from the beginning to insure all members of the team agree upon the
procedures and basic techniques that will be used to determine assess outcomes (Frankel, 1999; Cohen & Crabtree,
2008). Such standards have become especially salient with the rise of qualitative systematic reviews where the
diversity of methods and approaches makes consensus a challenging but necessary goal (Tracy, 2010).

The provision of evaluative standards specific to qualitative inquiry began with Lincoln and Guba (1985) and later gave
rise to standards in various disciplines, including medicine and healthcare (Mays & Pope, 1995). In 2008, the Robert
Wood Johnson Foundation’s Qualitative Research Guidelines Project reviewed the extant literature and promulgated
guidelines ([Link]/[Link]) for qualitative researchers. These guidelines begin with an overall indication of
quality, known as trustworthiness, meaning the qualitative study inspires confidence in the ways it was conducted and
the data were interpreted.

The Cochrane Collaboration has assembled a list of checklists and software tools that might be applied to assess the
quality of qualitative studies, and advocates a focus on four core principles: credibility, transferability, dependability,
and confirmability. Qualitative findings are “transferable” by yielding new concepts or frames of reference that can
apply to different settings. Examples of the type of criteria that might be applied include, assessing (1) the qualitative
methods employed, which often differ from the terms authors use to name their methods; (2) the relevance of sources
(informants, documents, etc.) to research questions; (3) efforts employed to ensure reliability in data analysis; (4)
consideration of how findings relate to context; and (5) extent to which sample size and composition are adequately
described and sufficient to support the findings (Hannes, 2011). Qualitative research methods value “thick and rich”
description (see Robert Wood Johnson Foundation at [Link]/[Link]).

Qualitative researchers do not try to fully eliminate subjective researcher bias from a study. In other words, they accept
that scientific inquiry cannot be value-free. This contrasts with much quantitative research, which seeks to remove
subjective bias and control for or eliminate other sources of bias. However, diligent qualitative researchers seek to
minimize bias, where possible, by identifying their biases early on and reckoning with them throughout a study. This
is accomplished through “reflexivity,” or maintaining vigilance and mindfulness during the study, acknowledging one’s
biases and reflecting on them in a way that lends credence to the findings.

Some qualitative researchers believe that they need not (and should not) adhere to quantitative standards of internal
and external validity as these would compromise the very nature and value of qualitative methods. However, qualitative
researchers do not share complete consensus on evaluative standards nor the optimal means of meeting these
standards. We suggest that the best approach for methodological integrity is for implementation researchers to
articulate the standards they use for trustworthiness and be transparent in describing the methods used and rationale
for their use. Once the study is complete, the researchers are accountable for demonstrating that their chosen
standards were met in the design, conduct, and analysis.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 14
 Enhancing trustworthiness implies use of strategies for rigor throughout the study (Padgett, 2016). Unlike evaluative
standards that inform the study and establish criteria for evaluating its overall quality, strategies for rigor are specific
techniques or actions taken before and during the course of the study. When it comes to rigor strategies, there are no
rigid formulaic expectations but there is a menu of options from which the researcher can choose, as befits the study
design. Below is a list of the most commonly used strategies and what they entail.

• Triangulation of data—using more than one source of data (e.g., interview transcripts, field notes, archival
material, survey results).

• Prolonged engagement—with study participants and in study settings, via multiple interviews, extended
periods of observations, etc.

• Member checking—returning to participants to verify or validate portions of the data or interpretations. This
can be especially important in community-based research as well as patient-centered research.

• Audit trail—the use of analytic memos, journals, and other means of documenting the study’s procedures to
enhance transparency of methods.

• Peer debriefing and support—meeting regularly with the study team to share study findings and processes
to identify biases as well as support one another.

• Negative or deviant case analysis—seeking alternative explanations in the data to avoid foreclosing
analyses and conclusions prematurely.

For implementation science that incorporates qualitative methods, some, but not necessarily all, of these strategies
would be appropriate. For example, prolonged engagement, the sine qua non of ethnography, is more challenging in
implementation science, but not impossible. In the European hospital study by Sax and colleagues (2013), prolonged
engagement entailed devoting two days to ethnographic site visits to hospitals. Their rationale stated, “Staying for two
days rather than just one allows us to take advantage of the exposure re-exposure effect. This helps to counter the
Hawthorne effect,” (Sax et al., 2013) wherein individuals modify their behavior in response to their awareness of being
observed (McCambridge et al., 2014).

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 15
 VIII . NEED AND OPPORTUNITIES FOR
INNOVATION IN US E OF QUALITATIVE
ME THODS IN IMPLEMENTATION SCIENCE

Use of qualitative methods in implementation science reflects practices consistent with their use in the broader fields
of health services and the social and behavioral sciences. However, the demand for such methods in implementation
science represent both needs and opportunities for using existing methods in innovative ways or developing entirely
new methods. First, though team-based qualitative methods are commonly used in implementation science (e.g.,
Reger et al., 2017; Damschroder & Lowery, 2013; Moeckli et al., 2013), there is a need to bring greater transparency
to and documentation of team-based analysis (Guest & MacQueen, 2007). See Section VI. Most qualitative studies of
implementation describe holding meetings that are designed to reach consensus in coding of data or construction of
themes. However, it is unclear what constitutes consensus (e.g., is it 100 percent agreement on all decisions, greater
than 50 percent agreement), and whether reaching consensus represents a uniform process with a uniform outcome
such that it can be replicated within the same study or compared from one study to the next.

Second, we need to continue to strengthen tools and techniques for conducting rapid qualitative assessment and
analysis. Rapid assessment procedures (RAP) are frequently used in program planning and evaluation (Cifuentes et
al., 2006; Scrimshaw & Hurtado, 1987; Scrimshaw et al., 1991). RAP is generally characterized by a small research
team and compressed time frame, in which specifically described and quite limited research questions are addressed
using multiple sources of qualitative data such as observation and interviews (Beebe, 2001; Utarini et al., 2001). RAP
is especially appropriate for implementation research because it (1) reduces the time required to conduct the study,
thereby reducing the gap between research and practice, and (2) is more likely to involve a partnership between
researchers and practitioners, enabling both groups to contribute to study results and to learn from one another in
the process. Innovations in the use of RAP might include developing standardized procedures for training research
assistants, use of social media and other forms of technology for data collection and analysis, and developing quality
control procedures for insuring consistency and rigor in activities conducted by teams of researchers and practitioners.
(See also Ash et al., 2008, for a variation on RAP.)

The template or framework approach mentioned earlier is well-suited to RAP as it provides a ready-made structure for
the inquiry based upon key constructs identified in advance (Padgett, 2016). Pioneered by Crabtree and Miller (1999)
and Miles and Huberman (1994), the template approach was used by Hamilton (2013) to quickly abstract urgently
needed data from a broader, deeper data pool following a predetermined template. More specification of the template
approach and its utility is needed in future implementation science. See Section VI.

Third, it is important that qualitative methodologists explore methods of qualitative data collection and analysis not
commonly used in implementation science (Jennings, Sandelowski, & Higgins, 2013). These include phenomenological
methods to understand the lived experience (van Manen, 2015) of implementation. As well, elicitation techniques such
as pile sorts, photo elicitation interviews, semantic frames, and rank order tasks represent more structured, quasi-
statistical ways to assess stakeholder perceptions of implementation goals, requirements, barriers, facilitators, and
outcomes.

Fourth, there is a need to contribute to the development of a common language when conducting qualitative research
in implementation science. Semantic consistency is needed for clear communication and meaningful comparisons,
especially in a multidisciplinary field such as implementation science. Work on pragmatic, consistent, intelligible
terminology to describe qualitative methods in implementation science is much needed (Colquhohn et al., 2014;
Rabin et al., 2008). For example, qualitative reports in implementation science should be transparent and explicit in
describing the methods used, e.g., clarifying how grounded theory informed one’s analytic approach. They should also
describe what and when strategies for rigor were used to enhance trustworthiness (Padgett, 2016).

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 16
 Fifth, a field that is as context-dependent as implementation science needs to develop meaningful approaches for
cross-context comparison and synthesis of qualitative data. Too much emphasis on standardization can be counter to
the intent and value of qualitative inquiry. Still, a theoretically sound and demonstrably useful approach to contextual
comparisons of qualitative data for the specific purpose of informing implementation science is much needed.

There are currently several methods that synthesize and compare qualitative research findings such as meta-synthesis,
meta-ethnography, qualitative synthesis, and others. Leeman et al. (2010) used realist synthesis methods to identify
and summarize context and process related evidence that might advance understanding of factors that influence the
implementation of antiretroviral therapy adherence interventions. All contend with the challenge that qualitative data
is inherently contextual and that qualitative methodologists do not always agree on the uniform standards of quality
that such syntheses necessitate (Padgett, 2016). This tension between maintaining contextualization and diversity
of approach while trying to synthesize findings across very different implementation sites and qualitative studies
represents an ongoing challenge in implementation science (see Benzer et al., 2013, on between and within site
variation). The emphasis by some researchers on technology of synthesis or meta-analysis at the expense of attention
to epistemology and conceptualization has been cautioned against by several authors (Sandelowski et al., 2006;
Bondas & Hall, 2007; Finfgeld-Connett, 2016; Ludvigsen et al., 2016; Thorne, 2017). The use of these approaches
in implementation science are important to increase the utility of qualitative studies, yet as the authors cited above
have noted, this must be based on theoretically sound and solid methodological research. Certainly, there is much
methodological work to be done.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 17
 IX . CONCLUSION

Rigorous qualitative methods are essential to the field of implementation science. Many implementation researchers
have limited or no experience in qualitative methods, despite the wide use of these methods in the field. Furthermore,
many qualitative experts have little to no knowledge of the sphere and unique challenges of implementation science,
despite the centrality of qualitative research in implementation science. Hence, the QUALRIS group has offered its
collective knowledge and experience to describe the value of qualitative methods, methodological considerations,
typical approaches to the use of qualitative methods, and needs and opportunities for innovation.

This manuscript has notable limitations. The authors have tried to communicate at a level accessible to researchers
new to qualitative approaches, avoiding jargon and advanced methodological concepts. The paper is not intended
to be a comprehensive rendering of the difficulties and dilemmas encountered in the conduct of qualitative inquiry
in implementation science. It does not systematically review the multitude of studies that have encountered and
attempted to address the challenges posed. Another important limitation is the paper’s tendency to focus on
implementation science conducted within the United States. This likely reflects the research emphases of the authors
and limits to the group’s time and resources, which prohibited a broader international scope. The authors urge other
researchers to expand this focus to a global exploration of how qualitative methods are used in implementation
science, the difficulties encountered, and solutions that have been generated.

There is a pressing need for methodological innovations to meet the challenges for the rigorous use of qualitative
methods in implementation science. There is also great opportunity to advance both qualitative methodology and
implementation science in pursuing such innovations. This can be facilitated by an implementation science-wide
conversation about these topics. We have offered here some perspectives on the value and use of qualitative methods
in implementation science. We invite the larger research community to contribute to the dialogue.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 18
 X . ACKNOWLEDGEMENT S

The members of the QUALRIS group would like to sincerely thank our group of advisors (listed below) for their
exceptional contributions to this manuscript. They read several iterations of this paper, giving generously of their
time, intellect, and experience. Our advisors’ thoughtful comments and discussion helped us to greatly improve our
final product.

Ross Brownson, Ph.D. Bryan J. Weiner, Ph.D.

Brown School Department of Global Health
School of Medicine Department of Health Services
Washington University in St. Louis University of Washington

Cam Escoffery, Ph.D., M.P.H. Anne Sales, Ph.D., R.N., M.S.N.

Rollins School of Public Health University of Michigan Medical School
Emory University VA Ann Arbor Healthcare System

Russell E. Glasgow, Ph.D.

School of Medicine
University of Colorado

We would also like to thank David Chambers who first gave his support for the idea of this effort. We thank him for
his guidance and forbearance through the non-linear writing process.

We wish to thank Kaelin Rapport and Jordan Tompkins for their assistance with the literature review and
substantive consultations.

Finally, thanks go to NCI’s Implementation Science Team for their feedback and support through the production of
this paper.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 19
 REFERENCES

Aakhus, E., Granlund, I., Oxman, A. D., & Flottorp, S. A. (2015). Tailoring interventions to implement recommendations for the treatment of elderly
patients with depression: A qualitative study. International Journal of Mental Health Systems, 9(1), 36.
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service
sectors. Administration and Policy in Mental Health, 38, 4-23.
Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration
and Policy in Mental Health and Mental Health Services Research, 34, 411-419.
Abbott, P. A., Foster, J., Marin, H., de F., & Dykes, P. C. (2014). Complexity and the science of implementation in health IT—Knowledge gaps and
future visions. International Journal of Medical Informatics, 83(7), e12-22.
Asbury, J. E. (1995). Overview of focus group research. Qualitative Health Research 5(4), 414-420.
Ash, J. S., Sittig, D. F., McMullen, C. K., Guappone, K., Dykstra, R., & Carpenter, J. (2008). A rapid assessment process for clinical informatics
interventions. AMIA Annual Symposium Proceedings, 26-30.
Azar, J., Adams, N., & Boustani, M. (2015). The Indiana University Center for Healthcare Innovation and Implementation Science: Bridging
healthcare research and delivery to build a learning healthcare system. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen,
109(2), 138-143.
Bajunirwe, F., Tumwebaze, F., Abongomera, G., Akakimpa, D., Kityo, C., & Mugyenyi, P. N. (2016). Identification of gaps for implementation science
in the HIV prevention, care and treatment cascade; A qualitative study in 19 districts in Uganda. BMC Research Notes, 9, 217.
Bate, P., Mendel, P., & Robert, G. (2008). Organizing for quality: The improvement journeys of leading hospitals in Europe and the United States.
NY: Radcliffe Publishing.
Beebe, J. (2001). Rapid assessment process: An introduction. Walnut Creek, CA: Altamira Press.
Beidas, R. S., Wolk, C. L., Walsh, L. M., Evans, A. C., Hurford, M. O., & Barg, F. K. (2014). A complementary marriage of perspectives:
Understanding organizational social context using mixed methods. Implementation Science, 9(1), 1-15, 175.
Bekelman, D. B., Borsika, R. A., Nowels, C. T., Sahay, A., Heidenreich, P. A., Fischer, S. M., & Main, D. S. (2016). Barriers and facilitators to scaling
up outpatient palliative care. Journal of Palliative Medicine, 19(4), 456-459. doi:10.1089/jpm.2015.0280
Benzer, J. K., Beehler, S., Cramer, I. E., Mohr, D. C., Charns, M. P., & Burgess, J. F. (2013). Between and within-site variation in qualitative
implementation research. Implementation Science, 8, 4.
Bishop, F. L. (2015). Using mixed methods research designs in health psychology: An illustrated discussion from a pragmatist perspective. British
Journal of Health Psychology, 20(1), 5-20.
Blasinsky, M., Goldman, H. H., & Unützer, J. (2006). Project IMPACT: A report on barriers and facilitators to sustainability. Administration and Policy
in Mental Health and Mental Health Services, 33, 718-729.
Bondas, T., & Hall, E. O. C. (2007). Challenges in approaching metasynthesis research. Qualitative Health Research, 17(1), 113-121.
Bradley, E. H., Curry, L. A., & Devers, K. J. (2007). Qualitative data analysis for health services research: Developing taxonomy, themes and theory.
Health Services Research, 42, 1758-1772.
Brear, M. R., Shabangu, P. N., Fisher, J. R., Hammarberg, K., Keleher, H. M., & Livingstone, C. (2018, April 1). Health capability deprivations in
a rural Swazi community: Understanding complexity with theoretically informed, qualitatively driven, mixed-method design, participatory action
research. Qualitative Health Research. doi: 10.1177/1049732318768236
Brives, C., Le Marcis, F., & Sanabria, E. (2016). What’s in a context? Tenses and tensions in evidence-based medicine. Medical Anthropology,
35(5), 369-376.
Brownson, R. C., Colditz, G. A., & Proctor, E. K. (2018) Dissemination and implementation research in health: Translating science to practice (2nd
ed.) NY: Oxford University Press.
Brownson, R. C., Dreisinger, M., Colditz, G. A., & Proctor, E. K. (2012). The path forward in dissemination and implementation research. New York,
NY: Oxford University Press.
Bunger, A. C., Powell, B. J., Robertson, H. A., MacDowell, H. A., Birken, S. A., & Shea, C. (2017). Tracking implementation strategies: A description
of a practical approach and early findings. Health Research Policy and Systems, 15(1), 15.
Cadogan, S. L., McHugh, S. M., Bradley, C. P., Browne, J. P., & Cahill, M. R. (2016). General practitioner views on the determinants of test ordering:
A theory-based qualitative approach to the development of an intervention to improve immunoglobulin requests in primary care. Implementation
Science, 11(1), 102.
Carey, M. A., & Smith, M. W. (1994). Capturing the group effect in focus groups: A special concern in analysis. Qualitative Health Research, 4(1),
123-127.
Casillas, A., Zingg, W., Clack, L., Jantarada, F., da L., Touveneau, S., & Sax, H. (2013). Infection control through in depth eyes: A thematic study of
infection control practices across six European hospitals. Poster from 23rd European Congress of Clinical Microbiology. Berlin, Germany.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 20
 REFERENCES

Chalmers, I., & Glasziou, P. (2009). Avoidable waste in the production and reporting of research evidence. Lancet, 374(9683), 86-89.
Chambers, D. A. (2014). Guiding theory for dissemination and implementation research: A reflection on models used in research and practice. In
R. S. Beidas & P.C. Kendall (Eds.), Child and Adolescent Therapy: Dissemination and Implementation of Empirically Supported Treatments. New
York: Oxford University Press.
Chambers, D. A., Feero, W. G., & Khoury, M. J. (2016). Convergence of implementation science, precision medicine, and the learning health care
system: A new model for biomedical research. JAMA, 315(18), 1941-1942.
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013). The dynamic sustainability framework: Addressing the paradox of sustainment amid
ongoing change. Implementation Science, 8, 117.
Chambers, D. A., & Norton, W. E. (2016). The adaptome: Advancing the science of intervention adaptation. American Journal of Preventive
Medicine, 51(4 Suppl 2), S124-131.
Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. London, UK: Sage.
Cifuentes, E., Alamo, U., Kendall, T., Brunkard, J., & Scrimshaw, S. (2006). Rapid assessment procedures in environmental sanitation research: A
case study from the northern border of Mexico. Canadian Journal of Public Health, 97(1), 24-28.
Clack, L., Casillas, A., Touveneau, S., Jantarada, F. da L., Zingg, W., & Sax, H. (2013). Factors influencing leadership dynamics in the context
of infection control: A cross-case analysis of six European hospitals. 19ème Journée Romande d’Hygiène Hospitalière (19th Romandy Day of
Hospital Hygiene). Montreux, Switzerland.
Cohen, D. J., Balasubramanian, B. A., Isaacson, N. F., Clark, E. C., Etz, R. S., & Crabtree, B. F. (2011). Coordination of health behavior counseling
in primary care. Annals of Family Medicine, 9(5), 406-415.
Cohen, D. J., Balasubramanian, B. A., Gordon, L., Marino, M., Ono. S., Solberg, L. I., & Cresswell, J. (2016). A national evaluation of a
dissemination and implementation initiative to enhance primary care practice capacity and improve cardiovascular disease care: The ESCALATES
study protocol. Implementation Science, 11, 86.
Cohen, D. J. & Crabtree, B. F. (2008). Evaluative criteria for qualitative research in health care: Controversies and recommendations. Annals of
Family Medicine, 6(4), 331-339.
Cohen, D. J., Crabtree, B. F., Etz, R. S., Balasubramanian, B. A., Donahue, K., Leviton, L. C., . . . Green, L. W. (2008). Fidelity vs. flexibility:
Translating evidence-based research into practice. American Journal of Preventive Medicine, 35(5 Suppl), S381-389.
Colon-Emeric, C., Toles, M., Cary, M. P., Batchelor-Murphy, M., Yap, T., Song, Y., . . . Anderson, R. A. (2016). Sustaining complex interventions in
long-term care: a qualitative study of direct care staff and managers. Implement Science, 11, 94.
Colquhoun, H., Leeman, J., Michie, S., Lokker, C., Bragge, P., Hempel, S., & Grimshaw, J. (2014). Towards a common terminology: A simplified
framework of interventions to promote and integrate evidence into health practices, systems, and policies. Implementation Science, 9, 51.
Concannon, T. W., Meissner, P., Grunbaum, J. A., McElwee, N., Guise, J. M., Santa, J., . . . Leslie, L. K. (2012). A new taxonomy for stakeholder
engagement in patient-centered outcomes research. Journal of General Internal Medicine, 27, 985–991.
Corbin, J. M. & Strauss, A. (2008). Basics of qualitative research: Techniques and procedures for developing grounded theory (3rd ed.). Thousand
Oaks, CA: Sage Publications.
Crabtree, B. F. & Miller, W. (Eds.). (1999). Doing qualitative research (2nd ed.). London, UK: Sage.
Crabtree, B. F., Miller, W., Addison, R., Gilchrist, V. & Kuzel, A. (Eds.). (1994). Exploring collaborative research. Thousand Oaks, CA: Sage.
Creswell, J. W. (2015). A concise introduction to mixed-methods research. Thousand Oaks, CA: Sage Publications.
Creswell, J. W., Klassen, A. C., Clark, V. L. P., & Smith, K. C. for the Office of Behavioral and Social Sciences Research, National Institutes of
Health. (2011). Best practices for mixed methods research in the health sciences. Retrieved March 26, 2018 from [Link]
mixed-methods-research/
Creswell, J. W., & Clark, V. L. P. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.
Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012). Effectiveness-implementation hybrid designs: Combining elements of
clinical effectiveness and implementation research to enhance public health impact. Medical Care, 50(3), 217-226.
Dalkin, S. M., Greenhalgh, J., Jones, D., Cunningham, B., & Lhussier, M. (2015). What’s in a mechanism? Development of a key concept in realist
evaluation. Implementation Science, 10(1), 49.
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services
research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50.
Damschroder, L. J., Goodrich, D. E., Robinson, C. H., Fletcher, C. E., & Lowery, J. C. (2011). A systematic exploration of differences in contextual
factors related to implementing the MOVE! weight management program in VA: A mixed methods study. BMC Health Services Research, 11(1), 248.
Damschroder, L. J., & Lowery, J. C. (2013). Evaluation of a large-scale weight management program using the consolidated framework for
implementation research (CFIR). Implementation Science, 8, 51.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 21
 REFERENCES

Damschroder, L. J., Reardon, C. M., AuYoung, M., Moin, T., Datta, S. K., Sparks, J. B., . . . Richardson, C. R. (2017). Implementation findings
from a hybrid III implementation-effectiveness trial of the Diabetes Prevention Program (DPP) in the Veterans Health Administration (VHA).
Implementation Science, 12(1), 94.
Damschroder, L. J., Reardon, C. M., Sperber, N., Robinson, C. H., Fickel, J. J., & Oddone, E. Z. (2017). Implementation evaluation of the Telephone
Lifestyle Coaching (TLC) program: Organizational factors associated with successful implementation. Translational Behavioral Medicine, 7(2), 233-241.
Davidoff, F., Dixon-Woods, M., Leviton, L., & Michie, S. (2015). Demystifying theory and its use in improvement. BMJ Quality & Safe, 24(3), 228-238.
Davies, P., Walker, A., & Grimshaw, J. (2010). A systematic review of the use of theory in the design and implementation strategies and
interpretation of the results of rigorous evaluations. Implementation Science, 5.
DiCicco-Bloom, B., & Crabtree, B. F. (2006). The qualitative research interview. Medical Education, 40(4), 314-321.
Dopson, S., & Fitzgerald, L. (2006). The active role of context. In S. Dopson & L. Fitzgerald (Eds.), Knowledge to action? Evidence-based health
care in context (p. 223). Oxford, UK: Oxford University Press.
DuBois, L. Z., Macapagal, K. R., Rivera, Z., Prescott, T. L., Ybarra, M. L., & Mustanski, B. (2015). To have sex or not to have sex? An online focus
group study of sexual decision making among sexually experienced and inexperienced gay and bisexual adolescent men. Archives of Sexual
Behavior, 44(7), 2027-2040.
Eccles, M., Grimshaw, J., Walker, A., Johnston, M., & Pitts, N. (2005). Changing the behavior of healthcare professionals: The use of theory in
promoting the uptake of research findings. Journal of Clinical Epidemiology, 58, 107-112.
Elsey, H., Khanal, S., Manandhar, S., Sah, D., Baral, S. C., Siddiqi, K., & Newell, J. N. (2016). Understanding implementation and feasibility of
tobacco cessation in routine primary care in Nepal: A mixed methods study. Implementation Science, 11, 104.
Escoffery, C., Glanz, K., Hall, D., & Elliott, T. (2009). A multi-method process evaluation of a skin cancer prevention diffusion study. Evaluation & the
Health Professions, 32(2), 184-203.
Fernald, D. H., & Duclos, C. W. (2005). Enhance your team-based qualitative research. Annals of Family Medicine, 3(4), 360-364.
Finfgeld-Connett, D. (2016). The future of theory-generating meta-synthesis research. Qualitative Health Research, 26(3), 291–293.
Forman, J., Heisler, M., Damschroder, L. J., Kaselitz, E., & Kerr, E. A. (2017). Development and application of the RE-AIM QuEST mixed methods
framework for program evaluation. Preventive Medicine Report, 6, 322-328.
Fox, A. B., Hamilton, A. B., Frayne, S. N., Wiltsey-Stirman, S., Bean-Mayberry, B., Carney, D., . . . Vogt, D. (2016). Effectiveness of an evidence-
based quality improvement approach to cultural competence training: The Veterans Affairs’ “Caring for Women Veterans” program. Journal for
Continuing Education in the Health Professions, 36, 96-103.
Foy, R., Ovretveit, J., Shekelle, P. G., Pronovost, P. J., Taylor, S. L., Dy, S., . . . Wachter, R. M. (2011). The role of theory in research to develop and
evaluate the implementation of patient safety practices. BMJ Quality & Safety, 20(5), 453-459.
Francis, J. J., Johnston, M., Robertson, C., Glidewell, L., Entwistle, V., Eccles, M. P., & Grimshaw, J. M. (2010). What is an adequate sample size?
Operationalizing data saturation for theory-based interview studies. Psychology & Health, 25(1), 1229-1245.
Frankel, R. (1999). Standards of qualitative research. In B. F. Crabtree & W. L. Miller (eds.), Doing qualitative research (2nd ed.) (pp. 333-346).
Thousand Oaks, CA: Sage.
Freedman, D. A., Whiteside, Y. O., Brandt, H. M., Young, V., Friedman, D. B., & Hebert, J. R. (2012). Assessing readiness for establishing a farmers’
market at a community health center. Journal of Community Health, 37(1), 80-88.
Gale, N. K., Heath, G., Cameron, E., Rashid, S., & Redwood, S. (2013). Using the framework method for the analysis of qualitative data in multi-
disciplinary health research. BMC Medical Research Methodology, 13, 117.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New York: Aldine de Gruyter.
Glasgow, R. E. (2008). What types of evidence are most needed to advance behavioral medicine? Annals of Behavioral Medicine, 35(1), 19-25.
Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003). Why don’t we see more translation of health promotion research to practice? Rethinking
the efficacy-to-effectiveness transition. American Journal of Public Health, 93(8), 1261-1267.
Glasgow, R. E., Vinson, C., Chambers, D., Khoury, M. J., Kaplan, R. M., & Hunter, C. (2012). National Institutes of Health approaches to
dissemination and implementation science: Current and future directions. American Journal of Public Health 102(7), 1274-1281.
Grol, R., & Wensing, M. (2004). What drives change? Barriers to and incentives for achieving evidence-based practice. Medical Journal of
Australia, 180(6 Suppl), S57–60.
Guest, G., & MacQueen, K. (2008). Handbook for team-based qualitative research. Lanham, MD: Altamira Press.
Guetterman, T. C., Fetters, M. D., & Creswell, J. W. (2015). Integrating quantitative and qualitative results in health science mixed methods research
through joint displays. Annals of Family Medicine, 13(6), 554-561.
Hadi, M. A., & Closs, S. J. (2016). Applications of mixed-methods methodology in clinical pharmacy research. International Journal of Clinical
Pharmacy, 38(3), 635-640.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 22
 REFERENCES

Hamilton, A. B. (2013, December 11). Qualitative methods in rapid turn-around health services research. Retrieved from [Link]
[Link]/for_researchers/cyber_seminars/archives/[Link]
Hamilton, A. B., Cohen, D., Heurtin-Roberts, S., Padgett, D., Palinkas, L., & Rabin, B. (2015, December). The value of qualitative methods in
implementation research. Oral presentation from the 8th Annual Conference on the Science of Dissemination and Implementation in Health.
Washington, DC.
Hamilton, A. B., Mittman, B. S., Campbell, D., Hutchinson, C., Liu, H., Moss, N. J., & Wyatt, G. E. (2018). Understanding the impact of external
context on community-based implementation of an evidence-based HIV risk reduction intervention. BMC Health Services Research, 18(1), 11.
Hamilton, A. B., Brunner, J., Cain, C., Chuang, E., Luger, T., Canelo, I., Rubenstein, L., & Yano, E. M. (2017). Engaging multilevel stakeholders in an
implementation trial of evidence-based quality improvement in VA women’s health primary care. Translational Behavioral Medicine, 7(3), 478-485.
Hamilton, A. B., Cohen, A. N., Glover, D. L., Whelan, F., Chemerinski, E., McNagny K. P., . . . Young, A. S. (2013). Implementation of evidence-
based employment services in specialty mental health. Health Services Research, 48(6 Pt 2), 2224-2244.
Hannes, K. (2011). Critical appraisal of qualitative research. In J. Noyes, A. Booth, K. Hannes, A. Harden, J. Harris, S. Lewin, & C. Lockwood
(Eds.), Supplementary guidance for inclusion of qualitative research in Cochrane systematic reviews of interventions. Retrieved from [Link]
[Link]/supplemental-handbook-guidance
Harvey, G., & Kitson, A. (2015). Implementing evidence-based practice in healthcare: A facilitation guide. New York, NY: Routledge.
Harvey, G., & Kitson, A. (2016). PARIHS revisited: From heuristic to integrated framework for the successful implementation of knowledge into
practice. Implementation Science, 11, 33.
Helfrich, C. D., Weiner, B. J., McKinney, M. M., & Minasian, L. (2007). Determinants of implementation effectiveness: Adapting a framework for
complex innovations. Medical Care Research and Review, 64(3), 279-303.
Hesse-Biber, S. (2016). Doing interdisciplinary mixed methods health care research: Working the boundaries, tensions, and synergistic potential of
team-based research. Qualitative Health Research, 26(5), 649-658.
American Psychological Association. (2012). Consensual qualitative research: A practical resource for investigating social science phenomena.
Washington, DC: Hill, C. E.
Hoagwood, K. E., Vogel, J. M., Levitt, J. M., D’Amico, P. J., Paisner, W. I., & Kaplan, S. J. (2007). Implementing an evidence-based trauma treatment
in a state system after September 11: the CATS Project. Journal of the American Academy of Child Adolescent Psychiatry, 46(6), 773-779.
Howard, B. N., Van Dorn, R., Myers, B. J., Zule, W. A., Browne, F. A., Carney, T., & Wechsberg, W. M. (2017). Barriers and facilitators to
implementing an evidence-based woman-focused intervention in South African health services. BMC Health Services Research, 17(1), 746.
Hsieh, H. F., & Shannon, S. E. (2009). Three approaches to qualitative content analysis. Qualitative Health Research, 15, 1277-1288.
Ivankova, N. V. (2015). Mixed methods applications in action research. Thousand Oaks, CA: Sage Publications.
Jennings, B.M., Sandelowski, M., & Higgins, M.K. (2013). Turning over patient turnover: An ethnographic study of admissions, discharges, and
transfers. Research in Nursing & Health, 36, 554-566.
Kane, H., Lewis, M. A., Williams, P. A., & Kahwati, L. C. (2014). Using qualitative comparative analysis to understand and quantify translation and
implementation. Translational Behavioral Medicine, 4(2), 201-208.
Kane, J. C., Adaku, A., Nakku, J., Odokonyero, R., Okello, J., Musisi, S., & Toi, W. A. (2016). Challenges for the implementation of World Health
Organization guidelines for acute stress, PTSD, and bereavement: A qualitative study. Implementation Science, 11, 36.
Kazdin, A. (2007). Mediators and mechanisms of change in psychotherapy research. Annual Review Clinical Psychology, 3(1), 1-27.
Kennedy, A., Vassilev. I., Jame, E., & Rogers, A. (2016). Implementing a social network intervention designed to enhance and diversify support for
people with long-term conditions. A qualitative study. Implementation Science, 11, 27.
Kessler, R. S., Purcell, E. P., Glasgow, R. E., Klesges, L. M., Benkeser, R. M., & Peek, C. J. (2013). What does it mean to “employ” the RE-AIM
model? Evaluation & the Health Professions, 36(1), 44-66.
Kirk, M. A., Kelley, C., Yankey, N., Birken, S. A., Abadie. B., & Damschroder, L. J. (2016) A systematic review of the use of the Consolidated
Framework for Implementation Research. Implementation Science, 11(1), 1-13.
Kitson, A., Brook, A., Harvey, G., Jordan, Z., Marshal, R., O’Shea, R., & Wilson, D. (2017). Using complexity and network concepts to inform
healthcare knowledge translation. International Journal of Health Policy and Management, 6, 1-13.
Kitson, A., Rycroft-Malone, J., Harvey, G., McCormack, B., Seers, K., & Titchen, A. (2008). Evaluating the successful implementation of evidence
into practice using the PARiHS framework: Theoretical and practical challenges. Implementation Science, 3, 1.
Kitzinger, J. (2006). Focus groups. In C. Pope & N. Mays (Eds.), Qualitative research in health care (3rd ed.) (pp. 21-31). Malden, MA: Blackwell.
Klein, K. J., Conn, A. B., & Sorra, J. S. (2001). Implementing computerized technology: An organizational analysis. Journal of Applied Psychology,
86(5), 811-824.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 23
 REFERENCES

Klein, K. J., & Sorra, J. S. (1996). The challenge of innovation implementation. The Academy of Management Review, 21(4), 1055-1080.
Koenig, C. J., Abraham, T., Zamora, K. A., Hill, C., Kelly. P. A., Uddo, M., Hamilton, M., Pyne, J. M., & Seal, K. H. (2016). Pre-implementation
strategies to adapt and implement a veteran peer coaching intervention to improve mental health treatment engagement among rural veterans.
Journal of Rural Health, 32(4), 418-428.
Kothari, A., Rudman, D., Dobbins, M., Rouse, M., Sibbald, S., & Edwards, N. (2012). The use of tacit and explicit knowledge in public health: a
qualitative study. Implementation Science, 7, 20
Kowitt, S., Woods-Jaeger, B., Lomas, J., Taggart, T., Thayer, L., Sutton, S., & Lightfoot, A. F. (2015). Using Photovoice to understand barriers to and
facilitators of cardiovascular health among African American adults and adolescents, North Carolina, 2011-2012. Preventing Chronic Disease, 12, E164.
Krueger, R. A., & Casey, M. A. (1994). Focus groups: A practical guide for applied research (2nd ed.). Thousand Oaks, CA: Sage.
Leeman, J., Chang, Y. K., Lee, E. J., Voils, C., Crandell, J. L., & Sandelowski, M. (2010). Implementation of antiretroviral therapy adherence
interventions: A realist synthesis of evidence. Journal of Advanced Nursing, 66, 1915-1930.
Leeman, J., Myers, A., Grant, J. C., Wangen, M., & Queen, T. L. (2017). Implementation strategies to promote community-engaged efforts to
counter tobacco marketing at the point of sale. Translational Behavioral Medicine, 7, 405-414.
Leeman, J., Teal, R., Jernigan, J., Reed, J. H., Farris, R., & Ammerman, A. (2014). What evidence and support do state-level public health
practitioners need to address obesity prevention. American Journal of Health Promotion, 28(3), 189-196.
Lessard, S., Barei, C., Lalonde, L., Duhamel, F., Hudon, E., Goudreau, J., & Levesque, L. (2016). External facilitators and interprofessional
facilitation teams: A qualitative study of their roles in supporting practice change. Implementation Science, 11, 97.
Lewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio. L., Jones, S., Walsh-Bailey, C., & Weiner, B. (2018). From classification to causality:
Advancing understanding of mechanisms of change in implementation science. Frontiers in Public Health, 6(136).
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic Inquiry. Newbury Park, CA: Sage Publications.
Lord, S., Moore, S. K., Ramsey, A., Dinauer, S., & Johnson, K. (2016). Implementation of a substance use recovery support mobile phone app in
community settings: Qualitative study of clinician and staff perspectives of facilitators and barriers. JMIR Mental Health, 3(2).
Ludvigsen, M. S., Hall, E. O. C., Meyer, G., Fegran, L., Aagaard, H., & Uhrenfeldt, L. (2016) Using Sandelowski and Barroso’s meta-synthesis
method in advancing qualitative evidence. Qualitative Health Research, 26(3), 320–329.
Lukas, C. V., Holmes, S. K., Cohen, A. B., Restuccia. J., Cramer, I. E., Shwartz, M., & Charns, M. P. (2007). Transformational change in health care
systems: An organizational model. Health Care Management Review, 32(4), 309-320.
Marshall, T., Rapp, C. A., Becker, D. R., & Bond, G. R. (2008). Key factors for implementing supported employment. Psychiatric Services, 59, 886-892.
Martinez, R. N., Hogan, T. P., Lones, K., Balbale, S., Scholten, J., Bidelspach, D., & Smith, B. M. (2016). Evaluation and treatment of mild traumatic
brain injury through the implementation of clinical video telehealth: Provider perspectives from the Veterans Health Administration. Physical
Medicine and Rehabilitation Journal, 9, 231-240
Marty, D., Rapp, C., McHugo, G., & Whitley, R. (2008). Factors influencing consumer outcome monitoring in implementation of evidence-based
practices: results from the National EBP Implementation Project. Administration and Policy in Mental Health, 35, 204-211.
May, C., & Finch, T. (2009). Implementation, embedding and integration: An outline of normalization process theory. Sociology, 43, 535-554.
May, C. (2013). Towards a general theory of implementation. Implementation Science, 8, 18.
May, C. (2006). A rational model for assessing and evaluating complex interventions in health care. BMC Health Services Research, 6, 86.
May, C., Mair, F., Finch, T., MacFarlane, A., Dowrick, C., Treweek, S., . . . Montori, V. M. (2009). Development of a theory of implementation and
integration: Normalization Process Theory. Implementation Science, 4, 29.
Mays, N., & Pope, C. (1995). Rigour and qualitative research. BMJ, 311(6997): 109-112.
McCambridge, J., Witton, J., & Elbourne, D. R. (2014). Systematic review of the Hawthorne effect: New concepts are needed to study research
participation effects. Journal of Clinical Epidemiology, 67(3), 267-277.
Mealer, M., & Jones, J. (2014). Methodological and ethical issues related to qualitative telephone interviews on sensitive topics. Nurse Researcher,
21(4), 32-37.
Meyer, J. (2000). Using qualitative methods in health-related action research. BMJ, 320, 178-181.
Michie, S., Atkins, L., & West, R. (2014). The behavior change wheel: A guide to designing interventions. London, UK: Silverback Publishing.
Miles, M. B., & Hubermann, M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage Publications.
Moeckli, J., Cram, P., Cunningham, C., & Reisinger, H. S. (2013). Staff acceptance of a telemedicine intensive care unit program: A qualitative
study. Journal of Critical Care, 28(6), 890-901.
Morgan, D. L. (1997). Focus groups as qualitative research. Newbury Park, CA: Sage.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 24
 REFERENCES

Mozaffar, H., Cresswell, K. M., Lee, L., Williams, R., & Sheikh, A. (2016). Taxonomy of delays in the implementation of hospital computerized
physician order entry and clinical decision support systems for prescribing: A longitudinal qualitative study. BMC Medical Informatics and Decision
Making, 16, 25.
National Institutes of Health. (2018). PARs Dissemination and Implementation Research in Health for R01 (Clinical Trial Optional; PAR-18-007), R21
(Clinical Trial Optional; PAR-17-017), R03 (PAR-16-237). Retrieved July 5, 2018 from [Link]
Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10, 53.
Nilsson, G., Hansson, K., Tiberg, I., & Hallström, I. (2018). How dislocation and professional anxiety influence readiness for change during the
implementation of hospital-based home care for children newly diagnosed with diabetes—An ethnographic analysis of the logic of workplace
change. BMC Health Services Research, 18(1), 61.
Nordmark, S., Zingmark, K., & Lindberg, I. (2016). Process evaluation of discharge planning implementation in healthcare using normalization
process theory. BMC Medical Informatics and Decision Making, 16, 48.
Nutting, P. A., Crabtree, B. F., Miller, W. L., Stewart, E. E., Stange, K. C., & Jaén, C. R. (2010). Journey to the patient-centered medical home: a
qualitative analysis of the experiences of practices in the National Demonstration Project. Annals of Family Medicine, 2010(8 Suppl 1), S45-56,
S92. Erratum in Annals of Family Medicine, 8(4), 369.
Padgett, D. (2016). Qualitative methods in social work research (3rd ed.). Thousand Oaks, CA: Sage.
Padwa, H., Teruya, C., Tran, E., Lovinger, K., Antonini, V. P., Overholt, C. C., & Urada, D. (2016). The implementation of integrated behavioral health
protocols in primary care settings in project care. Journal of Substance Abuse Treatment, 62, 74-83.
Palinkas, L. A. (2014). Qualitative and mixed methods in mental health services and implementation research. Journal of Clinical Child and
Adolescent Psychology, 43(6), 851-861. doi: 10.1080/15374416.2014.910791
Palinkas, L. A., & Aarons, G. A. (2009). A view from the top: Executive and management challenges in a statewide implementation of an evidence-
based practice to reduce child neglect. International Journal of Child Health and Human Development, 2, 47-55.
Palinkas, L. A.., Aarons, G. A., Horwitz, S. M., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed method designs in implementation
research. Administration and Policy in Mental Health, 38, 44-53.
Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015). Purposeful sampling for qualitative data collection
and analysis in mixed method implementation research. Administration and Policy in Mental Health, 42(5), 533-544.
Palinkas, L. A., Schoenwald, S. K., Hoagwood, K., Landsverk, J., Chorpita, B. R., Weisz, J. R., & MacArthur Research Network on Youth Mental
Health. (2008). An ethnographic study of implementation of evidence-based treatments in child mental health: First steps. Psychiatric Services,
59(7), 738-746.
Palinkas, L. A., Spear, S. E., Mendon, S. J., Villamar, J., Valente, T., Chou, C. P., . . . Brown, C. H. (2016). Measuring sustainment of prevention
programs and initiatives. Implementation Science, 11, 95.
Palinkas, L. A., Weisz, J. R., Chorpita, B. R., Levine, B., Garland, A.F., Hoagwood, K. E., & Landsverk, J. (2013). Continued use of evidence-based
treatments after a randomized controlled effectiveness trial: A qualitative study. Psychiatric Services, 64(11), 1110-1118.
Parkinson, S., Eatough, V., Holmes, J., Stapley, E., & Midgley, N. (2016). Framework analysis: a worked example of a study exploring young
people’s experiences of depression. Qualitative Research in Psychology, 13(2), 109-129.
Pelcastre-Villafuerte, B., Ruiz, M., Meneses, S., Amaya, C., Márquez, M., Taboada, A., & Careaga, K. (2014). Community-based health care for
indigenous women in Mexico: A qualitative evaluation. International Journal for Equity in Health, 13(1), 2.
Pope, C., Mays, N., & Popay, J. (2006). How can we synthesize qualitative and quantitative evidence for healthcare policy-makers and managers?
Healthcare Management Forum, 19(1), 27-31.
Powell, B. J., Beidas, R. S., Lewis, C. C., Aarons, G. A., McMillen, J. C., Proctor, E. K., & Mandell, D. S. (2017) Methods to improve the selection
and tailoring of implementation strategies. Journal of Behavioral Health Services Research, 44(2), 177-194.
Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C., . . . York, J. L. (2012). A compilation of strategies for
implementing clinical innovations in health and mental health. Medical Care Research and Review, 69(2), 123-157.
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., . . . Kirchner, J. E. (2015). A refined compilation of
implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10, 21.
Proctor, E. K., Knudsen, K. J., Fedoracivius, N., Hovmand, P., Rosen, A., & Perron, B. (2007). Implementation of evidence-based practice in
community behavioral health: Agency director perspectives. Administration and Policy in Mental Health, 34, 479-488.
Proctor, E. K., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G. A., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation
research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38, 65-76.
Rabin, B. A., Brownson, R., Haire-Joshu, D., Kreuter, M., & Weaver, N. (2008). A glossary for dissemination and implementation research in health.
Journal of Public Health Management and Practice, 14, 117-123.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 25
 REFERENCES

Rabin, B. A., Lewis, C. C., Norton, W. E., Neta, G., Chambers, D., Tobin, J. N., . . . Glasgow, R. E. (2016). Measurement resources for
dissemination and implementation research in health. Implementation Science, 11, 42.
Rapp, C. A., Etzel-Wise, D., Marty, D., Coffman, M., Carlson, L., Asher, D., . . . Holter M. (2010). Barriers to evidence-based practice
implementation: Results of a qualitative study. Community Mental Health Journal, 46(2), 112-118.
Rapp, C. A., Etzel-Wise, D., Marty, D., Coffman, M., Carlson, L., Asher, D., . . . Whitley, R. (2008). Evidence-based practice implementation
strategies: Results of a qualitative study. Community Mental Health Journal, 44(3), 213-224.
Reason, P., & Bradbury, H. (2008). Handbook of action research. Thousand Oaks, CA: Sage.
Reger, G. M., Browne, K. C., Campellone, T. R., Simons, C., Kuhn, E., Fortney, J., . . . Reisinger, H. S. (2017). Barriers and facilitators to mobile
application use during PTSD treatment: Clinician adoption of PE coach. Professional Psychology: Research and Practice, 48(6), 510-517.
Reichstadt, J., Depp, C. A., Palinkas, L. A., Folsom, D. P., & Jeste, D. V. (2007). Building blocks of successful aging: A focus group study of older
adults’ perceived contributors to successful aging. American Journal of Geriatric Psychiatry, 15(3), 194-201.
Ritchie, J., & Lewis, J. (2003). Qualitative research practice: A guide for social science students & researchers. Thousand Oaks, CA: Sage.
Robert Wood Johnson Foundation. (2018). Qualitative Research Guidelines Project. Retrieved from [Link]
Rychetnik, L., Frommer, M., Hawe, P., & Shiell, A. (2002). Criteria for evaluating evidence on public health interventions. Journal of Epidemiology
and Community Health, 56, 119-127.
Rycroft-Malone, J., McCormack, B., Hutchinson, A. M., DeCorby, K., Bucknall, T. K., Kent, B., . . . Wilson V. (2012). Realist synthesis: illustrating the
method for implementation research. Implementation Science, 7(1), 33.
Saldana, L. (2014). The stages of implementation completion for evidence-based practice: Protocol for a mixed methods study. Implementation
Science, 9, 43.
Sandelowski, M., Trimble, F., Woodard, E. K., & Barroso, J. (2006). From synthesis to script: Transforming qualitative research findings for use in
practice. Qualitative Health Research, 16(10), 1350-1370.
Sandelowski, M., & Leeman, J. (2012). Writing usable qualitative health research findings. Qualitative Health Research, 22(10), 1404-1413.
Sax, H., Clack, L., Touveneau, S., Jantarad, F. da L., Pittet, D., Zingg, W., & PROHIBIT study group. (2013). Implementation of infection control best
practice in intensive care units throughout Europe: A mixed-method evaluation study. Implementation Science, 8, 24.
Schierhout, G., Hains, J., Si, D., Kennedy, C., Cox, R., Kwedza, R., . . . Bailie, R. (2013). Evaluating the effectiveness of a multifaceted, multilevel
continuous quality improvement program in primary health care: Developing a realist theory of change. Implementation Science, 8(1), 119.
Scrimshaw, S. C., Carballo, M., Ramos, L., & Blair, B. A. (1991). The AIDS Rapid Anthropological Assessment Procedures: A tool for health
education planning and evaluation. Health Education Quarterly, 18(1), 111-123.
Scrimshaw, S. C. M., & Hurtado, E. (1987). Rapid assessment procedures for nutrition and primary health care: Anthropological approaches to
improving programme effectiveness. Los Angeles, CA: UCLA Latin American Center.
Shea, C. M., Teal, R., Haynes-Maslow, L., McIntyre, M., Weiner, B. J., Wheeler, S. B., . . . Shea, T. C. (2014). Assessing the feasibility of a virtual
tumor board program: a case study. Journal of Healthcare Management, 59(3), 177-193.
Smithson, J. (2000). Using and analysing focus groups: Limitations and possibilities. International Journal of Social Research Methodology, 3(2), 103-119.
Sommerbakk, R., Haugen, D. F., Tjora, A., Kaasa, S., & Hjermstad, M. J. (2016). Barriers to and facilitators for implementing quality improvements
in palliative care—Results for a qualitative interview study in Norway. BMC Palliative Care, 15, 61.
Southam-Gerow, M. A., & Dorsey, S. (2014). Qualitative and mixed methods research in dissemination and implementation science: Introduction to
the special issue. Journal of Clinical Child & Adolescent Psychology, 43(6), 845-850.
Srivistava, A., & Thomson, S. B. (2009) Framework analysis: A qualitative methodology for applied policy research. Journal of Administration and
Governance, 4, 72-79.
Stewart, E. E., Nutting, P. A., Crabtree, B. F., Stange, K. C., Miller, W. L., & Jaén, C. R. (2010) Implementing the patient-centered medical home:
Observation and description of the national demonstration project. Annals of Family Medicine, 8(Suppl 1), S21-32; S92. Erratum in Annals of Family
Medicine, 8(4), 369.
Strauss, A., & Corbin, J. M. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory. Thousand Oaks,
CA: Sage.
Swain, K., Whitley, R., McHugo, G. J., & Drake, R. E. (2009). The sustainability of evidence-based practices in routine mental health agencies.
Community Mental Health Journal, 46(2), 119-129.
Tabak, R. G., Chambers, D. A., Hook, M., & Brownson, R. C. (2018). The conceptual basis for dissemination and implementation research:
Lessons from existing models and frameworks. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and implementation
research in health: Translating science into practice (pp. 73-88). New York, NY: Oxford University Press.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 26
 REFERENCES

Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. (2012) Bridging research and practice: Models for dissemination and
implementation research. American Journal on Preventative Medicine, 43, 337–350.
Tausch, A. P., & Menold, N. (2016). Methodological aspects of focus groups in health research: Results of qualitative interviews with focus group
moderators. Global Qualitative Nursing Research, 3, 1–12.
Thomas, S., Steven, I., Browning, C., Dickens, E., Eckermann, L., Carey, L., & Pollard, S. (1992). Focus groups in health research: A methodological
review. Annual Review of Health Social Science, 2(1), 7-20.
Thorne, S. (2017). Metasynthetic madness: What kind of monster have we created? Qualitative Health Research, 27(1), 2-12.
Tonkin-Crine, S., Anthierens, S., Hood, K., Yardley, L., Cals, J. W., Francis, N.A., . . . Little, P. (2016). Discrepancies between qualitative and
quantitative evaluation of randomised controlled trial results: Achieving clarity through mixed methods triangulation. Implementation Science, 11(1) 66.
Tracy, S. J. (2010). Qualitative quality: Eight ‘big tent’ criteria for excellent qualitative research. Qualitative Inquiry, 16(10), 837-851.
Tunis, S. R., Stryer, D. B., & Clancy, C. M. (2003). Practical clinical trials: Increasing the value of clinical research for decision making in clinical and
health policy. JAMA, 290(12), 1624-1632.
Turner, S., Morris, S., Sheringham, J., Hudson, E., & Fulop, N. J. (2016). Study protocol: Decisions in health care to introduce or diffuse innovations
using evidence (DECIDE). Implementation Science, 1, 48.
Utarini, A., Winkvist, A., & Pelto, G. H. (2001). Appraising studies in health using rapid assessment procedures (RAP): Eleven critical criteria,
Human Organization 60(4), 390-400.
van Manen, M. (2015). Researching lived experience: Human science for an action sensitive pedagogy (2nd ed). New York, NY: Routledge.
Varsi, C., Ekstedt, M., Gammon, D., & Ruland, C. M. (2015). Using the Consolidated Framework for Implementation Research to identify barriers
and facilitators for the implementation of an internet-based patient-provider communication service in five settings: A qualitative study. Journal of
Medical Internet Research, 17(11), e262.
Vogel, A. L., Stipelman, B. A., Hall, K. L., Nebeling, L., Stokols, D., & Spruijt-Metz, D. (2014). Pioneering the transdisciplinary team science
approach: Lessons learned from National Cancer Institute grantees. Journal of Translational Medicine & Epidemiology, 2(2).
Weiner, B. J., Amick, H. R., Lund, J. L., Lee, S. Y., & Hoff, T. J. (2011). Use of qualitative methods in published health services and management
research: A 10-year review. Medical Care Research and Review, 68(1), 3-33.
Whitley, R., Gingerich, S., Lutz, W. J., & Mueser, K. T. (2009). Implementing the illness management and recovery program in community mental
health settings: Facilitators and barriers. Psychiatric Services 60, 202-209.
Woltmann E. M., Whitley, R., McHugo, G. J., Brunette, M., Torrey, W. C., Coots, L., Lynde, D., & Drake, R. E. (2008). The role of staff turnover in the
implementation of evidence-based practices in health care. Psychiatric Services, 59, 732-737.
Wozniak, L., Soprovich, A., Rees, S., Al Sayah, F., Majumdar, S. R., & Johnson, J. A. (2015). Contextualizing the effectiveness of a collaborative
care model for primary care patients with diabetes and depression (Teamcare): A qualitative assessment using RE-AIM. Canadian Journal of
Diabetes, 39(Suppl 3), S83-91.
Zazzali, J. L., Sherbourne, C., Hoagwood, K. E., Greene, D., Bigley, M. F., & Sexton, T. L. (2008). The adoption and implementation of an evidence
based practice in child and family mental health services organizations: A pilot study of functional family therapy in New York State. Administration
and Policy in Mental Health, 35, 38-49.
Zhang, W., & Watanabe-Galloway, S. (2014). Using mixed methods effectively in prevention science: Designs, procedures, and examples.
Prevention Science: The Official Journal of the Society for Prevention Research, 15(5), 654-662.

Q UA L I TAT I V E M E T H O D S I N I M PL EM EN TAT I O N S C I EN C E 27