DOI: 10.1111/nae2.
28
REVIEW
State of Reviews
Systematic reviews: Brief overview of methods, limitations,
and resources
Jacqueline K. Owens
Correspondence
Jacqueline K. Owens.
Email: Jowens2@[Link]
INTRODUCTION CONDUCTING A SYSTEMATIC REVIEW
Systematic reviews are a valuable resource for nurses in academia Sutton and colleagues9 noted that while all review types follow a sys-
and practice.1‐3 Well done systematic reviews, which include but tem of inquiry, and are thus systematic in nature, not all reviews are
are not limited to meta‐analyses, offer an efficient way to evaluate systematic reviews. To decrease confusion between systematic
large amounts of information for decision‐makers in areas of reviews and other types of reviews, it is important to follow the exist-
research, policy, and patient care. Systematic reviews can help us ing well‐established published methodology. Fully describing the pro-
know what we know about a topic, and what is not yet known, cess used for the review and using consistent terminology when writing
often to a greater extent than the findings of a single study.4 The a review or referring to review types is also helpful. An effort to clearly
process is comprehensive enough to establish consistency and describe review types that may be within the same family, such as
generalizability of research findings across settings and pop- systematic review and meta‐analysis described above, can minimize
ulations.3 A meta‐analysis is a type of systematic review that sta- conceptual overlap.9 These important steps in any type of review assure
tistically combines results of multiple quantitative studies. consumers that it was conducted with thoughtful intent and rigor.
According to Grant and Booth,5 “although many systematic reviews There are number of checklists available to guide the systematic
present their results without statistically combining data in this review process that range from a few steps to many; the best choice
way, a good systematic review is essential to a meta‐analysis of the is often guided by level of expertise and the need for detailed in-
literature” (p. 98). struction. One example is the well‐known one that builds on the
Preferred Reporting Items for Systematic reviews and Meta‐
Analyses literature search extension (PRISMA‐S).8,10‐12 The last
PURPOSE AND BACKGROUND part of the process is preparation of the final report, ideally with the
inclusion of external peer review that evaluates both the methodol-
Systematic reviews have grown in numbers since they first emerged ogy and dissemination for transparency.1
4,6
in the field of medicine in the late 1970s. Other types of scholarly The rationale to perform a systematic review and the methodol-
reviews are often mistitled as a systematic review.4 Conceptual ogy to be used should be determined before the review process be-
overlap and muddiness exist among the many types of reviews gins.1 A systematic review typically begins with a well‐defined guiding
6 7
found in the literature. According to research by Martinic, a research statement or question. This is often a question that includes
complicating factor is the lack of consensus for a standard definition the PICO(T) elements of Population, Intervention, Comparison,
for systematic reviews. However, systematic reviews synthesize the Outcome, and sometimes Time frame.8,13 At this early point, it is
findings of multiple primary research reports on a topic with an advisable to register the review in a database, such as the International
4 1
effort to limit bias and random error. Baker noted three criteria Prospective Register of Systematic Reviews (PROSPERO, https://
essential to a systematic review: it is explicit, rigorous, and repro- [Link]/prospero/) as this allows updates about review
ducible. These criteria have withstood the test of time, appearing in status to provide transparency and avoid duplication.13,14
earlier literature describing systematic review methods.4,8 This Once the PICOT question is developed, inclusion and exclusion
article offers a brief overview of the methodology for systematic criteria for inclusion of studies can be determined. An exhaustive
reviews, discussion of potential limitations, and resources for au- search is completed with the following steps, with attention to
-
thors and editors. transparency throughout the process:
Nurse Author & Editor. 2021;31:69–72. [Link]/journal/nae2 © 2021 John Wiley & Sons Ltd. 69
17504910, 2021, 3-4, Downloaded from [Link] by Cochrane Colombia, Wiley Online Library on [28/04/2025]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
70
- OWENS
TABLE 1 Resources for systematic review
Resource and authors Helpful information
A typology of reviews: An analysis of 14 review types and associated � Discusses 14 main review types to discriminate unique attributes
methodologies5
� Describes and summarizes methods for systematic reviews, including
meta‐analysis and qualitative systematic review
� Lists strengths and weaknesses and examples of systematic reviews
Charting the landscape of graphical displays for meta‐analysis and � Discussion of visuals used to disseminate data from systematic reviews
and meta‐analyses
systematic reviews: A comprehensive review, taxonomy, and
feature analysis16 � Includes a taxonomy of graphical displays for meta‐analysis (e.g., forest
plot like, funnel plot like, trends, effect size distribution, more than one
effect size)
� Discusses historical development of data visualization and functionality
features of plots
How to read and understand and use systematic reviews and meta‐ � Reviews methodology, significance, and limitations of systematic
analyses 2 reviews
� Includes details about effect size measures for dichotomous and
continuous variables; calculation of mean effect size; methods to
examine publication bias; presentation of results
� Discusses multiple potential methodological problems
PRISMA‐S: An extension to the PRISMA statement for reporting � Includes the PRISMA‐S checklist with detailed information about use;
11 explanations and examples; and definition of terms in a glossary
literature searches in systematic reviews
How to do a systematic review: A best practice guide for conducting � Describes core standards and principles for systematic review and
and reporting narrative reviews, meta‐analyses, and meta‐ frequently encountered problems
17
syntheses � Differentiates between literature reviews and reviewing literature
� Addresses quantitative and qualitative research synthesis
Preferred reporting items for systematic review and meta‐analysis � Reviews PRISMA checklist items necessary to report to eliminate bias
12 when disseminating findings from systematic reviews and meta‐
protocols (PRISMA‐P) 2015: Elaboration and explanation
analyses
� Includes evidence support and examples for each of the 17 items and
related subitems
Meeting the review family: Exploring review types and associated � Characterizes existing review types based on research study by authors
9
information retrieval requirements � Defines systematic review family (and other review types) with specific
review types and subtypes
Methods guide for effectiveness and comparative effectiveness � Includes links to AHRQ and other publications about many topics
18 related to systematic review's, such as comparing interventions;
reviews
selecting topics, and assessing levels of evidence and risk of bias
� Includes resources about assessing harms
� Offers information about how to find grey literature evidence
Training modules for the systematic reviews methods guide19 � Provides a series of PowerPoint presentations designed for use by
faculty and clinicians to familiarize novice researchers with the meth-
odology of systematic review
� Includes presentation slides and quiz slides
Cochrane handbook for systematic reviews of interventions20 � Describes basic methodology of systematic reviews and is considered
the official guide for Cochrane reviews
� Includes links to the Cochrane Handbook for Systematic Reviews of In-
terventions with online book option; ability to download some PDFs; or
link to purchase the full book
17504910, 2021, 3-4, Downloaded from [Link] by Cochrane Colombia, Wiley Online Library on [28/04/2025]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
SYSTEMATIC REVIEWS: BRIEF OVERVIEW OF METHODS, LIMITATIONS, AND RESOURCES
- 71
T A B L E 1 (Continued)
Resource and authors Helpful information
Cochrane Methods Groups21 � Provides information about Cochrane Methods Groups for potential
participants to discuss various features of systematic reviews in a global
community
Abbreviations: AHRQ, Agency for Healthcare Research and Quality; PRISMA‐S, Preferred Reporting Items for Systematic reviews and Meta‐Analyses
literature search extension.
� Predetermine relevant databases and specific terms to search in amount of research available. The use of electronic data manage-
keywords, title, and abstract fields of each database; ment products to organize and analyze the information retrieved
� Search keywords and index terms from this literature search can potentially increase the accuracy of the review and add effi-
across all appropriate databases; ciency to the process. These reviews are often completed by a
� Expand the search by also searching the reference list in the team. Such a team should include at minimum one author who has
studies obtained in the previous steps.1 methodological expertise. A health sciences librarian can do more
than assist with the search; many also have experience using
The second and third steps are sometimes called snowballing and products for electronic data management and analysis. Systematic
reverse snowballing6; this builds the sample of potentially relevant reviews may be appropriate for graduate level students with
studies and assures the exhaustive nature of the search. This process consideration of the complexity inherent in the process and inclu-
should be continued until repetition search results is evident within sion of persons with methodological expertise on the dissertation or
various sources. In addition to published literature, it is important to project committee.
include unpublished sources (grey literature). A health sciences
librarian can provide invaluable support to determine a comprehen-
sive and relevant search strategy for evidence.1 RESOURCES FOR AUTHORS AND EDITORS
Use of two screeners is strongly recommended.1,2,6,10 Each
person reviews the results independently, followed by comparison, As systematic reviews have been undertaken for about 50 years,
with discussion to reach consensus as needed, and final conclusions. there are many resources available to authors and editors. Table 1
Another recommended technique is a two‐step process for screening lists both recent and time‐honored resources that offer detailed
that first considers the abstract of each study in the sample, followed information related to performing or evaluating a systematic review.
by review of the full manuscript, including listed references.1 These resources can be helpful with challenges unique to the
systematic review methodology, but also are useful to determine
what is, and what is not, a systematic review.
CONSIDERATIONS
Systematic reviews use a retrospective, observational research CONCLUSION
design, and as such are subject to systematic and random error.4 A
recent article by Doleman et al.13 offers a detailed explanation about Siddaway16 noted that, “The best reviews synthesize studies to draw
these limitations. In‐depth discussion about the limitations is beyond broad theoretical conclusions about what the literature means,
the scope of this article; however, it is important to mention potential linking theory to evidence and evidence to theory” (p. 747). To that
limitations of systematic reviews that should be considered by both end, high quality systematic reviews are explicit, rigorous, and
authors and editors. These may include risks of bias, such as selection reproducible. It is these three criteria that should guide authors
bias, inadequate blinding, attrition bias, and selective outcome seeking to write a systematic review or editors seeking to publish
reporting; inconsistency that includes clinical or statistical hetero- one. The research question should be explicitly stated, with clearly
geneity; and imprecision that can lead to Type I and Type II errors. defined inclusion and exclusion criteria. The search should be
Another important consideration is publication bias. It is often pref- exhaustive in nature and include multiple levels of review supported
erable to publish studies which have demonstrated statistically by further searching of references listed in the initial results. Finally,
significant results even though studies with results that do not dissemination of the results should include sufficient description of
demonstrate statistical significance may be very clinically significant, the methodology so as to demonstrate the required steps to repro-
and thus important to the findings of a systematic review, especially duce the study. Careful attention to these established criteria, easily
for clinical providers.6,13,15 found in the literature, has the potential to assure the rigor of a
A systematic review is a complex undertaking that may involve systematic review and avoid conceptual overlap with other review
hundreds, and possibly several thousand articles, depending on the types.
17504910, 2021, 3-4, Downloaded from [Link] by Cochrane Colombia, Wiley Online Library on [28/04/2025]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
72
- OWENS
O RC ID 15. Carpenter R, Waldrop J, Carter‐Templeton H. Statistical, practical
Jacqueline K. Owens [Link] and clinical significance and Doctor of Nursing Practice projects.
Nurse Author Ed. 2021;1–4. doi:10.1111/nae2.27
16. Kossmeier M, Tran US, Voracke M. Charting the landscape of
REFERENCES graphical displays for meta‐analysis and systematic reviews: a
1. Baker KA, Weeks SM. An overview of systematic review. J Perianesth comprehensive review, taxonomy, and feature analysis. BMC Med
Nurs. 2014;29(6):454‐458. doi:10.1016/[Link].2014.07.002 Res Methodol. 2020;20(1):26. doi:10.1186/s12874‐020‐0911‐9
2. Leucht S, Kissling W, Davis JM. How to read and understand and use 17. Siddaway AP, Wood AM, Hedges LV. How to do a systematic review:
systematic reviews and meta‐analyses. Acta Psychiatr Scand. 2009; a best practice guide for conducting and reporting narrative reviews,
119:443‐450. doi:10.1111/j.1600‐0447.2009.01388.x meta‐analyses, and meta‐syntheses. Annu Rev Psychol. 2019;70:
3. Mulrow CD. Rationale for systematic reviews. BMJ. 1994;309(6954): 747‐770. doi:10.1146/annurev‐psych‐010418‐102803
597‐599. doi:10.1136/bmj.309.6954.597 18. Methods guide for effectiveness and comparative effectiveness re-
4. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of views. Agency for Healthcare Research and Quality (AHRQ). February
best evidence for clinical decisions. Ann Intern Med. 1997;126(5): 26, 2015. Accessed September 29, 2021. [Link]
376‐380. doi:10.7326/0003‐4819‐126‐5‐199703010‐00006 [Link]/products/cer‐methods‐guide/overview/
5. Grant MJ, Booth A. A typology of reviews: an analysis of 14 review 19. Training modules for the systematic reviews methods guide. Agency
types and associated methodologies. Health Inf Libr J. 2009;12(2): for Healthcare Research and Quality (AHRQ). December 30, 2019.
91‐108. doi:10.1111/j.1471‐1842.2009.00848.x Accessed September 29, 2021. [Link]
6. Baird R. Systematic reviews and meta‐analytic techniques. Semin products/cer‐methods‐guide/presentations/
Pediatr Surg. 2018;27(6):338‐344. doi:10.1053/[Link].2018. 20. Cochrane handbook for systematic reviews of interventions. 2021.
10.009 Accessed September 29, 2021. [Link]
7. Martinic MK, Pieper D, Glatt A, Puljak L. Definition of a systematic handbook
review used in overviews of systematic reviews, meta‐epidemiological 21. Cochrane Methods Groups. 2021. Accessed September 29, 2021.
studies and textbooks. BMC Med Res Methodol. 2019;19(1):203. doi: [Link]
10.1186/s12874‐019‐0855‐0 methods‐groups
8. Lockwood C, Oh EG. Systematic reviews: guidelines, tools and
checklists for authors. Nurs Health Sci. 2017;19(3):273‐277.
doi:10.1111/j.1471‐1842.2009.00848.x A U T HO R B I O GR A PH Y
9. Sutton A, Clowes M, Preston L, Booth A. Meeting the review family:
exploring review types and associated information retrieval re- Jacqueline K. Owens, PhD, RN, CNE is a professor of nursing at
quirements. Health Inf Libr J. 2019;36(3):202‐222. doi:10.1111/
Ashland University Schar College of Nursing and Health Sciences
hir.12276
10. Muka T, Glissic M, Milic J, et al. A 24‐step guide on how to design,
in Ashland, OH. She is Editor‐in‐Chief of OJIN: The Online Journal
conduct, and successfully publish a systematic review and meta‐ of Issues in Nursing. Prior to this position, she served as Associate
analysis in medical research. Eur J Epidemiol. 2019;35(1):49‐60. Editor from 2007 to 2011 and became the journal's first Editorial
doi:10.1007/s10654‐019‐00576‐5 Assistant in 2003. Dr. Owens has presented many writing
11. Rethlefsen ML, Kirtley S, Waffenschmidt S, et al. PRISMA‐S: an
workshops for nurses at regional research days throughout
extension to the PRISMA statement for reporting literature
searches in systematic reviews. Syst Rev. 2021;10(1):39. doi:10. Ohio. Particular areas of interest are online scholarly
1186/s13643‐020‐01542‐z publishing, predatory publishing practices, and novice author
12. Shamseer L, Moher D, Clarke M, et al. Preferred reporting items for publication. She is also a member of the Authors‐in‐Residence
systematic review and meta‐analysis protocols (PRISMA‐P) 2015:
for Nurse Author & Editor.
elaboration and explanation. BMJ. 2015;350:7647. doi:10.1136/bmj.
g7647
13. Doleman B, Mathiesen O, Jakobsen JC, et al. Methodologies for
systematic reviews with meta‐analysis of randomised clinical trials
in pain, anaesthesia, and perioperative medicine. Br J Anaesth.
2021;126(4):903‐911. doi:10.1016/[Link].2021.01.004 How to cite this article: Owens JK. Systematic reviews: brief
14. Allers K, Hoffmann F, Mathes T, Pieper D. Systematic reviews with
overview of methods, limitations, and resources. Nurse Author
published protocols compared to those without: more effort, older
search. J Clin Epidemiol. 2018;95:102‐110. doi:10.1016/[Link]. Ed. 2021;31(3‐4):69‐72. [Link]
2017.12.005