A brief on Systematic Reviews

( Adapted from Briner and Rousseau,2011)

Systematic reviews are fundamental to
evidence-based practice. As such, much is
written about them in other fields (Gough
et al., 2003; Soukup, 2000; Tranfield,
Denyer, & Smart, 2003). Experience indicates
that it is impossible to fully engage
in evidence-based practice without them:
Such reviews provide one of the four
sources of information required when making
evidence-based decisions. A systematic
review’s purpose would be to search for,
collect, critically appraise, and pull together
research evidence relevant to the causes
and possible solutions to the research problem.

Systematic reviews are literature reviews

that adhere closely to a set of scientific
methods that explicitly aim to limit systematic
error (bias), mainly by attempting
to identify, appraise and synthesize all
relevant studies (of whatever design) in
order to answer a particular question (or
set of questions). In carrying out this task
they set out their methods in advance,
and in detail, as one would for any piece
of social research.
Systematic reviews are, essentially, a
way of analyzing existing research using
explicit and replicable methods, allowing
conclusions to be drawn about what is
known and what is not known in relation
to the review question (and within the
limitations of the method). Similar, but not
identical to the traditional meta-analysis,
systematic reviews are studies of studies.
Meta-analyses are a type of systematic
review but one that uses only quantitative
data and statistical synthesis and focuses
on a question repeatedly addressed in the
same way by researchers rather than a
practice question or problem. As with meta analyses,
systematic reviews are conducted
out of recognition that single empirical
studies, although useful and sometimes
informative, should not be emphasized
because their biases and limitations cannot
be fully accounted for. Looking at all
relevant studies, systematically gathered,
constitutes more reliable evidence.
Thus, in the context of evidence based
practice, neither traditional literature
reviews nor meta-analyses are especially
useful. First, traditional literature reviews
are open to many forms of bias. For
example, reviewers do not make clear how
they have selected the studies they have
included, do not critically appraise them
in an explicit or systematic way, and do
not usually pull them together or synthesize
findings across studies. Second, traditional
reviews do not usually focus on
a specific research, practice question, or
problem. It is this latter point that also
differentiates a systematic review from the
quantitative meta-analysis used traditionally
in I–O psychology. The process of
making evidence-based decisions requires
more focused and tailored reviews of evidence
where both a practice question or
problem and the conditions to which the
evidence might be applied are taken into
account. For example, if the study is on “high absence”
of employees, a systematic review
would attempt to find evidence about the
relative effectiveness of different forms of
absence management interventions given
the current and desired absence rates and
taking into account as much as possible
aspects of the context such as the type of
employees involved, the sector, and the
existing work arrangements and absence
policies. In the context of evidence-based
practice, systematic reviews can take forms
akin to phenomena-oriented evidence or
practice-oriented evidence, depending on
the review questions and their intended
use as well as the kinds of research available.
In evidence-based fields, an important
result of systematic reviews is guidelines for
practice.
Systematic reviews can be useful for
purely academic research purposes too.
We may, for example, be interested in
collecting all available evidence about
absence-management interventions to provide
a more general overview about what
is known, and not known, about the efficacy
of such interventions. In this respect,
a systematic review might differ from the
traditional meta-analysis in that it would
also consider qualitative information and
descriptions, not being limited to effect
sizes alone depending on the review question
(Rousseau, Manning, & Denyer, 2008).
All systematic reviews follow a process of
clearly specified stages. One example of this
process (adapted from Petticrew & Roberts,
2006) contains seven stages.
1. Identify and clearly define the question
the review will address. The question
needs to be sufficiently specific so
that it is clear, in principle, what types
of data would be relevant. Aspects of
the context (e.g., population, sector,
and organizational type), the interventions
(what qualifies as a relevant
intervention?), the mechanisms linking
intervention to outcomes (e.g.,
processes, mediators, and moderators),
and the outcomes themselves
(which data are the outcomes of interest)
are also clearly specified.
16 R.B. Briner and D.M. Rousseau
2. Determine the types of studies and
data that will answer the question.
The criteria used to decide which
studies will be selected or excluded
from the review are identified. For
example, a review that addresses a
causal question might exclude studies
with cross-sectional designs. The aim
is to increase the chances that all
relevant studies are included and all
those that are irrelevant are excluded.
3. Search the literature to locate relevant
studies. A clear search strategy is used
specifying, for example, key words,
the databases to be searched, and
how, and whether unpublished data
will be found and included.
4. Sift through all the retrieved studies to
identify those that meet the inclusion
criteria (and need to be examined
further) and those that do not and
should be excluded. Each study is
examined, usually by two review
team members, and checked against
the inclusion and exclusion criteria.
Where agreement is not possible, a
third reviewer assesses the study. At
this stage, it is not uncommon to find
that only a fraction of the initial pool
of studies can be included.
5. Critically appraise the studies by
assessing the study quality determined
in relation to the review question.
The quality of each study is
critically appraised or evaluated in
relation to the review question as
research quality can only be judged in
relation to the question. Even where
studies meet the inclusion criteria,
they are still likely to vary in terms
of quality. Assessing quality allows
the review conclusions to clearly state
how many of the studies included
were, for example, of very high,
medium, and low quality.
6. Synthesize the findings from the studies.
A key part of any systematic
review is the pulling together of findings
from across the studies to represent
what is known and not known.
Synthesis may be quantitative, qualitative,
or both. Review findings are
often described in terms of the overall
number of studies found, the quality
profile of this group of studies, and
the number of studies that obtained
particular results.
7. Disseminate the review findings. A
full report of a systematic review can
be quite large. In addition, shorter
journal article length versions or even
shorter summaries may be produced.
Dissemination is often planned at the
outset of a systematic review given
the aim is often to inform practice.
The underlying logic of systematic
review is similar to that of many psychological
research methods and is similar
to meta-analyses. Meta-analysis’ exclusive
use of quantitative data and statistical rather
than other forms of synthesis sets it apart
from a systematic review, which often uses
different types of data and different forms
of synthesis. In addition, meta-analyses
can only address questions that have been
addressed many times in more or less the
same way by researchers (e.g., the correlation
between job satisfaction and performance)
rather than questions that arise
from practice problems, where an array of
data types may be required to formulate an
answer.