1

SYSTEMATIC REVIEWS AND META-ANALYSES

Systematic review is a comprehensive review of primary studies addressing a clearly

formulated question using systematic and explicit methods to identify, select and critically
appraise relevant research. Meta-analyses are the mathematical synthesis of the primary
studies in a systematic review to give a pooled estimate of effect size.

ADVANTAGE OF SYSTEMATIC REVIEWS AND META-ANALYSES

• Explicit methods are used in identifying and excluding studies based on strict inclusions
and exclusions criteria. This limits bias.
• It increases the power of the study by combining different studies, thus increasing the
probability of finding a true significant effect and reducing type II error, i.e. false
negative. (This is mainly through a meta-analysis, i.e. quantitative systematic review)
• Large amounts of information can be assimilated quickly by healthcare providers,
researchers and policymakers.
• Prevents or reduces delay between research discoveries and implementation of effective
diagnostic and therapeutic strategies.
• Results of different studies can be formally compared to establish generalisability of
findings and consistency of results.
• Reasons for heterogeneity (inconsistencies) can be identified and new hypothesis can
be generated for different subgroups.
• Allows for more objective appraisal of evidence and a more reliable and accurate result
due to the methods used.
• Promising research questions to be addressed in future studies may be generated, and
the sample size needed in future studies may be calculated accurately.

DESIGN

1. FORMULATE AN IMPORTANT AND RELEVANT CLINICAL

QUESTION
A clearly formulated, primary question is central to a high-quality meta-analysis. This
question is based on PICO (Patient type, Intervention used, Comparison and Outcome
measured)

2. SELECTION BIAS
Publication Bias: Studies with positive results are more likely to be published.
Language Bias: Studies with positive results are more likely to be published in English
language journals.
Indexing Bias: Studies with significant results are more likely to be published in a
journal indexed in a literature database, e.g. Medline.

2
 Inclusion Bias: Knowledge of the results of the studies may result in differential
inclusion of studies.
Multiple Publication Bias (Citation Bias): Studies with significant results are more
likely to be cited and may be published in multiple journals.

Minimising selection bias thorough search of databases and other sources to find relevant
studies. This is done after setting inclusion and exclusion criteria for studies.

CHECKLIST OF DATA SOURCES FOR SYSTEMATIC REVIEW:

• Medline Database, EMBASE, PsychINFO, CINAHL
• Cochrane Controlled Clinical Trials Register
• Foreign language literature
• ‘Grey literature’ (theses, internal reports, non-peer reviewed journals, pharmaceutical
industry files)
• References and cross-references
• Other unpublished data obtained by contacting experts, pharmaceutical companies
• Raw data from unpublished trials
• Hand-searching

It is important to use ‘grey literature’ (i.e. unpublished and difficult to retrieve studies) in a
meta-analysis. Exclusion of these studies introduces the publication bias. Also possible is the
‘file drawer’ effect (negative studies that are filed away in a drawer). The exclusion of the so-
called grey literature from meta-analysis has been reported to overestimate the treatment effect
by 15%.

Once all potential studies have been identified, the next step is to set up inclusion criteria for
the study. Two reviewers using defined criteria to select studies independently can improve the
reliability of the selection process.

PUBLICATION BIAS AND FUNNEL PLOTS

A funnel plot is a graphical way of identifying whether or not publication bias exists.

If publication bias is not present, you would expect your funnel plot to be roughly symmetrical,
as in the example below: This is because both positive and negative studies would be expected
to be included. The ones at the top indicate those with a smaller standard error due to large
sample size and those lower the opposite.

3
Figure 17. Funnel Plot : The above plot shows no publication bias

As the studies become less precise (i.e. higher standard error), the results (given here as a log
odds ratio) of the studies are expected to be more variable, scattered to both sides of the more
precise larger studies.

When the studies are plotted onto a funnel plot, it may not be symmetrical and may not
resemble an inverted funnel. This may be due to publication bias; however other factors may
also lead to an asymmetrical plot.

Figure 18. Funnel Plot: The above graph shows a publication bias

Here, it appears that the smaller, less precise studies are all much more positive than the larger,
more precise studies, and there are no smaller studies to the left (negative) side of the graph.
This represents a clear example of publication bias.

3. METHODOLOGICAL QUALITY AND WEIGHTING OF STUDIES

(This is usually based on the PRISMA Statement.)

PRISMA – Preferred Reporting Items for Systematic reviews and Meta Analyses

4
A systematic reviewer draws up a list of criteria, both generic (common to all research studies)
and particular (specific to the field) aspects of quality against which to judge each trial.

Usually trials are weighted on the following

• Allocation concealment
• Randomisation
• Blinding
• Intention to Treat analysis

4. INTERPRETATION OF RESULTS

3. QUANTITATIVE STUDIES

Systematic Review and Meta Analysis

Identify all relevant published Effect Size
Line of No effect
and unpublished evidence

Select studies
or reports for inclusion

Assess the quality of each

study or report

Synthesise the findings from

individual studies or reports
in an unbiased way

Interpret the findings and

present a balanced and impartial
summary of the findings Pooled Effect Size
Confidence Intervals

40

Figure 19. Interpreting a meta-analysis

5. HETEROGENEITY
Heterogeneity is the degree to which the trials included in the analysis are incompatible with
each other. Heterogeneity is of three types:
1. Clinical Heterogeneity: The type of heterogeneity that is introduced due to
clinical differences in populations included in the study, e.g. different diagnoses,
different interventions etc.
2. Statistical Heterogeneity: Due to differences in statistical analysis and significant
differences in results.

5
 3. Methodological heterogeneity: Due to differences in study design.
Statistical heterogeneity can be detected by statistical methods whilst clinical and
methodological heterogeneity is usually detected by looking at the design methods and
inclusion criteria.

Figure 20. Example of Heterogeneity

DETECTING HETEROGENEITY

• Eyeballing a forest plot: By visually inspecting the forest plot, one estimates whether the
confidence intervals of the studies overlap, i.e. can one draw a straight line that would
include the confidence intervals of all the studies. One could reasonably exclude
heterogeneity this way. Thus, in the above figure, heterogeneity exists as one is not able
to draw a straight line through the confidence intervals of all the studies.

6
 • Galbraith Plot

Figure 21

A Galbraith plot is a diagrammatic method for investigating heterogeneity in meta-analyses.

The ratio of the log odds ratio to its standard error for each study is plotted against the reciprocal
of the standard error. If there is no statistically significant heterogeneity, then the majority
(95%) of the plotted points should lie within a band two units above and below the line of the
overall log odds ratio.

More precise results appear to the right of the plotted figure. The reasons for trials falling
outside the lines should be investigated (Czech Republic, Netherlands, Germany, France,
Greece, Norway and Ireland). However, just by chance 5% of trials will fall outside this even
in the absence of statistically significant heterogeneity.
Chi-squared test or Q test heterogeneity

7
 3c. Bias

Detecting Heterogeneity in a Meta-analysis

211

Figure 22. Detecting Heterogeneity in Meta-Analysis

A p value of <0.05 ( in some cases <0.1) is taken to suggest there may be underlying
heterogeneity. The result of the test is: 1. a 'chi-squared' statistic, 2. a number called the degrees
of freedom (d.f) (which is usually one less than the number of studies), and 3. ‘p-value’. The
I2 statistic can be calculated from the Cochrane Q test and gives the extent of heterogeneity.

MANAGING HETEROGENEITY

Fixed effects model: A fixed effects analysis makes the assumption that all studies are
estimating a single underlying treatment effect.
Random effects model: This is a statistical model that makes the assumption that each study
is estimating a different treatment effect. If there is significant heterogeneity among the results
of the included studies, random effects models will give wider confidence intervals than fixed
effect models.
Sensitivity analysis: Sensitivity analysis is a technique whereby the robustness of the results is
checked by making certain assumptions or by changing certain parameters within the study.
For example, studies which show significant heterogeneity may be excluded and the effect of
this exclusion on effect size may be investigated
Data transformation: Where continuous data is converted into dichotomous data so that data
is more homogenous

8
Subgroup analyses: Subgroup analyses are meta-analyses on subgroups of the studies
Meta-regression analyses: Meta-regression can formally test whether there is evidence of
different effects in different subgroups of trials. For example, you can use meta-regression to
test whether treatment effects are greater in low quality studies than in high quality studies.