Fixed vs.

Random Effect
Meta-analysis
Abdur Rahman
Lecturer
Statistics Discipline
Khulna University
 PERFORMING A RANDOM-EFFECTS META-ANALYSIS
• In an actual meta-analysis, of course, rather than start with the
population effect and make projections about the observed effects, we
start with the observed effects and try to estimate the population effect.

• In other words our goal is to use the collection of Yi to estimate the

overall mean, 𝜇.

AR, Lecturer, Statistics Discipline, KU 2

 PERFORMING A RANDOM-EFFECTS META-ANALYSIS
• In order to obtain the most precise estimate of the overall mean (to
minimize the variance) we compute a weighted mean, where the weight
assigned to each study is the inverse of that study’s variance.

• To compute a study’s variance under the random-effects model, we

need to know both the within-study variance and between-studies
variance (𝜏 2 )

AR, Lecturer, Statistics Discipline, KU 3

 Estimating tau-square 2
(𝜏 )
• The parameter 𝜏 2 (tau-squared) is the between-studies variance (the

variance of the effect size parameters across the population of studies).

• In other words, if we somehow knew the true effect size for each study, and

computed the variance of these effect sizes (across an infinite number of

studies), this variance would be 𝜏 2 .

AR, Lecturer, Statistics Discipline, KU 4

 Estimating tau-square 2
(𝜏 )
• One method for estimating 𝜏 2 is the method of moments (or the DerSimonian and
Laird) method, as follows. We compute

𝑄 − 𝑑𝑓
𝑇2 =
𝐶

𝑊𝑖 𝑌 𝑖 2
Where, 𝑄 = 𝑊𝑖 𝑌𝑖2 − ; 𝑑𝑓 = 𝑘 − 1, 𝑘 𝑖𝑠 𝑡ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠𝑡𝑢𝑑𝑖𝑒𝑠
𝑊𝑖

𝑊𝑖2
𝐶 = W𝑖 −
W𝑖

AR, Lecturer, Statistics Discipline, KU 5

 Estimating the mean effect size
• In the fixed-effect analysis each study was weighted by the inverse of its
variance.

• In the random-effects analysis, too, each study will be weighted by the

inverse of its variance.

• The difference is that the variance now includes the original (within studies)
variance plus the estimate of the between-studies variance 𝑇 2 .

AR, Lecturer, Statistics Discipline, KU 6

 Estimating the mean effect size
• Since our goal is to estimate the mean of the distribution, we need to
take account of two sources of variance.
• First, there is within-study error in estimating the effect in each study.
• Second (even if we knew the true mean for each of our studies), there is
variation in the true effects across studies.

Study weights are assigned with the goal of minimizing both sources of
variance

AR, Lecturer, Statistics Discipline, KU 7

 Estimating the mean effect size
• In this course, we use 𝜏 2 to refer to the parameter and 𝑇 2 to refer to the

sample estimate of that parameter.

• To highlight the parallel between the formulas here (random effects) and

those in the previous chapter (fixed effect) we use the same notations but

add an asterisk (*) to represent the random-effects version.

AR, Lecturer, Statistics Discipline, KU 8

 Estimating the mean effect size
Under the random-effects model the weight assigned to each study is
∗ 1
𝑊𝑖 = ∗
𝑉𝑌𝑖
Where 𝑉𝑌∗𝑖 is the within-study variance for study i plus the between-studies variance, 𝑇 2 .
That is,

AR, Lecturer, Statistics Discipline, KU 9

The variance of the summary effect is estimated as the reciprocal of the
sum of the weights

and the estimated standard error of the summary effect is then the square
root of the variance

AR, Lecturer, Statistics Discipline, KU 10

SUMMARY EFFECT

AR, Lecturer, Statistics Discipline, KU 11

Extreme Effect Size in a Large Study or A Small Study
• How will the selection of a model influence the overall effect size?

• In this example Donat is the largest study, and also happens to have the highest effect size.

• Under the fixed-effect model Donat was assigned a large share (39%) of the total weight and
pulled the mean effect up to 0.41.

• By contrast, under the random-effects model Donat was assigned a relatively modest share of the
weight (23%).

• It therefore had less pull on the mean, which was computed as 0.36.

• Similarly, Carroll is one of the smaller studies and happens to have the smallest effect size.

AR, Lecturer, Statistics Discipline, KU 12

Extreme Effect Size in a Large Study or A Small Study
• From fixed effect to random effects, extreme studies will lose

influence if they are large, and will gain influence if they are small.

AR, Lecturer, Statistics Discipline, KU 13

 Confidence interval
• Under the fixed-effect model the only source of uncertainty is the within-study
(sampling or estimation) error.

• Under the random-effects model there is this same source of uncertainty plus
an additional source (between-studies variance).

• It follows that the variance, standard error, and confidence interval for the
summary effect will always be larger (or wider) under the random-effects
model than under the fixed-effect model (unless 𝑇 2 is zero, in which case the
two models are the same).

AR, Lecturer, Statistics Discipline, KU 14

 Confidence interval
• In this example, the standard error is 0.064 for the fixed-effect model,
and 0.105 for the random-effects model

• Under the fixed-effect model the summary effect would also have a
confidence interval with a width of zero, since we know the common
effect precisely (Figure 13.3).

AR, Lecturer, Statistics Discipline, KU 15

 Confidence interval
• By contrast, under the random-effects model the width of the confidence interval would not approach zero (Figure 13.4).

• While we know the effect in each study precisely, these effects have been sampled from a universe of possible effect sizes, and
provide only an estimate of the mean effect.

• Just as the error within a study will approach zero only as the sample size approaches infinity, so too the error of these studies as
an estimate of the mean effect will approach zero only as the number of studies approaches infinity.

AR, Lecturer, Statistics Discipline, KU 16

Under the fixed-effect model the standard error of the summary effect is given by

and this is true whether the sample size is concentrated on one or two studies, or
dispersed across any number of studies

The first term is identical to that for the fixed-effect model and, again, with a large
enough sample size, this term will approach zero.

AR, Lecturer, Statistics Discipline, KU 17

o The first term is identical to that for the fixed-effect model and, again, with a large enough sample
size, this term will approach zero.

o By contrast, the second term (which reflects the between-studies variance) will only approach zero as
the number of studies approaches infinity.

o The standard error of the summary effect and (it follows) the confidence intervals for the summary
effect are wider under the random-effects model than under the fixed-effect model

AR, Lecturer, Statistics Discipline, KU 18

 The null hypothesis

• Under the fixed-effect model the null hypothesis being tested is that

𝐻0 : 𝑡ℎ𝑒𝑟𝑒 𝑖𝑠 𝑧𝑒𝑟𝑜 𝑒𝑓𝑓𝑒𝑐𝑡 𝑖𝑛 𝑒𝑣𝑒𝑟𝑦 𝑠𝑡𝑢𝑑𝑦.

• Under the random-effects model the null hypothesis being tested is

that

H0 : 𝑡ℎ𝑒 𝑚𝑒𝑎𝑛 𝑒𝑓𝑓𝑒𝑐𝑡 𝑖𝑠 𝑧𝑒𝑟𝑜.

AR, Lecturer, Statistics Discipline, KU 19

WHICH MODEL SHOULD WE USE?

AR, Lecturer, Statistics Discipline, KU 20

 Precaution
• The selection of a model must be based solely on the question of which
model fits the distribution of effect sizes, and takes account of the relevant
source(s) of error.

• When studies are gathered from the published literature, the random effects
model is generally a more plausible match.

• The strategy of starting with a fixed-effect model and then moving to a

random-effects model if the test for heterogeneity is significant is a mistake,
and should be strongly discouraged.
AR, Lecturer, Statistics Discipline, KU 21
 Precaution
• The strategy of starting with a fixed-effect model and then moving
to a random-effects model if the test for heterogeneity is significant
is a mistake, and should be strongly discouraged.

AR, Lecturer, Statistics Discipline, KU 22

 Criteria to Choose Fixed effect
• It makes sense to use the fixed-effect model if two conditions are met.

• First, we believe that all the studies included in the analysis are functionally

identical.

• Second, our goal is to compute the common effect size for the identified

population, and not to generalize to other populations.

AR, Lecturer, Statistics Discipline, KU 23

 Choosing Fixed effect example
• For example, suppose that a pharmaceutical company will use a thousand patients to
compare a drug versus placebo.

• Because the staff can work with only 100 patients at a time, the company will run a
series of ten trials with 100 patients in each.

• The studies are identical in the sense that any variables which can have an impact on
the outcome are the same across the ten studies. Specifically, the studies draw
patients from a common pool, using the same researchers, dose, measure, and so on
(we assume that there is no concern about practice effects for the researchers, nor
for the different starting times of the various cohorts).
AR, Lecturer, Statistics Discipline, KU 24
 Choosing Fixed effect example
• All the studies are expected to share a common effect and so the first condition is
met.

• The goal of the analysis is to see if the drug works in the population from which the
patients were drawn (and not to extrapolate to other populations), and so the second
condition is met, as well.

• In this example the fixed-effect model is a plausible fit for the data and meets the
goal of the researchers. It should be clear, however, that this situation is relatively
rare.

AR, Lecturer, Statistics Discipline, KU 25

 Choosing a Random effects
• By contrast, when the researcher is accumulating data from a series of
studies that had been performed by researchers operating
independently, it would be unlikely that all the studies were
functionally equivalent.

• Typically, the subjects or interventions in these studies would have

differed in ways that would have impacted of the results, and therefore
we should not assume a common effect size.

AR, Lecturer, Statistics Discipline, KU 26

 Choosing a Random effects
• Therefore, in these cases the random-effects model is more easily justified than the
fixed-effect model.

• Additionally, the goal of this analysis is usually to generalize to a range of

scenarios.

• Therefore, if one did make the argument that all the studies used an identical,
narrowly defined population, then it would not be possible to extrapolate from this
population to others, and the utility of the analysis would be severely limited.

AR, Lecturer, Statistics Discipline, KU 27

 Why model should not be based on the test for heterogeneity?
• Some have adopted the practice of starting with a fixed-effect model and then
switching to a random-effects model if the test of homogeneity is statistically
significant.
• This practice should be strongly discouraged because the decision to use the
random-effects model should be based on our understanding of whether or not all
studies share a common effect size, and not on the outcome of a statistical test
(especially since the test for heterogeneity often suffers from low power).
• If the study effect sizes are seen as having been sampled from a distribution of
effect sizes, then the random-effects model, which reflects this idea, is the logical
one to use.
AR, Lecturer, Statistics Discipline, KU 28
 Why model should not be based on the test for heterogeneity?
• If the between-studies variance is substantial (and statistically significant) then
the fixed-effect model is inappropriate.
• However, even if the between-studies variance does not meet the criterion for
statistical significance (which may be due simply to low power) we should still
take account of this variance when assigning weights.
• If 𝑇 2 turns out to be zero, then the random-effects analysis reduces to the fixed-
effect analysis, and so there is no cost to using this model.
• On the other hand, if one has elected to use the fixed-effect model a priori but the
test of homogeneity is statistically significant, then it would be important to
revisit the assumptions that led to the selection of a fixed-effect model
AR, Lecturer, Statistics Discipline, KU 29