Introduction to Meta-Analysis:

Basic Steps and Recent Advances

Noel A. Card
Professor, Human Development & Family Science
Director, Quantitative Methods in Family and Social
Science doctoral certificate program
[Link]@[Link]

September 26, 2024– American Psychological Association Science Training Session

1
 2

Overview of talk
• What is meta-analysis?
• Basic steps in performing meta-analysis.
– Research questions appropriate for MA.
– Searching for literature.
– Coding the literature.
– Synthesizing research and drawing conclusions.
– Presenting findings.
• Recent advances in meta-analysis.
– Handling interdependent data.
– Lag-as-moderator MA.
– Cross temporal MA / Spatiotemporal MA.
– Network MA.
– Multivariate MA.
• Final thoughts
– A vision of MA as a critical tool for accumulation of knowledge
– Your questions
 Meta-analysis? Basic steps Recent advances Final thoughts

What is meta-analysis?

• Meta-analysis is…
– A type of literature review
– A methodological approach
– A statistical technique
– A tool for accumulation of knowledge

• “If I have seen further it is by standing upon the shoulders of

giants.” - Isaac Newton, 1675
 Meta-analysis? Basic steps Recent advances Final thoughts

Literature reviews

• A regular part of scholarship is to review literature.

– Literature reviews for comprehensive exams or
“Chapter 2” of dissertation.
– Writing chapter for edited book.
– Reviewing literature in Introduction section of
manuscript.
– Reading literature to inform practice or policy.
 Meta-analysis? Basic steps Recent advances Final thoughts

Literature reviews

• A regular part of scholarship is to review literature.

• However, few scholars formally trained in performing
literature reviews.
– Common doctoral coursework:
• 1-2 courses in research methodology.
• 2-5 courses in data analysis.
• Additional courses / work developing expertise in
conducting primary studies in specialized area.
• 0 courses / formal training in literature review / MA
 Meta-analysis? Basic steps Recent advances Final thoughts

Literature reviews

• A regular part of scholarship is to review literature.

• Literature reviews vary along several dimensions (Cooper
& Hedges, 1994)
– Audience (e.g., specialized scholars, general scholars, general
public).
– Perspective (e.g., neutral versus espousing position).
– Coverage (e.g., central, representative, exhaustive).
– Focus – see next slide.
– Method of synthesis – see next slide.
 Meta-analysis? Basic steps Recent advances Final thoughts

Literature reviews

Super-ordinate category: Literature review

Focus: Theories Research results Typical practices

Theoretical Research Survey

review synthesis

Unknown Tallying Statistical Statistical

Method of synthesis: significance analysis of analysis of
significance effect sizes

Narrative Informal Formal Meta-

research review vote-counting vote-counting analysis
 Meta-analysis? Basic steps Recent advances Final thoughts

Literature reviews

Super-ordinate category: Literature review

Focus: Theories Research results Typical practices

Theoretical Research Survey

review synthesis

Statistical Analysis implies that

study results are quantified
Method of synthesis:
Unknown Tallying
significance
(excludes
Statistical
analysis of
qualitative research)
Statistical
analysis of
significance effect sizes

Narrative Informal Formal Meta-

research review vote-counting vote-counting analysis
 Meta-analysis? Basic steps Recent advances Final thoughts

Terminology

• “Meta-Analysis” – narrow and broad meanings

– Narrow meaning: “Meta-analysis” is the statistical
analysis of coded study information.
• Other steps (e.g., literature search, coding information from
studies) is part of “Systematic review”
– Broad meaning: “Meta-analysis” is all steps needed to
perform and publish (or otherwise present) this
statistical analysis of quantified study results.

– “that which we call a rose

by any other name would smell as sweet“
– William Shakespeare, Romeo & Juliet, 1590s
 Meta-analysis? Basic steps Recent advances Final thoughts

10

Roadmap
• What is meta-analysis?
• Basic steps in performing meta-analysis.
– Research questions appropriate for MA.
– Searching for literature.
– Coding the literature.
– Synthesizing research and drawing conclusions.
– Presenting findings.
• Recent advances in meta-analysis.
– Handling interdependent data.
– Lag-as-moderator MA.
– Cross temporal MA / Spatiotemporal MA.
– Network MA.
– Multivariate MA.
• Final thoughts
– A vision of MA as a critical tool for accumulation of knowledge
– Your questions
 Meta-analysis? Basic steps Recent advances Final thoughts

11
Step 1: Research Questions for MA

• What is the study result of interest?

– Examples:
• An association between two variables.
• A difference between two groups (e.g., treatment versus control).
• The mean of a single scale.
• A proportion (e.g., prevalence of diagnosis).
• Internal consistency of a scale.
– Must be operationalized as an effect size.
 Meta-analysis? Basic steps Recent advances Final thoughts

12
Step 1: Research Questions for MA

• What is the study result of interest?

• How do you want to synthesize across studies?
– Combine: What is the overall average effect size
across all available studies?
– Compare: Do some types of studies have larger effect
sizes than others?
• Different types of intervention
• Different ages / characteristics of sample
• Different regions / countries
 Meta-analysis? Basic steps Recent advances Final thoughts

13
Step 1: Research Questions for MA

• What is the study result of interest?

• How do you want to synthesize across studies?
• How will this meta-analysis advance understanding?
– Conflicting findings across studies
– Uncertainties due to small samples (low power / precision)
– Increased generalizability relative to any one study
• Participant diversity
• Different measures / methodologies
• Different regions of US / countries around world
• Different times (i.e., historic effects)
 Meta-analysis? Basic steps Recent advances Final thoughts

14
Step 2: Searching for Literature

• Study eligibility criteria (AKA inclusion & exclusion criteria).

– Set of statements that define that population of
studies included in a meta-analysis.
• Guide decisions to include studies.
• Communicate to readers.
 Meta-analysis? Basic steps Recent advances Final thoughts

15
Step 2: Searching for Literature

• Study eligibility criteria (AKA inclusion & exclusion criteria).

– Set of statements that define that population of
studies included in a meta-analysis.
– Common criteria:
• Characteristics of intervention.
• Operational definitions of variables.
• Research methods (e.g., type of design, placebo control).
• Sample characteristics (e.g., age range).
• Cultural range (e.g., publication language).
 Meta-analysis? Basic steps Recent advances Final thoughts

16
Step 2: Searching for Literature

• Search strategy.
– Some approaches:
• Electronic bibliographic databases (e.g., PsycInfo):
Specify keywords, Boolean operations.

• Forward searches: Studies citing seminal works in area.

• Backward searches: Reading literature cited in other reports /
reviews.

• Conference programs: Captures newer work.

• Granting agencies: Research evaluated before results are known.
• Clinical trial / preregistration databases
• Email / listserve requests: Especially for unpublished work.
 Meta-analysis? Basic steps Recent advances Final thoughts

17
Step 2: Searching for Literature

• Electronic bibliographic databases

– Types of databases:
• Disciplinary specific database (e.g., PsychINFO, ERIC, MEDLINE)
• Transdisciplinary, curated databases (e.g., Web of Science, Scopus)
– Proquest / Dissertation Abstracts: Good source of dissertations (unpublished)
• Transdisciplinary, non-curated databases (e.g., Google Scholar)
– Strengths and limitations:
• Discipline specific: Likely more precision. Many are well indexed. Often
common terminology / keywords.
• Transdisciplinary: Allows us to find research outside of own discipline.
Increased recall, but (potentially very large) reduction in precision.
• Curated databases: Need to understand what sources, especially
unpublished works, are included.
• Google scholar: Gets anything posted on web (published, unpublished,
random opinions). What are potential biases?
 Meta-analysis? Basic steps Recent advances Final thoughts

18
Step 2: Searching for Literature

• Publication bias.

Population of effect sizes

ns results
-0.20 -0.10 0.00 0.10 0.20 0.30 0.40 0.50 0.60

Published effect sizes

-0.20 -0.10 0.00 0.10 0.20 0.30 0.40 0.50 0.60

 Meta-analysis? Basic steps Recent advances Final thoughts

19

Roadmap
• Steps in performing meta-analysis.
– Research questions appropriate for MA.
– Searching for literature.
– Coding the literature.
– Synthesizing research and drawing conclusions.
– Presenting findings.
 Meta-analysis? Basic steps Recent advances Final thoughts

20
Step 3: Coding the Literature

• Systematically code information from studies.

– Study characteristics.
• Features of sample, methodology, measurement.
• Used to…
– Describe state of research.
– Predict differences in study results.
 Meta-analysis? Basic steps Recent advances Final thoughts

21
Step 3: Coding the Literature

• Systematically code information from studies.

– Study characteristics.
– Study results.
• Coded as Effect Sizes (!!!)
 Meta-analysis? Basic steps Recent advances Final thoughts

22
Step 3: Coding the Literature

• Effect sizes.
– Effect sizes are not…
• Author conclusions in Discussion.
• Significance tests (i.e., p values).
 Meta-analysis? Basic steps Recent advances Final thoughts

23
Step 3: Coding the Literature

• Effect sizes.
– Effect sizes are not…
• Author conclusions in Discussion.
• Significance tests (i.e., p values).
– Effect sizes are…
• Comparable / combinable across studies.
• Calculable from a wide range of reported results
(e.g., summary statistics, t-tests, ANOVA results, correlations, odds ratios).

• Understandable / meaningful to readers.

– Convey information about direction and magnitude of
study result
 Meta-analysis? Basic steps Recent advances Final thoughts

24
Step 3: Coding the Literature

• Effect sizes.
– Many types of effect sizes
• r = association between two variables.
• d or g = standardized mean difference between two
groups.
• Odds ratio = contingency / association between two
dichotomous variables.
 Meta-analysis? Basic steps Recent advances Final thoughts

25
Step 3: Coding the Literature

• Effect sizes.
– Many types of effect sizes
• r = association between two (continuous) variables.
• d or g = standardized mean difference between two groups.
• Odds ratio = contingency / association between two dichotomous variables.
• Mean / proportion = central tendency of one variable.
• Unstandardized mean difference = retains meaningful
information about scale.
• Longitudinal change scores / pre-post differences.
• Psychometric properties of measures.
 Meta-analysis? Basic steps Recent advances Final thoughts

26
Step 3: Coding the Literature

• Effect sizes.
– Many types of effect sizes
– Tasks of meta-analyst:
• Decide type of study result to extract.
• Decide type of effect size best suited for this type of
result.
• Compute this effect size from diverse ways results
reported.
 Meta-analysis? Basic steps Recent advances Final thoughts

27
Step 3: Coding the Literature

• Effect sizes.
– Many types of effect sizes
– Tasks of meta-analyst.
– Computing effect sizes:
• Use formulas reported
in books and articles.
• Use online calculator.
• Use functions built into
MA software.
 Meta-analysis? Basic steps Recent advances Final thoughts

28
Step 3: Coding the Literature

• Computing effect sizes.

– Online calculator through Campbell Collaboration:
The URL seems to change occasionally: Just Google “Campbell Collaboration effect size calculator”
 Meta-analysis? Basic steps Recent advances Final thoughts

29
Step 3: Coding the Literature

• Computing effect sizes.

– Online calculator through Campbell Collaboration:
The URL seems to change occasionally: Just Google “Campbell Collaboration effect size calculator”
 Meta-analysis? Basic steps Recent advances Final thoughts

30
Step 3: Coding the Literature

• Computing effect sizes.

– Online calculator through Campbell Collaboration:
The URL seems to change occasionally: Just Google “Campbell Collaboration effect size calculator”

Enter values from study

Receive effect sizes for MA

 Meta-analysis? Basic steps Recent advances Final thoughts

31
Step 3: Coding the Literature

• Computing effect sizes.

– The metafor package contains a function, escalc
• Useful for converting information from many studies into an effect
size
• General syntax:
escalc(measure=[ES desired], [variables], [data])
E.g.,
escalc(measure = “SMD”, m1i = TreatMean, sd1i =
TreatSD, n1i = TreatN, m2i = ControlMean, sd2i =
ControlSD, n2i = ControlN, data = MyMetaAnalysis)

• Requires all studies in your MA report data in the same way (or a
limited number of ways)
 Meta-analysis? Basic steps Recent advances Final thoughts

32

Roadmap
• Steps in performing meta-analysis.
– Research questions appropriate for MA.
– Searching for literature.
– Coding the literature.
– Synthesizing research and drawing conclusions.
– Presenting findings.
 Meta-analysis? Basic steps Recent advances Final thoughts

33
Step 4: Synthesizing Effect Sizes

• Two major goals:.

– Combining effect sizes across studies.
• Overall average effect size.
– Comparing effect sizes across studies.
• Coded study characteristics as predictors of effect
sizes.
 Meta-analysis? Basic steps Recent advances Final thoughts

34
Step 4: Synthesizing Effect Sizes

• Combining effect sizes across studies.

– Studies are weighted by the precision of the point
estimate of effect size.
1
– A common weighting: wi = 2
SE i

• Where SEi is the standard error for study i.

• Larger N studies receive larger weight
 Role of Lit Rev. Steps in MA Other App.

35
Step 4: Synthesizing Effect Sizes

• Combining effect sizes across studies.

– Two situations:

Sampling error only

Frequency of observations

Effect sizes (T)

 Meta-analysis? Basic steps Recent advances Final thoughts

36
Step 4: Synthesizing Effect Sizes

• Combining effect sizes across studies.

– Two situations:

Sampling error only Sampling error + other variability

Frequency of observations
Frequency of observations

Effect sizes (T) Effect sizes (T)

 Meta-analysis? Basic steps Recent advances Final thoughts

37
Step 4: Synthesizing Effect Sizes

• Combining effect sizes across studies.

– When there is significant heterogeneity (i.e., other
between-study variability), it is necessary to account for both
sources of imprecision  Random-effects model
Fixed-effects model Random-effects model
μ
θ

Frequency of observations
Frequency of observations

Effect sizes (T) Effect sizes (T)

Ti = θ + εi Ti = μ + ξi + εi
where θi is the population effect size for each study, ξi
where εi is the sampling error of Ti (xi) is the study-deviation from μ, and εi is the sample-
sampling error
 Meta-analysis? Basic steps Recent advances Final thoughts

38
Step 4: Synthesizing Effect Sizes

• Combining effect sizes across studies.

– Another way to think about random-effects models:
Population of all possible
μ
studies of a phenomenon

Variability due to population

ξA ξB studied, methodology,
measurement, or ???.
Quantified as τ2

ε1 ε2 ε3 ε4 Variability due to sampling

error / fluctuation

Study 1 Study 2
Study 3 Study 4

Observed ES
Low High
 Meta-analysis? Basic steps Recent advances Final thoughts

39
Step 4: Synthesizing Effect Sizes

• Combining effect sizes across studies.

– Studies are weighted by the precision of the point
estimate of effect size.
– A more appropriate (random effects) weighting:
1
wi =
SEi2 + τ 2

• Where SEi is the standard error for study i.

• τ2 is the estimated between-study variance above
and beyond sampling error variance.
 Meta-analysis? Basic steps Recent advances Final thoughts

40
Step 4: Synthesizing Effect Sizes

• Combining effect sizes across studies.

• Comparing effect sizes across studies.
– Rather than simply modeling heterogeneity as “other
between-study variability”…
– … attempt to predict why studies vary in their effect
Population of all possible
sizes. μ
studies of a phenomenon

Study characteristics ξ ξ Variability due to population

studied, methodology,
predict this variability A B
measurement, or ???

ε1 ε2 ε3 ε4 Variability due to sampling

error / fluctuation

Study Study
1 2Study
Study
3 4
Observed ES
Low High
 Meta-analysis? Basic steps Recent advances Final thoughts

41
Step 4: Synthesizing Effect Sizes

• Use of software.
– Until fairly recently, meta-analysts had limited choices:
• Limit analyses to those that could be conducted by
hand (averages, ANOVA comparison of categorical predictors).
• Use common analysis packages (e.g., SPSS) and
hand correct results.
• Buy specialized commercial software (e.g., CMA)
 Meta-analysis? Basic steps Recent advances Final thoughts

42
Step 4: Synthesizing Effect Sizes

• Use of software.
– Until fairly recently, meta-analysts had limited choices
– Now, several packages available in R
• I will briefly introduce metafor as a package that
is highly flexible but reasonably easy to use
 Meta-analysis? Basic steps Recent advances Final thoughts

43
Step 4: Synthesizing Effect Sizes

• An example…
– 11 studies reporting correlation between proactive
and reactive aggression among children and
adolescents (partial results; Card & Little, 2006)

Study N Age Reporter r

Crain (2002) 134 10.2 peer 0.95
Hegland & Rix (1999) 32 6.2 observations 0.06
Poulin et al (1997) 66 8.0 observations 0.47
Price & Dodge (1989) 70 6.0 observations 0.04
Prinstein & Cillessen (2003) 235 16.3 peer 0.23
Pulkkinen (1996) 369 8.7 peer -0.27
Salmivalli & Nieminen(2002) 1062 12.0 peer 0.67
Salmivalli et al (2005) 589 12.0 peer 0.82
Schwartz (1994) 442 12.0 peer 0.89
Schwartz et al (1998) 66 8.0 observations 0.48
Strassberg et al. (1994) 273 5.0 observations 0.18
 Meta-analysis? Basic steps Recent advances Final thoughts

44
Step 4: Synthesizing Effect Sizes

• Initial step: Read data into R

– Typically code in something like LibreOffice Calc or
MS Excel, then bring into analysis package
– Sample script on next slide
 Meta-analysis? Basic steps Recent advances Final thoughts

45
Step 4: Synthesizing Effect Sizes
 Meta-analysis? Basic steps Recent advances Final thoughts

46
Step 4: Synthesizing Effect Sizes

• First analysis: Compute average ES and heterogeneity

– Average effect size: Answers research questions
about overall results (across all studies)
Σ(wiESi)
MES =
Σ(wi)
– Heterogeneity suggests potential to look at different
types of studies
Q = Σ [wi (ESi − MES)2]

– Results on next slide

 Meta-analysis? Basic steps Recent advances Final thoughts

47
Step 4: Synthesizing Effect Sizes

Heterogeneity
Strong positive
correlation
 Meta-analysis? Basic steps Recent advances Final thoughts

48
Step 4: Synthesizing Effect Sizes

• Second analysis: Does ‘reporter’ moderate the

correlation between types of aggression?
– Do some types of measures find higher co-
occurrence of types of aggression?
– Do some types of measures better distinguish these
types of aggression?
– Evaluated by adding ‘reporter’ as predictor in model
– Results on next slide
 Meta-analysis? Basic steps Recent advances Final thoughts

49
Step 4: Synthesizing Effect Sizes

Sig. prediction by
reporter

Higher r for peer reports

than observations
 Meta-analysis? Basic steps Recent advances Final thoughts

50
Step 4: Synthesizing Effect Sizes

• Third analysis: Does ‘age’ moderate the correlation

between types of aggression?
– Do these types of aggression converge with age?
– Do these types of aggression separate (diverge) with
age?
– Evaluated by adding ‘age’ as predictor in model
– Results on next slide
 Meta-analysis? Basic steps Recent advances Final thoughts

51
Step 4: Synthesizing Effect Sizes

Sig. prediction by age

r increases with age

 Meta-analysis? Basic steps Recent advances Final thoughts

52
Step 4: Synthesizing Effect Sizes

• Fourth analysis: Are both ‘reporter’ and ‘age’ unique

moderators of the correlation between types of
aggression?
– Reporter and age may be overlapping (observations in
childhood and peer reports in adolescence)

– Results on next slide

 Meta-analysis? Basic steps Recent advances Final thoughts

53
Step 4: Synthesizing Effect Sizes

Both
predictors
sig.

Plot of effect size by

age and reporter
 Meta-analysis? Basic steps Recent advances Final thoughts

54
Step 4: Synthesizing Effect Sizes

• Summary.
– Methods of analysis are not much more complex than
other data analyses (e.g., regression).
– For full details see e.g.,:
• Card (2012). Applied Meta-Analysis for Social Science
Research. Guilford. (Second edition in progress; likely late 2026)
• Lipsey & Wilson (2001). Practical Meta-Analysis. Sage.
• Viechtbauer, W. (2010). Conducting Meta-Analyses in R with
the metafor Package. Journal of Statistical Software, 36(3), 1–48
 Meta-analysis? Basic steps Recent advances Final thoughts

55

Roadmap
• Steps in performing meta-analysis.
– Research questions appropriate for MA.
– Searching for literature.
– Coding the literature.
– Synthesizing research and drawing conclusions.
– Presenting findings.
 Meta-analysis? Basic steps Recent advances Final thoughts

56

Step 5: Presenting Findings

• Some unique graphical

displays of MA

Identification
Records identified through Additional records identified
database searching through other sources
– PRISMA flow diagram (n = ) (n = )

• Documents search &

Records after duplicates removed
inclusion decisions (n = )

Screening
• Facilitates transparency
of reporting Records screened
(n = )
Records excluded
(n = )

Full-text articles Full-text articles

Eligibility

assessed for eligibility excluded, with reasons

(n = ) (n = )

Studies included in
qualitative synthesis
(n = )
Included

Studies included in
quantitative synthesis
(meta-analysis)
(n = )
 Meta-analysis? Basic steps Recent advances Final thoughts

57

Step 5: Presenting Findings

• Some unique graphical

displays of MA
– Funnel plot
• Typically used to
evaluate publication bias
• But also presents
information about central
tendency and variability
in effect sizes
 Meta-analysis? Basic steps Recent advances Final thoughts

58

Step 5: Presenting Findings

• Some unique graphical

displays of MA
– Forrest plot
• Shows effect sizes of
individual studies…
• … and the overall results
of the MA
 Meta-analysis? Basic steps Recent advances Final thoughts

59

Step 5: Presenting Findings

• There exist several reporting standards

– APA MARS (Meta-Analysis Reporting Standards)
 Meta-analysis? Basic steps Recent advances Final thoughts

60

Step 5: Presenting Findings

• There exist several reporting standards

– APA MARS (Meta-Analysis Reporting Standards)
– PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses)
 Meta-analysis? Basic steps Recent advances Final thoughts

61

Step 5: Presenting Findings

• There exist several reporting standards

– Advantages of Reporting Standards:
• Helps authors ensure that important aspects are not missed
• Provides reviewers / editors who are unfamiliar with MA tools for
evaluating MAs
• Possibly limits potential for misleading reporting and ‘fishing’
(multiple approaches to obtain desired results)
– Drawbacks of Reporting Standards:
• Assumes “one size fits all” (PRISMA website acknowledges focus on
randomized clinical trials, but then applied broadly)
• Limits analysts from tailoring methodology to specific research
questions, when warranted
• Potential for uninformed reviewers to be dogmatic
• Can encourage a ‘cookbook’ approach to manuscript writing
 Meta-analysis? Basic steps Recent advances Final thoughts

62

Roadmap
• What is meta-analysis?
• Basic steps in performing meta-analysis.
– Research questions appropriate for MA.
– Searching for literature.
– Coding the literature.
– Synthesizing research and drawing conclusions.
– Presenting findings.
• Recent advances in meta-analysis.
– Handling interdependent data.
– Lag-as-moderator MA.
– Cross temporal MA / Spatiotemporal MA.
– Network MA.
– Multivariate MA.
• Final thoughts
– A vision of MA as a critical tool for accumulation of knowledge
– Your questions
 Meta-analysis? Basic steps Recent advances Final thoughts

63

Handling interdependent data

• Studies in a MA often have multiple effect sizes

– Multiple measures of a construct
– Multiple follow-up occasions
– Multiple treatment groups
– Subgroup results
 Meta-analysis? Basic steps Recent advances Final thoughts

64

Handling interdependent data

• Studies in a MA often have multiple effect sizes

• Strategies for handling dependent ES:
– Ignore: Not a good option. Overweights studies with multiple
ES; standard errors (and inferential conclusions) wrong.
– Combine: Aggregate ES within study so each study has
one ES.
– Sub-classify: Separate ES with different values of
moderator variable.
– Model
• Multivariate MA
• Multilevel MA
• Robust variance estimation – implemented using
robumeta package in R
Pustejovsky, J. E., & Tipton, E. (2021). Meta-analysis with robust variance estimation:
Expanding the range of working models. Prevention Science, 23, 425-438.
 Meta-analysis? Basic steps Recent advances Final thoughts

65

Lag-as-Moderator Meta-Analysis

• Longitudinal studies (cross lag panel models) often test Time

1 X as predictor Time 2 Y (sometimes controlling for initial
levels of Y) :

Time 1
X

Time 1 Time 2
Y Y
 Meta-analysis? Basic steps Recent advances Final thoughts

66

Lag-as-moderator Meta-Analysis

• Longitudinal studies often test Time 1 X Time 2 Y

• However, results might differ depending on amount of
time between Times 1 and 2
Time 1
X

Time 1 Time 2
Y Y

Time 1
X

Time 1 Time 2
Y Y
 Meta-analysis? Basic steps Recent advances Final thoughts

67

Lag-as-moderator Meta-Analysis

• Lag-as-Moderator Meta-Analysis (LAMMA) takes

advantage of naturally occurring variability across
studies.
– Models effect sizes at different lag lengths
– Useful for understanding
• Decay of impact across time
• Optimal lags to test associations
• Expected effect sizes in future longitudinal research (sample
size planning)

Card, N. A. (2019). Lag as moderator meta-analysis: A methodological

approach for synthesizing longitudinal data. International Journal of
Behavioral Development, 49, 80-89
 Meta-analysis? Basic steps Recent advances Final thoughts

68

Cross temporal / Spatiotemporal Meta-Analysis

• Cross-temporal MA: Evaluates change in an effect

across historic time.
– Test year of study as predictor of effect
– Often focus on means of a common measure over long periods
of time (multiple decades)
– Method of testing historic change without conducting decades-
long prospective studies
– An example (Twenge et al., 2008):

Twenge, J. M., Konrath, S., Foster, J. D., Campbell, W. K., & Bushman, B. J.
(2008). Egos inflating over time: a cross-temporal meta-analysis of the
Narcissistic Personality Inventory. Journal of Personality, 76, 875-902.
 Meta-analysis? Basic steps Recent advances Final thoughts

69

Cross temporal / Spatiotemporal Meta-Analysis

• Cross-temporal MA: Evaluates change in an effect

across historic time.
• Spatiotemporal MA: Adds information about context in
which study was conducted.
– Contextual information found outside of study (e.g., census data,
weather / climate records, internet / social media use).
– Example (reviewed by Johnson et al., 2017):
As main effect: Frequency of internet
searches of racist term predicted
mortality of Black individuals.

As moderator: Community prejudice

moderated (reduced) effectiveness of
psychosocial HIV prevention trials
Johnson, B. T., Cromley, E. K., & Marrouch, N. (2017). Spatiotemporal
meta-analysis: reviewing health psychology phenomena over space
and time. Health Psychology Review, 11, 280-291.
 Meta-analysis? Basic steps Recent advances Final thoughts

70

Network Meta-Analysis

• Attempts to compare effectiveness of interventions that

have not been studied head-to head.

Study 1
Treatment 1 vs. control, d1 v C = .50

Study 2
Treatment 2 vs. control, d2 v C = .30

Possible conclusion
Treatment 1 vs. Treatment 2,
d1 v 2 = d1 v C – d2 v C = .50 - .30 = .20
 Meta-analysis? Basic steps Recent advances Final thoughts

71

Network Meta-Analysis

• Attempts to compare effectiveness of interventions that

have not been studied head-to head.

Study 1 Key assumption: Transitivity = Studies are

Treatment 1 vs. control, d1 v C = .50 similar/identical in terms of control group
and other features (e.g., participant
Study 2 characteristics).
Treatment 2 vs. control, d2 v C = .30

Ideally, identify a small number of head-to-

Possible conclusion
head comparison studies to test if transitivity
Treatment 1 vs. Treatment 2,
assumption holds for the larger collection of
d1 v 2 = d1 v C – d2 v C = .50 - .30 = .20
studies.
 Meta-analysis? Basic steps Recent advances Final thoughts

72

Multivariate Meta-Analysis

• Meta-analyses have historically been used to synthesize

“simple” study results (correlations, mean differences).
• In primary research, we often test more complex,
multivariate models (e.g., multiple regression, factor analysis).
• Difficult to meta-analytically synthesize multivariate effect
sizes.
– Necessary for all studies to include the same predictors in
model
– Example, Interested in X predicting Y, controlling for Z:
Study 1 Can use
Study 2 Cannot use
Study 3 Cannot use
 Meta-analysis? Basic steps Recent advances Final thoughts

73

Multivariate Meta-Analysis

• Instead, use MA to produce a series of correlations that

are sufficient statistics for multivariate analysis.
• rXZ Table 12.1. Example multivariate meta-analysis of correlations among relational aggression, overt

rXY
aggression, and peer rejection.

• Study
Andreou (2006)
Sample size Relational- Relational-
(N)
403
Overt r
.472
Overt-
Rejection r Rejection r

Arnold (1999)

rYZ
110 .707

• Berdugo-Arstark (2002)
Blachman (2003)
Brendgen et al. (2005)
128
228
468
.549
.440
.420
.483 .592

Butovskaya et al. (2007) 212 .576

Campbell (1999) 139 .641
Carpenter (2002) 75 .260
Carpenter & Nangle (2006) 82 .270
Cillessen & Mayeux (2004) 607 .561

rYX − rYZ rXZ

Cillessen et al. (2005) 224 .652
Cote et al. (2007) 1183 .345

βX =
Coyne & Archer (2005) 347 .540
Crain (2002) 134 .870

1 − rXZ
Crick (1995) 252 .656
2 Crick (1996)
Crick (1997)
245
1166
.770
.630
Crick & Grotpeter (1995) 491 .540 .121 .228
Crick et al. (1997) 65 .607 .280 .367
Geiger (2003) 458 .650 .520 .480
Hawley et al. (2007) 929 .669 .146 .089
Henington (1996) 904 .561 .310 .295
Johnson (2003) 74 .735 .368 .527
Leff (1995) 151 .790 .570 .570
Miller (2001) 150 .530 .530 .420
Murray-Close & Crick (2006) 590 .700 .540 .510
Nelson et al. (2005) 180 .584 .030 .304
a 139 .403 .332 .402
Ostrov (under review)
Ostrov & Crick (2007) 132 .030 .045 .155
Ostrov et al. (2004) 60 .000 b .100
Pakaslahti et al. (1998) 839 .580 .269 .250
Phillipsen et al. (1999) 262 -.045 .013
Rys & Bear (1997) 266 .423 .378
Salmivalli et al. (2000) 209 .681 .240 .385
Tomada & Schneider (1997) 314 .666 .153 .240
Werner (2000) 881 .440 .430
 Meta-analysis? Basic steps Recent advances Final thoughts

74

Multivariate Meta-Analysis

• Instead, use MA to produce a series of correlations that

are sufficient statistics for multivariate analysis.
• Challenging to implement until recent advances and
software.
 Meta-analysis? Basic steps Recent advances Final thoughts

75

Roadmap
• What is meta-analysis?
• Basic steps in performing meta-analysis.
– Research questions appropriate for MA.
– Searching for literature.
– Coding the literature.
– Synthesizing research and drawing conclusions.
– Presenting findings.
• Recent advances in meta-analysis.
– Handling interdependent data.
– Lag-as-moderator MA.
– Cross temporal MA / Spatiotemporal MA.
– Network MA.
– Multivariate MA.
• Final thoughts
– A vision of MA as a critical tool for accumulation of knowledge
– Your questions
 Meta-analysis? Basic steps Recent advances Final thoughts

76

Roadmap
• What is meta-analysis?
• Basic steps in performing meta-analysis.
– Research questions appropriate for MA.
– Searching for literature.
– Coding the literature.
– Synthesizing research and drawing conclusions.
– Presenting findings.
• Recent advances in meta-analysis.
– Handling interdependent data.
– Lag-as-moderator MA.
– Cross temporal MA / Spatiotemporal MA.
– Network MA.
– Multivariate MA.
• Final thoughts
– A vision of MA as a critical tool for accumulation of knowledge
– Your questions
 Meta-analysis? Basic steps Recent advances Final thoughts

77
MA as a critical tool for accumulation of knowledge

• Literature reviews are critical components of

accumulating knowledge, and meta-analysis is a well-
refined methodology for conducting literature reviews.
 Meta-analysis? Basic steps Recent advances Final thoughts

78
MA as a critical tool for accumulation of knowledge

• Literature reviews are critical components of

accumulating knowledge, and meta-analysis is a well-
refined methodology for conducting literature reviews.
• Recent advances in meta-analysis have allowed this
approach to be even more valuable.
 Meta-analysis? Basic steps Recent advances Final thoughts

79
MA as a critical tool for accumulation of knowledge

• Literature reviews are critical components of

accumulating knowledge, and meta-analysis is a well-
refined methodology for conducting literature reviews.
• Recent advances in meta-analysis have allowed this
approach to be even more valuable.
• However, meta-analysis typically is used as a
retrospective tool:
– After many studied have been conducted, MA is used
to summarize what is known.
 Meta-analysis? Basic steps Recent advances Final thoughts

80
MA as a critical tool for accumulation of knowledge

• An alternative perspective: Meta-analysis as a critical

tool at beginning and throughout research accumulation.
– Cumulative meta-analysis = studies are added one at
a time, and results are updated.
 Meta-analysis? Basic steps Recent advances Final thoughts

81
MA as a critical tool for accumulation of knowledge

• Early in understanding: Studies are exact (AKA direct)

replications of one another.
– The value of each study is in how it changes our point
estimate and confidence interval of the effect size
 Meta-analysis? Basic steps Recent advances Final thoughts

82
MA as a critical tool for accumulation of knowledge

• Early in understanding: Studies are exact (AKA direct)

replications of one another.
• However, studies are likely rarely exact replications, but
are unplanned inexact (AKA conceptual) replications
 Meta-analysis? Basic steps Recent advances Final thoughts

83
MA as a critical tool for accumulation of knowledge

• Unplanned inexact replications limit the precision of our

knowledge. So, we then perform planned inexact replications.
– Deliberately vary the procedures, measures, and/or populations
studied
– Guided by theory, needs of policy / practice, etc.
 Meta-analysis? Basic steps Recent advances Final thoughts

84
MA as a critical tool for accumulation of knowledge

• Challenges of this model

– De-emphasizing individual studies in favor of research
accumulation is a large shift in practices
• Results emerge slowly and less dramatically than “A new study
shows…”
• How do we evaluate researchers’ efforts (e.g., P&T, prestigious
positions, awards, etc)
– Requires large amount of coordination and collaboration of
efforts
– Uncertain if one person could change the practices in a
field
 Meta-analysis? Basic steps Recent advances Final thoughts

85
MA as a critical tool for accumulation of knowledge

• Open synthesis communities

• Nakagawa et al. (2020). A new ecosystem for evidence synthesis. Nature Ecology & Evolution
 Meta-analysis? Basic steps Recent advances Final thoughts

86

Roadmap
• What is meta-analysis?
• Basic steps in performing meta-analysis.
– Research questions appropriate for MA.
– Searching for literature.
– Coding the literature.
– Synthesizing research and drawing conclusions.
– Presenting findings.
• Recent advances in meta-analysis.
– Handling interdependent data.
– Lag-as-moderator MA.
– Cross temporal MA / Spatiotemporal MA.
– Network MA.
– Multivariate MA.
• Final thoughts
– A vision of MA as a critical tool for accumulation of knowledge
– Your questions
 87

Thanks!!

Questions / Comments / Discussion.

Contact: [Link]@[Link]