1

RESEARCH SYNTHESIS AS A SCIENTIFIC PROCESS

HARRIS COOPER LARRY V. HEDGES

Duke University Northwestern University

CONTENTS
1.1 Introduction 4
1.1.1 Replication and Research Synthesis 4
1.2 Research Synthesis in Context 4
1.2.1 A Definition of the Literature Review 4
1.2.2 A Definition of Research Synthesis 6
1.3 A Brief History of Research Synthesis as a Scientific Process 7
1.3.1 Early Developments 7
1.3.2 Research Synthesis Comes of Age 7
1.3.3 Rationale for the Handbook 10
1.4 Stages of Research Synthesis 11
1.4.1 Problem Formulation 11
1.4.2 Literature Search 12
1.4.3 Data Evaluation 12
1.4.4 Data Analysis 13
1.4.5 Interpretation of Results 14
1.4.6 Public Presentation 14
1.5 Chapters in Perspective 15
1.6 References 15

3
4 INTRODUCTION

It is necessary, while formulating the problems of which in our advance we are to find
the solutions, to call into council the views of those of our predecessors who have de-
clared an opinion on the subject, in order that we may profit by whatever is sound in
their suggestions and avoid their errors.
Aristotle, De Anima, Book 1, chapter 2

1.1 INTRODUCTION
criterion) produced the correct finding. If results that are
The moment we are introduced to science we are told it expected to be very similar show variability, the scientific
is a cooperative, cumulative enterprise. Like the artisans instinct should be to account for the variability by further
who construct a building from blueprints, bricks, and systematic work.
mortar, scientists contribute to a common edifice, called
knowledge. Theorists provide our blueprints and re- 1.2 RESEARCH SYNTHESIS IN CONTEXT
searchers collect the data that are our bricks.
To extend the analogy further yet, we might say that 1.2.1 A Definition of the Literature Review
research synthesists are the bricklayers and hodcarriers of
the science guild. It is their job to stack the bricks accord- The American Psychological Association’s PsycINFO
ing to plan and apply the mortar that makes the whole reference database defines a literature review as “the
thing stick. process of conducting surveys of previously published
Anyone who has attempted a research synthesis is en- material” ([Link] Common to all
titled to a wry smile as the analogy continues. They know definitions of literature reviews is the notion that they are
that several sets of theory-blueprints often exist, describ- “not based primarily on new facts and findings, but on
ing structures that vary in form and function, with few a publications containing such primary information, whereby
priori criteria for selecting between them. They also know the latter is digested, sifted, classified, simplified, and
that our data-bricks are not all six-sided and right-angled. synthesized” (Manten 1973, 75).
They come in a baffling array of sizes and shapes. Mak- Table 1.1 presents a taxonomy of literature reviews
ing them fit, securing them with mortar, and seeing that captures six distinctions used by literature review au-
whether the resulting construction looks anything like the thors to describe their own work (Cooper 1988, 2003).
blueprint is a challenge worthy of the most dedicated, in- The taxonomy was developed and can be applied to liter-
spired artisan. ature reviews appearing throughout a broad range of both
the behavioral and physical sciences. The six features and
1.1.1 Replication and Research Synthesis their subordinate categories permit a rich level of distinc-
tion among works of synthesis.
Scientific literatures are cluttered with repeated studies of The first distinction among literature reviews concerns
the same phenomena. Multiple studies on the same prob- the focus of the review, or, the material that is of central
lem or hypothesis arise because investigators are unaware interest to the reviewer. Most literature reviews center on
of what others are doing, because they are skeptical about one or more of four areas: the findings of individual pri-
the results of past investigations, or because they wish to mary studies; the methods used to carry out research;
extend (that is, generalize or search for influences on) theories meant to explain the same or related phenomena;
previous findings. Experience has shown that even when and the practices, programs, or treatments being used in
considerable effort is made to achieve strict replication, an applied context.
results across studies are rarely identical at any high level The second characteristic of a literature review is its
of precision, even in the physical sciences (see Hedges goals. Goals concern what the preparers of the review
1987). No two bricks are exactly alike. hope to accomplish. The most frequent goal for a review
How should scientists proceed when results differ? is to integrate past literature that is believed to relate to a
First, it is clear how they should not proceed: they should common topic. Integration includes: formulating general
not pretend that there is no problem, or decide that just one statements that characterize multiple specific instances
study (perhaps the most recent one, or the one they con- (of research, methods, theories, or practices); resolving
ducted, or a study chosen via some other equally arbitrary conflict between contradictory results, ideas, or statements
 RESEARCH SYNTHESIS AS A SCIENTIFIC PROCESS 5

Table 1.1 A Taxonomy of Literature Reviews should stimulate future work, and methodological prob-
lems or problems in logic and conceptualization that have
Characteristic Categories impeded progress within a topic area or field.
Of course, reviews more often than not have multiple
Focus research findings
goals. So, for example, it is rare to see the integration or
research methods
theories critical examination of existing work without also seeing
practices or applications the identification of central issues for future endeavors.
A third characteristic that distinguishes among litera-
Goal integration
ture reviews, perspective, relates to whether the reviewers
generalization
conflict resolution have an initial point of view that might influence the dis-
linguistic bridge-building cussion of the literature. The endpoints on the continuum
criticism of perspective might be called neutral representation and
identification of central issues espousal of a position. In the former, reviewers attempt to
Perspective neutral representation present all arguments or evidence for and against various
espousal of position interpretations of the problem. The presentation is meant
to be as similar as possible to those that would be pro-
Coverage exhaustive
vided by the originators of the arguments or evidence. At
exhaustive with selective citation
representative the opposite extreme of perspective, the viewpoints of re-
central or pivotal viewers play an active role in how material is presented.
The reviewers accumulate and synthesize the literature in
Organization historical
the service of demonstrating the value of the particular
conceptual
methodological point of view that they espouse. The reviewers muster ar-
guments and evidence so that it presents their contentions
Audience specialized scholars in the most convincing manner.
general scholars
Of course, reviewers attempting to achieve complete
practitioners or policy makers
general public neutrality are likely doomed to failure. Further, reviewers
who attempt to present all sides of an argument do not
source: Cooper 1988. Reprinted with permission from Transaction preclude themselves from ultimately taking a strong posi-
Publishers. tion based on the cumulative evidence. Similarly, review-
ers can be thoughtful and fair while presenting conflicting
of fact by proposing a new conception that accounts evidence or opinions and still advocate for a particular
for the inconsistency; and bridging the gap between con- interpretation.
cepts or theories by creating a new, common linguistic The next characteristic, coverage, concerns the extent
framework. to which reviewers find and include relevant works in their
Another goal for literature reviews can be to critically paper. It is possible to distinguish at least four types of
analyze the existing literature. Unlike a review that seeks coverage. The first type, exhaustive coverage, suggests
to integrate existing work, a review that involves a critical that the reviewers hope to be comprehensive in the presen-
assessment does not necessarily summate conclusions or tation of the relevant work. An effort is made to include
compare the covered works one to another. Instead, it the entire literature and to base conclusions and discus-
holds each work up against a criterion and finds it more sions on this comprehensive information base. The second
or less acceptable. Most often, the criterion will include type of coverage also bases conclusions on entire litera-
issues related to the methodological quality if empirical tures, but only a selection of works is actually described in
studies, the logical rigor, completeness or breadth of ex- the literature review. The authors choose a purposive sam-
planation if theories are involved, or comparison with the ple of works to cite but claim that the inferences drawn are
ideal treatment, when practices, policies or applications based on a more extensive literature. Third, some review-
are involved. ers will present works that are broadly representative of
A third goal that often motivates literature reviews is to many other works in a field. They hope to describe just
identify issues central to a field. These issues may include a few exemplars that are descriptive of numerous other
questions that have given rise to past work, questions that works. The reviewers discuss the characteristics that make
6 INTRODUCTION

the chosen works paradigmatic of the larger group. In the empirical research for the purpose of creating generaliza-
final coverage strategy, reviewers concentrate on works tions. Implicit in this definition is the notion that seeking
that were highly original when they appeared and influ- generalizations also involves seeking the limits of gener-
enced the development of future efforts in the topic area. alizations. Also, research syntheses almost always pay
This may include materials that initiated a line of investi- attention to relevant theories, critically analyze the re-
gation or thinking, changed how questions were framed, search they cover, try to resolve conflicts in the literature,
introduced new methods, engendered important debate, or and attempt to identify central issues for future research.
performed a heuristic function for other scholars. According to Derek Price, research syntheses are in-
A fifth characteristic of literature reviews concerns how tended to “replace those papers that have been lost from
a paper is organized. Reviews may be arranged histori- sight behind the research front” (1965, 513). Research
cally, so that topics are introduced in the chronological synthesis is one of a broad array of integrative activities
order in which they appeared in the literature, conceptu- that scientists engage in; as demonstrated in the quote
ally, so that works relating to the same abstract ideas leading this chapter, its intellectual heritage can be traced
appear together, or methodologically, so that works em- back at least as far as Aristotle.
ploying similar methods are grouped together. Using the taxonomy described, we can make further
Finally, the intended audiences of reviews can vary. specifications concerning the type of research syntheses
Reviews can be written for groups of specialized research- that are the focus of this book. With regard to perspective,
ers, general researchers, policy makers, practitioners, or readers will note that much of the material is meant to
the general public. As reviewers move from addressing help synthesists produce statements about evidence that
specialized researchers to addressing the general public, are neutral in perspective, that is, less likely to be affected
they employ less technical jargon and detail, while often by bias or by their own subjective outlooks. The material
paying greater attention to the implications of the work on searching the literature for evidence is meant to help
being covered. synthesists uncover all the evidence, not simply positive
studies (that might be overrepresented in published re-
1.2.2. A Definition of Research Synthesis search) or evidence that is easy for them to find (that
might be overly sympathetic to their point of view or dis-
The terms research synthesis, research review, and sys- ciplinary perspective). The material on the reliability of
tematic review are often used interchangeably in the so- extracting information from research reports and how
cial science literature, though they sometimes connote methodological variations in research should be handled
subtly different meanings. Regrettably, there is no con- is meant to increase the interjudge reliability of these pro-
sensus about whether these differences are really mean- cesses. The methods proposed for the statistical integra-
ingful. Therefore, we will use the term research synthesis tion of findings is meant to ensure the same rules about
most frequently throughout this book. The reason for this data analysis are consistently applied to all conclusions
choice is simple. In addition to its use in the context of about what is and is not significant, both statistically and
research synthesis, the term research review is used as practically. Finally, the material on explicit and exhaus-
well to describe the activities of evaluating the quality of tive reporting of the methods used in syntheses is meant
research. For example, a journal editor will obtain re- to assist users in evaluating if or where subjectivity may
search reviews when deciding whether to publish a man- have crept into the process and to replicate findings for
uscript. Because research syntheses often include this themselves if they choose to do so.
type of evaluative review of research, using the term re- Finally, the term meta-analysis often is used as a syn-
search synthesis avoids confusion. The term systematic onym for research synthesis. However, in this volume, it
review is less often used in the context of research evalu- will be used in its more precise and original meaning—to
ation, though the confusion is still there, and also it is describe the quantitative procedures that a research syn-
a term less familiar to social scientists than is research thesist may use to statistically combine the results of
synthesis. studies. Gene Glass coined the term to refer to “the statis-
A research synthesis can be defined as the conjunction tical analysis of a large collection of analysis results from
of a particular set of literature review characteristics. Most individual studies for the purpose of integrating the find-
definitional about research syntheses are their primary ings” (1976, 3). The authors of the Handbook reserve the
focus and goal: research syntheses attempt to integrate term to refer specifically to statistical analysis in research
 RESEARCH SYNTHESIS AS A SCIENTIFIC PROCESS 7

synthesis and not to the entire enterprise of research syn- the century (1990). For example, Pearson took the aver-
thesis. Not all research syntheses are appropriate for age of estimates from five separate samples of the corre-
meta-analysis. lation between inoculation for enteric fever and mortality
(1904). He used this average to better estimate the typical
1.3. A BRIEF HISTORY OF RESEARCH SYNTHESIS effect of inoculation and to compare this effect with that
AS A SCIENTIFIC ENTERPRISE of inoculation for other diseases. Early work on the meth-
odology for combination of estimates across studies in-
1.3.1 Early Developments cludes papers in the physical sciences by Raymond Birge
(1932) and in statistics by William G. Cochran (1937),
In 1971, Kenneth Feldman demonstrated remarkable pre- Frank Yates and Cochran (1938), and Cochran (1954).
science: “Systematically reviewing and integrating . . . Methods for combining probabilities across studies
the literature of a field may be considered a type of re- also have a long history, dating at least from procedures
search in its own right—one using a characteristic set of suggested in Leonard Tippett’s The Method of Statistics
research techniques and methods” (86). He described (1931) and Ronald Fisher’s Statistical Methods for
four steps in the synthesis process: sampling topics and Research Workers (1932). The most frequently used
studies, developing a scheme for indexing and coding probability-combining method was described by Freder-
material, integrating the studies, and writing the report. ick Mosteller and Robert Bush in the first edition of the
The same year, Richard Light and Paul Smith (1971) Handbook of Social Psychology more than fifty years ago
presented what they called a cluster approach to research (1954).
synthesis, meant to redress some of the deficiencies in the Still, the use of quantitative synthesis techniques in
existing strategies for integration. They argued that if the social sciences was rare before the 1970s. Late in
treated properly the variation in outcomes among related that decade, several applications of meta-analytic tech-
studies could be a valuable source of information—rather niques captured the imagination of behavioral scientists:
than merely a source of consternation as it appeared to be in clinical psychology, Mary Smith and Gene Glass’s
when treated with traditional cumulating methods. meta-analysis of psychotherapy research (1977); in
Three years later, Thomas Taveggia struck a comple- industrial-organizational psychology, Frank Schmidt and
mentary theme: John Hunter’s validity generalization of employment
tests; in social psychology, Robert Rosenthal and Donald
A methodological principle overlooked by [synthesists] . . . Rubin’s integration of interpersonal expectancy effect re-
is that research results are probabilistic. What this principle search (1977), and, in education, Glass and Smith’s syn-
suggests is that, in and of themselves, the findings of any thesis of the literature on class size and achievement
single research are meaningless—they may have occurred (1978).
simply by chance. It also follows that, if a large enough
number of researches has been done on a particular topic,
chance alone dictates that studies will exist that report in- 1.3.2 Research Synthesis Comes of Age
consistent and contradictory findings! Thus, what appears
to be contradictory may simply be the positive and negative Two papers that appeared in the Review of Educational
details of a distribution of findings. (1974, 397–98) Research in the early 1980s brought the meta-analytic
and synthesis-as-research perspectives together. The first
Taveggia went on to describe six common problems in proposed six synthesis tasks “analogous to those per-
research synthesis: selecting research; retrieving, index- formed during primary research” (Jackson 1980, 441).
ing, and coding information from studies; analyzing the Gregg Jackson portrayed meta-analysis as an aid to the
comparability of findings; accumulating comparable task of analyzing primary studies but emphasized its lim-
findings; analyzing distributions of results, and; reporting itations as well as its strengths. Also noteworthy about his
of results. paper was his use of a sample of thirty-six articles from
The development of meta-analytic techniques extends prestigious social science periodicals to examine the
back further in time but their routine use by research syn- methods used in integrative empirical syntheses. He re-
thesists is also relatively recent. Where Glass gave us the ported, for example, that only one of the thirty-six syn-
term meta-analysis in 1976, Ingram Olkin pointed out that theses reported the indexes or retrieval systems used to
ways to estimate effect sizes have existed since the turn of locate primary studies. He concluded that “relatively little
8 INTRODUCTION

thought has been given to the methods for doing integra- an independent specialty within the statistical sciences.
tive reviews. Such reviews are critical to science and This book, summarizing and expanding nearly a decade
social policy making and yet most are done far less rigor- of programmatic developments by the authors, not only
ously than is currently possible” (459). covered the widest array of meta-analytic procedures but
Harris Cooper drew the analogy between research syn- also presented rigorous statistical proofs establishing
thesis and primary research to its logical conclusion their legitimacy.
(1982). He presented a five stage model of synthesis as a Another text that appeared in 1984 also helped elevate
research project. For each stage, he codified the research research synthesis to a more rigorous level. Richard Light
question asked, its primary function in the synthesis, and and David Pillemer focused on the use of research synthe-
the procedural differences that might cause variation in ses to help decision-making in the social policy domain.
synthesis conclusions. In addition, Cooper applied the Their approach placed special emphasis on the importance
notion of threats-to-inferential-validity—which Donald of meshing both numbers and narrative for the effective
Campbell and Julian Stanley introduced for evaluating interpretation and communication of synthesis results.
the utility of primary research designs—to the conduct of Several other books appeared on the topic of meta-
research synthesis (1966; also see Shadish, Cook, and analysis during the 1980s and early 1990s. Some of these
Campbell 2002) . Cooper identified ten threats to validity treated the topic generally (for the most recent editions,
specifically associated with synthesis procedures that see, for example, Cooper 2009; Hunter and Schmidt
might undermine the trustworthiness of a research syn- 2004; Lipsey and Wilson 2001; Wolf 1986), some treated
thesis’ findings. He also suggested that other threats might it from the perspective of particular research design con-
exist and that the validity of any particular synthesis ceptualizations (for example, Mullen 1989), some were
could be threatened by consistent deficiencies in the set tied to particular software packages (for example, John-
of studies that formed its database. Table 1.2 presents a son 1993), and some looked to the future of research syn-
recent revision of this schema (Cooper 2007) that pro- thesis as a scientific endeavor (for example, Wachter and
poses a six stage model that separates the original analy- Straf 1990; Cook et al. 1992). The first edition of the
sis and interpretation stage into two distinct stages. Handbook of Research Synthesis appeared in 1994.
The first half of the 1980s also witnessed the appear- Readers interested in a popular history of meta-analysis
ance of four books primarily devoted to meta-analytic can consult Morton Hunt’s 1997 How Science Takes Stock:
methods. In 1981, Gene Glass, Barry McGaw, and Mary The Story of Meta-Analysis. In 2006, Mark Petticrew and
Smith presented meta-analysis as a new application of Helen Roberts provided a brief “unsystematic history of
analysis of variance and multiple regression procedures, systematic reviews” that includes discussion of early at-
with effect sizes treated as the dependent variable. In tempts to appraise the quality of studies and synthesize re-
1982, John Hunter, Frank Schmidt, and Greg Jackson in- sults. In 1990, Olkin did the same for meta-analysis.
troduced meta-analytic procedures that focused on com- Literally thousands of research syntheses have been
paring the observed variation in study outcomes to that published since the first edition of the Handbook in 1994.
expected by chance (the statistical realization of Taveg- Figure 1.1 presents some evidence of the increasing im-
gia’s point) and correcting observed effect size estimates pact of research syntheses on knowledge in the sciences
and their variance for known sources of bias (for exam- and social sciences. The figure is based on entries in the
ple, sampling error, range restrictions, unreliability of Science Citation Index Expanded and the Social Sciences
measurements). In 1984, Rosenthal presented a compen- Citation Index, according to the Web of Science reference
dium of meta-analytic methods covering, among other database (2006). It charts the growth in the number of cita-
topics, the combining of significance levels, effect size tions to documents including the terms research synthesis,
estimation, and the analysis of variation in effect sizes. systematic review, research review, and meta-analysis in
Rosenthal’s procedures for testing moderators of varia- their title or abstract during the years 1995 to 2005 and this
tion in effect sizes were not based on traditional inferen- number of citations divided by the total number of docu-
tial statistics, but on a new set of techniques involving ments in the two databases. The figure indicates that both
assumptions tailored specifically for the analysis of study the total number of citations and the number of citations
outcomes. Finally, in 1985, with the publication of Statis- per document in the databases has risen every year without
tical Procedures for Meta-Analysis, Hedges and Olkin exception. Clearly, the role that research syntheses play in
helped to elevate the quantitative synthesis of research to our knowledge claims is large and growing larger.
 RESEARCH SYNTHESIS AS A SCIENTIFIC PROCESS 9

Table 1.2 Research Synthesis Conceptualized as a Research Process

Stage Characteristics

Stage Research Question Primary Function Procedural Variation

Define the Problem What research evidence will Define the variables and Variation in the conceptual
be relevant to the problem relationships of interest so breadth and detail of
or hypothesis of interest in that relevant and irrelevant definitions might lead to
the synthesis? studies can be differences in the research
distinguished operations deemed
relevant and/or tested as
moderating influences
Collect the Research Evidence What procedures should be Identify sources (e.g., Variation in searched sources
used to find relevant reference databases, and extraction procedures
research? journals) and terms used might lead to systematic
to search for relevant differences in the retrieved
research and extract research and what is
information from reports known about each study
Evaluate the Correspondence What retrieved research Identify and apply criteria to Variation in criteria for
between Methods and should be included or separate correspondent decisions about study
Implementation of Studies excluded from the from incommensurate inclusion might lead to
and the Desired Synthesis synthesis based on the research results systematic differences in
Inferences suitability of the methods which studies remain in
for studying the synthesis the synthesis
question or problems in
research implementation?
Analyze (Integrate) the What procedures should be Identify and apply Variation in procedures used
Evidence from Individual used to summarize and procedures for combining to analyze results of
Studies integrate the research results across studies and individual studies
results? testing for differences in (narrative, vote count,
results between studies averaged effect sizes) can
lead to differences in
cumulative results
Interpret the Cumulative What conclusions can be Summarize the cumulative Variation in criteria for
Evidence drawn about the research evidence with labeling results as
cumulative state of the regard to its strength, important and attention to
research evidence? generality, and limitations details of studies might
lead to differences in
interpretation of findings
Present the Synthesis Methods What information should be Identify and apply editorial Variation in reporting might
and Results included in the report of guidelines and judgment lead readers to place more
the synthesis? to determine the aspects of or less trust in synthesis
methods and results outcomes and influences
readers of the synthesis others ability to
report need to know replication results

source: Cooper 2007.

10 INTRODUCTION

4,000 0.30%

3,500
0.25%

Citations per Document

3,000
Number of Citations

0.20%
2,500

(lines)
(bars)

2,000 0.15%

1,500
0.10%
1,000
0.05%
500

0 0.00%
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
Year

Figure 1.1 Citations to Articles, per Document

source: Authors’ compilation.
note. Based on entries in the Science Citation Index Expanded and the Social Sciences Citation Index, according to the Web of Science
reference database (retrieved June 28, 2006). Bars chart the growth in the number of citations to documents including the terms research
synthesis, systematic review, research review or meta-analysis in their title or abstract during the years following the publication of the first
edition of the Handbook of Research Synthesis. Lines chart this number of citations divided by the total number of documents in the two
databases.

In the past two decades the use of research synthesis ([Link] was begun with
has spread from psychology and education through many similar objectives for the domain of social policy analy-
disciplines, especially the medical sciences and social sis, focusing initially on policies concerning education,
policy analysis. Indeed, the development of scientific social welfare, and crime and justice.
methods for research synthesis has its own largely inde- Because of the efforts of scholars who chose to apply
pendent history in the medical sciences (see Chalmers, their skills to how research syntheses might be improved,
Hedges, and Cooper 2002). A most notable event in syntheses written since the 1980s have been held to stan-
medicine was the establishment of the U.K. Cochrane dards far more demanding than those applied to their pre-
Centre in 1992. The Centre was meant to facilitate the decessors. The process of elevating the rigor of syntheses
creation of an international network to prepare and main- is certain to continue into the twenty-first century.
tain systematic reviews of the effects of interventions
across the spectrum of health care practices. At the end 1.3.3 Rationale for the Handbook
of 1993, an international network of individuals, called
the Cochrane Collaboration ([Link] The Handbook of Research Synthesis and Meta-Analysis
[Link]), emerged from this initiative (Chalmers 1993; is meant to be the definitive volume for behavioral and
Bero and Rennie 1995). By 2006, the Cochrane Collabo- social scientists intent on applying the synthesis craft. It
ration was an internationally renowned initiative with distills the products of thirty years of developments in
11,000 people contributing to its work, in more than how research integrations should be conducted so as to
ninety countries. It is now the leading producer of re- minimize the chances of conclusions that are not truly re-
search syntheses in health care and is considered by flective of the cumulated evidence. Research synthesis in
many to be the gold standard for determining the effec- the 1960s was at best an art, at worst a form of yellow
tiveness of different health care interventions. Its library journalism. Today, the summarization and integration of
of systematic reviews numbers in the thousands. In studies is viewed as a research process in its own right, is
2000, an initiative called the Campbell Collaboration held to the standards of a scientific endeavor, and entails
 RESEARCH SYNTHESIS AS A SCIENTIFIC PROCESS 11

the application of data gathering and analyses techniques syntheses require considerably more than just the appli-
developed for its unique purpose. cation of quantitative procedures.
Numerous excellent texts on research synthesis exist. Finally, the handbook is meant for those who carry out
None, however, are as comprehensive as this volume. research syntheses. Discussions of theory and proof are
Some texts focus on statistical methods. These often kept to a minimum in favor of descriptions of the practi-
emphasize different aspects of statistical integration (for cal mechanics needed to apply well the synthesis craft.
example, combining probabilities, regression-analog mod- The chapters present multiple approaches to problem
els, estimating population effects from sampled effects solving and discuss the strengths and weaknesses of each
with known biases) and often approach research accumu- approach. Readers with a comfortable background in
lation from different perspectives. Although these texts analysis of variance and multiple regression and who
are complete within their domains, no single sourcebook have access to a research library should find the chapters
describes and integrates all the meta-analytic approaches. accessible. The Handbook authors want to supply work-
This volume incorporates quantitative statistical tech- ing synthesists with the needed expertise to interpret their
niques from all the synthesis traditions. It brings the lead- blueprints, to wield their mortar hoe and trowel.
ing authorities on the various meta-analytic perspectives
together in a single volume. As such, it is an explicit 1.4 STAGES OF RESEARCH SYNTHESIS
statement by its authors that all the statistical approaches
share a common assumptive base. The common base is The description of the stages of research synthesis pre-
not only statistical but also philosophical. Philosophi- sented in table 1.2 provides the conceptual organization
cally, all the approaches rest on the presupposition that of the handbook. In this section, we raise the principal is-
research syntheses need to be held to the same standards sues associated with each stage. This allows us to briefly
of rigor, systematicity, and transparency as the research introduce the content of each of the chapters that follow.
on which they are based. The second and later users of
data must be held as accountable for the validity of their 1.4.1 Problem Formulation
methods as were the first.
Several problems arising in the course of conducting a Formulating a problem in research synthesis is con-
quantitative synthesis have not received adequate treat- strained by one major factor: primary research on a topic
ment in any existing text. These include nonindependence must exist before a synthesis can be conducted. How
of data sets, synthesis of multivariate data sets, and sensi- much research? If the research question is important, it
tivity analysis, to name just a few. Every research syn- would be interesting to know how much research there is
thesist faces these problems, and strategies have been on the problem, even if the answer was none at all. The
developed for dealing with them. Some of their solutions methods of meta-analysis can be applied to literatures
are published in widely scattered journals, others are containing as few as two hypothesis tests. Under certain
passed on to colleagues through informal contacts. They circumstances—for instance, researchers synthesizing a
have never received complete treatment within the same pair of replicate studies from their own lab—the use of
text. The Handbook brings these topics together for the meta-analysis in this fashion might be legitimate. Yet,
first time. most scientists would argue that the benefits of such a
Further, texts focusing on the statistical aspects of inte- synthesis would be limited (and its chances for publica-
gration tend to give only passing consideration to other tion even more limited).
activities of research synthesis. These activities include: A more general answer to the “how much research”
the unique characteristics of problem formulation in re- question is that it varies depending on a number of char-
search synthesis, methods of literature search, coding and acteristics of the problem. All else being equal, conceptu-
evaluation of research reports, and the meaningful inter- ally broad topics would seem to profit from a synthesis
pretation and effective communication of synthesis re- only after the accumulation of a more varied and larger
sults. The existing texts that focus on these aspects of number of studies than would narrowly defined topics.
research synthesis tend not to be comprehensive in their Similarly, literatures that contain diverse types of opera-
coverage of statistical issues. Fully half of the chapters in tions also would seem to require a relatively large number
this volume deal with issues that are not statistical, evi- of studies before relatively firm conclusions could be
dencing the authors’ collective belief that high quality drawn from a synthesis. Ultimately, the arbiter of whether
12 INTRODUCTION

a synthesis is needed will not be numerical standards, but About one-third said they stopped their search when they
the fresh insights a synthesis can bring to a field. Indeed, felt their understanding and conclusions about the topic
although a meta-analysis cannot be performed without would not be affected by additional material. About one-
data, many social scientists see value in empty syntheses sixth said they stopped searching when retrieval became
that point to important gaps in our knowledge. When unacceptably difficult.
done properly, empty syntheses should have proceeded In contrast to the relatively well-defined sampling
through the stages of research synthesis, including careful frames available to primary researchers, literature search-
problem formulation and a thorough literature search. ers confront the fact that any single source of primary re-
Once enough literature on a problem has collected, ports will lead them to only a fraction of the relevant
the challenge—and promise—of research synthesis be- studies, and a biased fraction at that. For example, among
comes evident. The problems that constrain primary the most egalitarian sources of literature are the reference
researchers—small and homogeneous samples, limited databases, such as PsycINFO, ERIC, and Medline. Still,
time and money for turning constructs of interest into these broad, non-evaluative systems exclude much of the
operations—are less severe for synthesists. They can unpublished literature. Conversely, the least equitable
capitalize on the diversity in methods that has occurred literature searching technique involves accessing one’s
naturally across primary studies. The heterogeneity of close colleagues and other researchers with an active in-
methods across studies may permit tests of theoretical terest in the topic area. In spite of the obvious biases,
hypotheses concerning the moderators and mediators of there is no better source of unpublished and recent works.
relations that have never been tested in any single pri- Further complicating the sampling frame problem for
mary study. Conclusions about the population and eco- synthesist is the fact that the relative utility and biases as-
logical validity of relations uncovered in primary research sociated with any single source will vary as a function of
may also receive more thorough tests in syntheses. characteristics of the research problem, including for ex-
Part II of the handbook focuses on issues in problem ample how long the topic has been the focus of study and
formulation. In chapter 2, Harris Cooper discusses in de- whether the topic is interdisciplinary in nature.
tail the issues mentioned above. In chapter 3, Larry These problems imply that research synthesists must
Hedges looks at the implications of different problem carefully consider multiple channels for accessing litera-
definitions for how study results will be statistically mod- ture and how the channels they choose complement one
eled. The major issues involve the populations of people another. The three chapters in part III are devoted to help-
and measurements that are the target of a synthesis’ infer- ing the synthesist consider and carry out this unique task.
ences; how broadly the key constructs are defined, espe- In chapter 4, Howard White presents an overview of
cially in terms of whether fixed or random effect models searching issues from the viewpoint of an information
are envisioned; and how choices among models influence scientist. Both chapter 5, by Jeffrey Reed and Pam Bax-
the precision of estimates and the statistical power of ter, and chapter 6, by Hannah Rothstein and Sally
meta-analytic tests. Hopewell, present in detail the practical considerations of
how to scour our brickyards and quarries.
1.4.2. Literature Search
1.4.3 Data Evaluation
The literature search is the stage of research synthesis
that is most different from primary research. Still, culling Once the synthesists have gathered the relevant literature,
through the literature for relevant studies is not unlike they must extract from each document those pieces of in-
gathering a sample of primary data. The target of a litera- formation that will help answer the questions that impel
ture search that is part of a synthesis attempting exhaus- research in the field. The problems faced during data cod-
tive coverage would be all the research conducted on the ing provide a strong test of the synthesist’s competence,
topic of interest. thoughtfulness, and ingenuity. The solutions found for
Cooper found that about half of the reviewers he sur- these problems will have considerable influence on the
veyed claimed that they conducted their search with the contribution of the synthesis. Part IV contains four chap-
intention of identifying all or most of the relevant litera- ters on data coding and evaluation.
ture (1987). About three-quarters said they did not stop The aspect of coding studies that engenders the
searching until they felt they had accomplished their goal. most debate involves how synthesists should represent
 RESEARCH SYNTHESIS AS A SCIENTIFIC PROCESS 13

differences in the design and implementation of primary usually involved intuitive processes taking place inside
studies. What is meant by quality in evaluating research the heads of the synthesists. Meta-analysis made these
methods? Should studies be weighted differently if they processes public and based them on explicit, shared, sta-
differ in quality? Should studies be excluded if they con- tistical assumptions (however well these assumptions
tain too many flaws? How does one rate the quality of were met). We would not accept as valid a primary re-
studies described in incomplete research reports? In searcher’s conclusion if it were substantiated solely by
chapter 7, Jeff Valentine examines the alternative ap- the statement “I looked at the treatment and control scores
proaches available to synthesists for representing primary and I think the treated group did better.” We would de-
research methodology. Beyond quality judgments, syn- mand some sort of statistical test (for example, a t-test) to
thesists make decisions about the classes of variables back up the claim. Likewise, we no longer accept “I ex-
that are of potential interest to them. These can relate to amined the study outcomes and I think the treatment is
variables that predict outcomes, potential mediators of ef- effective” as sufficient warrant for the conclusion of a re-
fects, and the differences in how outcomes are conceptu- search synthesis.
alized (and, therefore, measured). If a synthesist chooses Part V contains chapters that cover the components of
not to code a particular feature of studies, then it cannot synthesis dealing with combining study results. Brad
be considered in the analysis of results. General guide- Bushman and Morgan Wang, in chapter 11, describe pro-
lines for what information should be extracted from pri- cedures based on viewing the outcomes of studies dichot-
mary research reports are difficult to develop, beyond omously, as either supporting or refuting a hypothesis.
those that are very abstract. Instead, direction will come The next two chapters cover methods for estimating the
from the issues that arise in the particular literature, cou- magnitude of an effect, commonly called the effect size.
pled with the synthesist’s personal insights into the topic. Cohen defined an effect size as “the degree to which the
Still, there are commonalities that emerge in how these phenomenon is present in the population, or the degree to
decisions are forged. Mark Lipsey, in chapter 8 (“Identi- which the null hypothesis is false” (1988, 9–10). In chap-
fying Variables and Analysis Opportunities”), and David ter 12, Michael Borenstein describes the parametric mea-
Wilson, in chapter 9 (“Systematic Coding”), present sures of effect size. In chapter 13, Joseph Fleiss and Jesse
complementing templates for what generally should be Berlin cover the measures of effect size for categorical
included in coding frames. data, paying special attention to the estimate of effect
Once decisions on what to code have been made, syn- based on rates and ratios.
thesists needs to consider how to carry out the coding of To most research synthesists, the search for influences
the literature and how to assess the trustworthiness with on study results is the most exciting and rewarding part of
which the coding frame is implemented. There are nu- the process. The four chapters of part VI all deal with tech-
merous indexes of coder reliability available, each with niques for analyzing whether and why there are differ-
different strengths and weaknesses. Chapter 10, by Rob- ences in the outcomes of studies. As an analog to analysis
ert Orwin and Jack Vevea, describes strategies for reduc- of variance or multiple regression procedures, effect sizes
ing the amount of error that enters a synthesis during the can be viewed as dependent (or criterion) variables and
coding of the literature’s features. Orwin and Vevea’s de- the features of study designs as independent (or predic-
scription of reliability assessment focuses on three major tor) variables. Because effect-size estimates do not all
approaches: measuring interrater agreement; obtaining have the same sampling uncertainty, however, they can-
confidence ratings from coders; and conducting sensitiv- not be combined using traditional inferential statistics. In
ity analysis, or using multiple coding strategies to deter- chapter 14, William Shadish and Keith Haddock describe
mine whether a synthesis’ conclusions are robust across procedures for averaging effects and generating confi-
different coding decision rules. dence intervals around these population estimates. In
chapter 15, Spyros Konstantopoulos and Larry Hedges
1.4.4 Data Analysis discuss analysis strategies when fixed-effects models are
used. In chapter 16, Stephen Raudenbush describes the
As our brief history of research synthesis revealed, the strategies appropriate for random-effects models.
analysis of accumulated research outcomes has been a Further, effect-size estimates may be affected by fac-
separate area of specialization within statistics. Three de- tors that attenuate their magnitudes. These may include,
cades ago, the actual mechanics of integrating research for example, a lack of reliability in the measurement
14 INTRODUCTION

instruments or restrictions in the range of values in the many salutary changes, most primary researchers still do
subject sample. Attenuating biases may be estimated and not think about a research report in terms of the needs of
corrected using the procedures that Frank Schmidt, Huy the next user of the data. Also, problems with missing
Le, and In-Sue Oh describe in chapter 17. data would still occur even if the needs of the synthesist
Part VII includes three chapters that focus on some were the paramount consideration in constructing a re-
special issues in the statistical treatment of research out- port. There is simply no way primary researchers can an-
comes. In chapter 18, Larry Hedges discusses combining ticipate the subtle nuances in their methods and results
special types of studies, in particular the increasing use of that might be identified as critical to piecing together
cluster- or group-randomized trials in social research and the literature five, ten, or more years later. In chapter 21,
the implication of this for the definition of effect sizes, Theresa Piggott describes how lost information can bear
their estimation, and particularly the computation of stan- on the results and interpretation of a research synthesis.
dard errors of the estimates. Chapter 19 tackles the She shows how the impact of missing data can be as-
vexing problem of what to do when the same study (or sessed by paying attention to some of its discoverable
laboratory, or researcher, if one wishes to push the point) characteristics. She also describes both simple and com-
provides more than one effect size to a meta-analysis. plex strategies for handling missing data in research syn-
Because estimates from the same study may not be inde- thesis. Joel Greenhouse and Satish Iyengar, in chapter 22,
pendent sources of information about the underlying describe sensitivity analysis in detail, including several
population value, their contributions to any overall esti- diagnostic procedures to assess the sensitivity and robust-
mates must be adjusted accordingly. In chapter 19, Leon ness of the conclusions of a meta-analysis. Alex Sutton, in
Gleser and Ingram Olkin describe both simple and more chapter 23, examines publication bias and describes pro-
complex strategies for dealing with dependencies among cedures for identifying whether data censoring has had an
effect sizes. impact on the meta-analytic result. He also details some
Sometimes, the focus of a research synthesis will not imputation procedures that can help synthesists estimate
be a simple bivariate relation but a model involving nu- what summary effect size estimates might have been in
merous variables and their interrelations. The model the absence of the selective inclusion of studies.
could require analysis of more than one simple relation The chapters in part IX bring us full circle, returning us
and/or complex partial or second-order relations. In to the nature of the problem that led to undertaking the
chapter 20, Betsy Becker looks at what models in meta- research synthesis in the first place. Collectively, these
analysis are, why they should be used, and how they chapters attempt to answer the question as to what can be
affect meta-analytic data analysis. learned from research synthesis. In chapter 24, Wendy
Wood and Alice Eagly examine how theories can be
1.4.5 Interpretation of Results tested in research synthesis. In chapter 25, David Cordray
and Paul Morphy look at research synthesis for policy
Estimating and averaging effect sizes and searching for analysis. These chapters are meant to help synthesists
moderators of their variability is how the interpretation of think about the substantive questions for which the read-
cumulative study results begins. However, it must be fol- ers of their syntheses will be seeking answers.
lowed by other procedures that help the synthesists prop-
erly interpret their outcomes. Proper interpretation of the 1.4.6 Public Presentation
results of a research synthesis requires careful use of de-
clarative statements regarding claims about the evidence, Presenting the background, methods, results, and mean-
specification of what results warrants each claim, and any ing of a research synthesis’ findings provide the final
appropriate qualifications to claims that need to be made. challenges to the synthesists’ skill and intellect. In chap-
In part VIII, three important issues in data interpretation ter 26, the first of part X, Geoffrey Borman and Jeffrey
are examined. Grigg address the visual presentation of research synthe-
Even the most trustworthy coding frame cannot solve sis results. Charts, graphs, and tables, they explain, should
the problem of missing data. Missing data will arise in be used to summarize the numbers in a meta-analysis,
every research synthesis. Reporting of study features and along with a careful intertwining of narrative explication
results, as now practiced, leaves much to be desired. Al- to contextualize the summaries. Proper use of interpretive
though the emergence of research synthesis has led to devices can make a complex set of results easily accessible
 RESEARCH SYNTHESIS AS A SCIENTIFIC PROCESS 15

to the intelligent reader. Likewise, inadequate use of these the Effects of Health Care.” Annals of the New York Acad-
aides can render incomprehensible even the simplest con- emy of Sciences 703: 156–63.
clusions. Chapter 27 describes the reporting format for Chalmers, Iain, Larry V. Hedges, and Harris Cooper. 2002.
research synthesis. As with the coding frame, there is no “A Brief History of Research Synthesis.” Evaluation and
simple reporting scheme that fits all syntheses. However, the Health Professions 25(1): 12–37.
certain commonalities do exist and Mike Clarke discusses Cochran, William G. 1937. “Problems Arising in the Analy-
them. Not too surprisingly, the organization that emerges sis of a Series of Similar Experiments.” Journal of the
bears considerable resemblance to that of a primary re- Royal Statistical Society 4(Supplement): 102–18.
search report although, also obviously, the content differs ———. 1954. “The Combination of Estimates from Diffe-
dramatically. rent Experiments.” Biometrics 10(1, March): 101–29.
Cohen, Jacob. 1988. Statistical Power Analysis for the Beha-
1.5 CHAPTERS IN PERSPECTIVE vioral Sciences. Hillsdale, NJ: Lawrence Erlbaum.
Cook, Thomas D., Harris M. Cooper, David S. Cordray,
In part XI, the final two chapters of the handbook are Heidi Hartmann, Larry V. Hedges, Richard J. Light, Tho-
meant to bring the detailed drawings of the preceding mas Louis, and Frederick Mosteller. 1992. Meta-Analysis
chapters onto a single blueprint. In chapter 28, Georg for Explanation: A Casebook. New York: Russell Sage
Matt and Thomas Cook present an expanded analysis of Foundation.
threats to the validity of research syntheses, provide an Cooper, Harris M. 1982. “Scientific Guidelines for Conduc-
overall appraisal of how inferences from syntheses may ting Integrative Research Reviews.” Review of Educational
be restricted or faulty, and bring together many of the Research 52(2, Summer): 291–302.
concerns expressed throughout the book. ———. 1987. “Literature Searching Strategies of Integra-
Finally, chapter 29 also attempts to draw this material tive Research Reviewers: A First Survey.” Knowledge:
together. In it, Harris Cooper and Larry Hedges look Creation, Diffusion, Utilization 8: 372–83.
at some of the potentials and limitations of research syn- ———. 1988. “Organizing Knowledge Synthesis: A Taxo-
thesis. Special attention is paid to possible future devel- nomy of Literature Reviews.” Knowledge in Society 1:
opments in synthesis methodology, the feasibility and 104–26.
expense associated with conducting a sound research ———. 2009. Research Synthesis and Meta-Analysis: A
synthesis, and a broad-based definition of what makes a Step-by-Step Approach. Thousand Oaks, Calif.: Sage
research synthesis good or bad. Publications.
No secret will be revealed by stating our conclusion in ———. 2003. “Editorial.” Psychological Bulletin 129(1):
advance. If procedures for the synthesis of research are 3–9.
held to standards of objectivity, systematicity, and rigor, ———. 2007. Evaluating and Interpreting Research Syn-
then our knowledge edifice will be made of bricks and theses in Adult Learning and Literacy. Boston, Mass.: Na-
mortar. If not, it will be a house of cards. tional College Transition Network, New England Literacy
Resource Center/World Education, Inc.
1.6 REFERENCES Feldman, Kenneth A. 1971. “Using the Work of Others:
Some Observations on Reviewing and Integrating.” Socio-
Bero, Lisa, and Drummond Rennie. 1995. “The Cochrane logy of Education 4(1, Winter): 86–102.
Collaboration: Preparing, Maintaining, and Disseminating Fisher, Ronald A. 1932. Statistical Methods for Research
Systematic Reviews of the Effects of Health Care.” Journal Workers, 4th ed. London: Oliver & Boyd.
of the American Medical Association 274(24): 1935–8. Glass, Gene V. 1976. “Primary, Secondary, and Meta-Analy-
Birge, Raymond T. 1932. “The Calculation of Error by sis.” Educational Researcher 5(10): 3–8.
the Method of Least Squares.” Physical Review 40(2): Glass, Gene V., and Mary L. Smith. 1978. Meta-Analysis of
207–27. Research on the Relationship of Class Size and Achieve-
Campbell, Donald T., and Julian C. Stanley. 1966. Experi- ment. San Francisco: Far West Laboratory for Educational
mental and Quasi-Experimental Designs for Research. Research and Development.
Chicago: Rand McNally. Glass, Gene V., Barry McGaw, and Mary L. Smith. 1981.
Chalmers, Iain. 1993. “The Cochrane Collaboration: Prepar- Meta-Analysis in Social Research. Beverly Hills, Calif.:
ing, Maintaining and Disseminating Systematic Reviews of Sage Publications.
16 INTRODUCTION

Hedges, Larry V. 1987. “How Hard Is Hard Science, How Pearson, Karl. 1904. “Report on Certain Enteric Fever Inno-
Soft is Soft Science?” American Psychologist 42(5): culation Statistics.” British Medical Journal 3: 1243–6.
443–55. Petticrew, Mark, and Helen Roberts. 2006. Systematic Re-
Hedges, Larry V., and Ingram Olkin. 1985. Statistical Meth- views in the Social Sciences: A Practical Guide. Oxford:
ods for Meta-Analysis. Orlando, Fl.: Academic Press. Blackwell Publishing.
Hunt, Morton. 1997. How Science Takes Stock: The Story of Price, Derek J. de Solla. 1965. “Networks of Scientific Pa-
Meta-Analysis. New York: Russell Sage Foundation. pers.” Science 149(3683): 56–64.
Hunter, John E., and Frank L. Schmidt. 2004. Methods of Rosenthal, Robert. 1984. Meta-Analytic Procedures for So-
Meta-Analysis: Correcting Error and Bias in Research Find- cial Research. Beverly Hills, Calif.: Sage Publications.
ings, 2nd ed. Thousand Oaks, Calif.: Sage Publications. Rosenthal, Robert, and Donald B. Rubin. 1978. “Interperso-
Hunter, John E., Frank L. Schmidt, and Greg B. Jackson. nal Expectancy Effects: The First 345 Studies.” The Beha-
1982. Meta-Analysis: Cumulating Research Findings vioral and Brain Sciences 3: 377–86.
Across Studies. Beverly Hills, Calif.: Sage Publications. Schmidt, Frank L., and John E. Hunter. 1977. “Development
Jackson, Gregg B. 1980. “Methods for Integrative Reviews.” of a General Solution to the Problem of Validity Generali-
Review of Educational Research 50: 438–60. zation.” Journal of Applied Psychology 62: 529–40.
Johnson, Blair T. 1993. DSTAT: Software for the Meta- Shadish, William. R., Thomas D. Cook, and Donald T. Camp-
Analytic Review of Research. Hillsdale, N.J.: Lawrence bell. 2002. Experimental and Quasi-Experimental Designs
Erlbaum. for Generalized Causal Inference. Boston, Mass.: Houghton
Light, Richard J., and David B. Pillemer. 1984. Summing Mifflin.
Up: The Science of Reviewing Research. Cambridge, Mass.: Smith, Mary L., and Gene V. Glass. 1977. “Meta-Analysis of
Harvard University. Psychotherapy Outcome Studies.” American Psychologist
Light, Richard J., and Paul V. Smith. 1971. “Accumulating 32(9): 752–60.
Evidence: Procedures for Resolving Contradictions Among Taveggia, Thomas C. 1974. “Resolving Research Contro-
Research Studies.” Harvard Educational Review 41(4): versy Through Empirical Cumulation: Toward Reliable So-
429–71. ciological Knowledge.” Sociological Methods & Research
Lipsey, Mark W., and David B. Wilson. 2001. Practical 2(4): 395–407.
Meta-Analysis. Thousand Oaks, Calif.: Sage Publications. Tippett, Leonard Henry Caleb. 1931. The Methods of Statis-
Manten, A. A. 1973. “Scientific Literature Review.” Scho- tics. London: Williams & Norgate.
larly Publishing 5: 75–89. Wachter, Kenneth W., and Miron L. Straf, eds. 1990. The
Mosteller, Frederick, and Robert Bush. 1954. “Selected Future of Meta-Analysis. New York: Russell Sage Founda-
Quantitative Techniques.” In Handbook of Social Psycho- tion.
logy, vol. 1, Theory and Method, edited by Gardner Lind- Wolf, Frederic M. 1986. Meta-Analysis: Quantitative Meth-
zey. Cambridge, Mass.: Addison-Wesley. ods for Research Synthesis. Beverly Hills, Calif.: Sage Publi-
Mullen, Brian. 1989. Advanced BASIC Meta-Analysis. Hill- cations.
sdale, N.J.: Lawrence Erlbaum. Yates, Frank, and William G. Cochran. 1938. “The Analysis
Olkin, Ingram. 1990. “History and Goals.” In The Future of of Groups of Experiments.” Journal of Agricultural Science
Meta-Analysis, edited by Kenneth W. Wachter and Miron L. 28: 556–80.
Straf. New York: Russell Sage Foundation.