JGIM

REVIEWS
Six Persistent Research Misconceptions
Kenneth J. Rothman, DrPH1,2
1
Research Triangle Institute, Research Triangle Park, NC, USA; 2Boston University School of Public Health, Boston, MA, USA.

Scientific knowledge changes rapidly, but the concepts Misconception 1. There is a hierarchy of study designs;
and methods of the conduct of research change more randomized trials provide the greatest validity, followed by
slowly. To stimulate discussion of outmoded thinking cohort studies, with case–control studies being least reliable.
regarding the conduct of research, I list six misconcep-
tions about research that persist long after their flaws
Randomized trials, though often considered the “gold
have become apparent. The misconceptions are: 1)
There is a hierarchy of study designs; randomized trials standard” of study types, are not perfect, even in concept.
provide the greatest validity, followed by cohort studies, Furthermore, the premise that the comparative validity of
with case–control studies being least reliable. 2) An study results can be inferred from the type of study is wrong.
essential element for valid generalization is that the Although some believe that evidence from a randomized
study subjects constitute a representative sample of a trial is as compelling as a logical proof, no empirical finding
target population. 3) If a term that denotes the product can provide absolute certainty. If randomized trials were
of two factors in a regression model is not statistically
perfect, how could they give divergent results? In fact, they
significant, then there is no biologic interaction between
those factors. 4) When categorizing a continuous vari- are subject to various errors.1 Obviously there is random
able, a reasonable scheme for choosing category cut- error, as one would expect from a study based on random
points is to use percentile-defined boundaries, such as assignment. But there is also systematic error, or bias. For
quartiles or quintiles of the distribution. 5) One should example, randomized trials are usually analyzed using the
always report P values or confidence intervals that have “intent to treat” principle, which compares the groups that
been adjusted for multiple comparisons. 6) Significance are initially assigned by randomization, regardless of any
testing is useful and important for the interpretation of
subsequent non-adherence. Non-adherence results in under-
data. These misconceptions have been perpetuated in
journals, classrooms and textbooks. They persist be-
estimation of any treatment effect. This bias is usually
cause they represent intellectual shortcuts that avoid considered acceptable because it is outweighed by the
more thoughtful approaches to research problems. I advantages achieved by random assignment. Underestima-
hope that calling attention to these misconceptions will tion of effects, however, is not acceptable in a safety trial
spark the debates needed to shelve these outmoded aimed at uncovering adverse effects of the treatment.
ideas for good. Another important source of bias in a randomized trial
comes from errors in assessing the outcome, such as
KEY WORDS: study design; data interpretation; epidemiologic methods; undercounting of outcome events. Also, even if randomi-
representativeness; evaluation of interaction; multiple comparisons; zation provides a balance of risk factors between groups at
percentile boundaries; statistical significance testing. the start of the trial, with extended follow-up, the study
J Gen Intern Med 29(7):1060–4
DOI: 10.1007/s11606-013-2755-z
groups may become progressively imbalanced through
© The Author(s) 2014. This article is published with open access at differential attrition or changes in risk factor distributions.
[Link] With long-term trials, the benefits of random assignment
may therefore fade with time.
In short, trials are far from perfect. Furthermore, both
surprising number of misconceptions persist in the cohort and case–control studies will yield valid results when
A conduct of research involving human subjects. Some properly designed and carried out. Therefore, mindlessly
persist despite teachings to the contrary, and some because ascribing greater validity to a study based on a hierarchy of
of teachings that should be to the contrary. To spark designs2,3 is fallacious. For example, the relation between
discussion of these issues, I list here six persistent research cigarette smoking and lung cancer is well established, based
misconceptions, and offer a capsule summary of the on findings from cohort and case–control studies. The
problems with each of them. connection was never shown clearly in a randomized trial. It
is not easy to assign people randomly to smoke or not
Received November 01, 2013
Revised November 27, 2013
smoke; however, when smoking cessation was studied as
Accepted December 18, 2013 part of a multi-pronged intervention in the randomized
Published online January 23, 2014 Multiple Risk Factor Intervention Trial,4 those who were

1060
JGIM Rothman: Six Persistent Research Misconceptions 1061

urged to cease smoking actually developed more lung whereas a poorly designed trial can be unreliable. The
cancer than those who did not receive the cessation type of study should not be taken as a guide to a study’s
encouragement. The results of the trial did not overthrow validity.
the findings of the many cohort and case–control studies
Misconception 2. An essential element of making valid
conducted without randomization. Rather, the discrepancy
generalizations from a study is that the study subjects
was ascribed to problems with the trial.
constitute a representative sample of a target population.
In another high-profile example, results from large cohort
studies5,6 indicated that risk of coronary heart disease was This misconception is tied to the view that scientific
reduced among postmenopausal hormone users, but later generalization involves the mechanical extrapolation of
results from two randomized trials indicated either no results from a sample to its source population. But that
association or an increased risk.7,8 The reaction in the describes statistical generalization; scientific generalization
scientific community and the popular press9 was to discredit is different: it is the process of constructing a correct
the results from the cohort studies, presuming that they had statement about the way nature works.
been refuted by the randomized trials. Many continue to Scientific generalization is the ultimate goal of scientific
believe that interpretation, but in an elegant reanalysis, inquiry, but a prerequisite is designing a study that has
Hernan et al.10 showed that the study populations in the internal validity, which is enhanced by keeping all
cohort studies and the randomized trials were different, and disturbing variables constant. When have we heard of
that the effects of postmenopausal hormone use varied animal researchers who seek a statistically representative
greatly according to age and time since menopause. When sample of animals? Instead, their operating principle is
studies were restricted to new users of hormones, Hernan et nearly the opposite of seeking representativeness. Thus,
al. showed that differences in the distribution of age and biologists studying mice prefer to study mice that are
time since menopause could explain all of the apparent homogeneous with respect to genes and environment, and
discrepancies. Although it is common to ascribe such that differ only in respect to the experimentally manipulated
discrepancies to inherent weaknesses of the nonexperimen- variable. Unlike the statistical generalization of opinion
tal studies, it is simplistic to assign validity based on a polls or survey sampling, which merely calls for extrapo-
presumed hierarchy of study types.11 lation from sample to source population, scientific general-
Similarly, discrepancies between cohort studies and case– ization proceeds by informed guesses, but only from the
control studies should not be explained away superficially secure platform of a valid study. Consequently, studies are
by a presumed validity advantage for cohort studies over stronger if they limit variability of confounding factors, as
case–control studies. Properly designed case–control studies opposed to seeking representativeness. Doll and Hill14
will produce the same results as properly designed cohort studied the mortality of male British physicians in relation
studies. When conflicts arise, they could stem from to their smoking habits. Their findings were considered
problems in either or both types of study. Although case– broadly generalizable despite the fact that their study
control studies have long been disparaged as being population was unrepresentative of the general population
backwards versions of cohort studies, starting from disease of tobacco users with regard to sex, race, ethnicity, social
and tracing back to possible causes, epidemiologists today class, nationality and many other variables.
understand case–control studies to be conceptually identical When there is a legitimate question about whether an
to cohort studies, apart from an efficiency gain that comes overall association varies by subgroup of some third
from sampling the denominators rather than conducting a variable, such as age or ethnic group, it may be necessary
complete census. Indeed, the efficiency gain may allow to include people drawn from a broad range of values of
more resources for exposure assessment or case validation that third variable, but even then it is counterproductive for
in case–control studies, resulting in less bias than in the study population to be representative of the source
corresponding cohort studies of the same relation. population for that variable. The goal in that case would be
Those who view case–control studies as backwards to include study subjects distributed evenly across the
versions of cohort studies sometimes make the false range, or in a distribution that enhances overall study
analogy that the controls should closely resemble the cases, efficiency. A sample that is representative of the source
except that they lack the case-defining disease. In fact, the population will be suboptimal.15,16
control group in a case–control study is intended to be a
Misconception 3. If a term that denotes the product of
sample of the population denominator that gives rise to the
two factors in a regression model is not statistically
cases, a substitute for the full denominators obtained in a
significant, then there is no biologic interaction between
cohort study. Thus, the control group should resemble the
those factors.
entire study population, rather than the cases.12,13 When
properly designed, case–control studies can achieve the “Biologic” is meant here broadly, to encompass biochem-
same excellent validity as properly designed cohort studies, ical, psychological, behavioral and physical interactions. The
1062 Rothman: Six Persistent Research Misconceptions JGIM

problem is that interaction is usually evaluated through appears attributable to biologic interaction, despite the
regression models, in which the product term addresses authors’ claim that there was no interaction.
statistical interaction rather than biologic interaction.
Misconception 4. When categorizing a continuous variable,
Biologic interaction refers to two or more causes acting
a reasonable scheme for choosing category cut-points is to
in the same mechanism, with effects that are mutually
use percentile-defined boundaries, such as quartiles or
dependent. It describes a state of nature. If basic effects are
quintiles of the distribution.
measured as changes in disease risk, synergistic (i.e.
positive) biologic interaction is present when the joint There are two reasons why using percentiles is a poor
effect of two causal factors is more than the sum of their method for choosing category boundaries. First, these bound-
effects acting separately.17 In contrast, statistical interaction aries may not correspond to the parts of the distribution where
does not describe nature; it describes a mathematical model. biologically meaningful changes occur. Suppose you were
It is typically assessed with a product term for two variables conducting a study of vitamin C intake and scurvy risk in the
in a regression model. Its magnitude depends on the choice U.S. If you decided to categorize vitamin C intake by quintiles,
of measures and scale of measurement. Statistical interac- you would find that the entire relation between vitamin C
tion implies only that the basic functional form of a specific consumption and scurvy was confined to the lowest quintile,
mathematical model is not an apt description of the relation and within that category, to only a small proportion of people
among variables. Two factors that show biologic interaction who were outliers in their low vitamin C intake. 10 mg/day of
may or may not exhibit statistical interaction, depending on vitamin C can prevent scurvy, but those consuming less than
the model used. that represent a fraction of 1 % of the population in the U. S.20
Product terms in regression models have units that can Using percentile-based categories would make it impossible to
defy interpretation. If one variable is fat consumption, find the effect of inadequate vitamin C intake on scurvy risk,
measured in grams per day, and another variable is pack- because all intake above 10 mg/d is essentially equivalent. If
years of cigarettes smoked, what is the interpretation of a we routinely use percentile cut-points, we may not know if we
variable that has units of grams/day multiplied by pack- are facing the same problem as we would face in the study of
years? The challenge of interpreting such product term vitamin C and scurvy. A more effective alternative would be to
coefficients has fostered a focus on the p value accompa- begin with many narrow categories, merging neighboring
nying the coefficient, rather than the magnitude of the categories until meaningful breaks in risk become evident.
coefficient itself. Focusing on the pvalue, or on whether the The second problem with percentile-based categories is the
coefficient of a product term is statistically significant, only difficulty in comparing results across studies, because catego-
worsens the problem of mistaking statistical interaction for ries across studies using percentile category boundaries are
biologic interaction (see misconception 6). A more mean- unlikely to correspond. This problem can be averted by
ingful assessment of interaction would be to focus on the expressing boundary points in terms of the natural units of the
proportion of cases of a disease that one could attribute to variable (such as mg/d for vitamin C intake). It is also useful to
biologic interaction.17,18 report within-category means or medians.
Consider a simple example from the TREAT trial (Trial to
Misconception 5. One should always report P values or
Reduce Cardiovascular Events with Aranesp Therapy),19
confidence intervals that have been adjusted for multiple
which evaluated the risk of stroke among 4,038 patients with
comparisons.
diabetes mellitus, chronic kidney disease, and anemia ran-
domized to receive darbepoetin alfa or placebo. Among Traditional adjustments for multiple comparisons involve
patients without a history of stroke, the risk of stroke during inflating the P value or the width of a confidence interval
the study period was 2 % among patients receiving placebo according to the number of comparisons conducted. If one
and 4 % among patients receiving darbepoeitin alfa. Among is analyzing biological data that are replete with actual
patients with a history of stroke, the corresponding risks were associations, the premise for traditional adjustments is
4 % and 12 %. The authors noted that the risk increase was shaky and the adjustments are difficult to defend. The
greater for darbepoeitin alfa among those with a history of concern for multiple comparisons stems from fear of finding
stroke, but they dismissed this interaction because the product falsely significant findings (type I errors in the lingo of
term in a logistic regression model was not statistically statistics). In misconception 6, we discuss the problems
significant. The increased risk attributable to darbepoeitin alfa with using statistical significance testing for data analysis in
was 2 % in the patients without a history of stroke and 8 % the first place. But before considering those problems, let us
among patients with a history of stroke, indicating strong consider the rationale for adjusting reported results for
biologic interaction between darbepoeitin alfa and history of multiple comparisons.
stroke. If the risks were merely additive, the risk would be 6 % Despite the fact that a single significance test is intended
among those with both risk factors, instead of the actual 12 %. to have a 5 % probability (at the conventionally used level)
Thus, half of the risk among those with both risk factors of being significant when the null hypothesis is true, and
JGIM Rothman: Six Persistent Research Misconceptions 1063

therefore multiple tests when properly carried out should goal: estimation of effects is decidedly preferable to
each have this property, there is a concern that when statistical testing. Ideally, a study estimates the magnitude
making multiple tests, the probability of a spurious result is of the effect size, and analyzes the possible errors that might
increased. Of course, as the number of tests increases, the have distorted it. Systematic errors such as confounding
probability that one or more of them would be falsely from measured factors can be dealt with through analytic
positive increases, but that is only because many tests are methods; other systematic errors, such as the effects of
being conducted. Adjustments for multiple comparisons measurement error or selection bias, can be addressed
will reduce these type I errors, but they do so at the expense through sensitivity analyses (also known as bias analysis).
of increasing type II errors, which are nonsignificant test Random error is typically expressed through confidence
results in the presence of a real association. When observed intervals, giving a range of parameter values that are
associations are all the result of chance, type I errors can consistent with the data to a specified level.
occur, but type II errors cannot occur. Conversely, when the It is unfortunate that a confidence interval, from which both
observed associations all reflect actual relationships, type II an estimate of effect size and its measurement precision can be
errors can occur, but type I errors cannot. Thus, the context drawn, is typically used merely to judge whether it contains
of any analysis has fundamental implications regarding the the null value or not, thus converting it to a significance test.
interpretation of the data. In particular, it is absurd to make Significance tests are a poor classification scheme for study
adjustments that reduce type I errors at the expense of results; strong effects may be incorrectly interpreted as null
increasing type II errors without some evaluation of the findings because authors fallaciously interpret lack of statis-
estimated relative cost and frequency of each type of error. tical significance to imply lack of effect, or weak effects may
If scientists were put to work studying random numbers be incorrectly interpreted as important because they are
instead of biologic data, all the significant results they statistically significant. Rather than be used as surrogate
reported would represent type I errors, and adjustments for significance tests, confidence intervals ought to be interpreted
multiple comparisons would make sense; some skeptics as quantitative measures indicating magnitude of effect size
believe that studies of genome-wide association scans may and degree of precision, with little attention paid to the precise
approximate this situation.21 But when scientists are location of the boundaries of the confidence interval. This
studying biological relations rather than random numbers, advice is backed by the Uniform Requirements for Manu-
the premise that type I errors are the major concern may be scripts Submitted to Biomedical Journals, but nevertheless
wrong.22 A more rigorous evaluation of the need for often overlooked even by reviewers and editors whose
multiplicity adjustments would begin with an assessment journals support the requirements.29
of the tenability of the thesis that the data are essentially Many misconceptions derive from reliance on statistical
random numbers. If one is studying experiments on psychic significance testing. The focus on the statistical significance
phenomena, skepticism about the results might lend support of interaction terms instead of measuring interaction, as
to multiplicity adjustments. If one is studying physiologic discussed above, is one example. The evaluation of dose–
effects of pharmaceutical agents, real associations are to be response trends simply by declaring that there is or is not a
expected and the adjustments are more difficult to defend. significant trend, rather than expressing the magnitude and
Studying single nucleotide polymorphisms in relation to a ideally the shape of that trend, is another. Yet another is the
given disease might be a middle ground. One approach to advice sometimes offered to calculate the power of a study
this issue that is theoretically more defensible is a Bayesian when reporting results, especially if those results are not
approach, which assigns prior credibility to various levels statistically significant. Reporting the power of a study as
of association and adjusts by using Bayes’ theorem to part of the results is called “post-hoc” power calculation.30
calculate posterior credibility.23,24 Power calculations are based on a hypothesis about the
level of association that is to be distinguished from a null
Misconception 6. Significance testing is useful and impor-
association, but when the study results are on hand, there is
tant for the interpretation of data.
no longer any need to hypothesize about the magnitude of
Significance testing has led to far more misunderstanding the association, because you now have an estimate of it. A
and misinterpretation than clarity in interpreting study confidence interval for the estimated association conveys all
results.25–28 A significance test is a degraded version of the relevant information; nothing further is to be gained
the P value, a statistic that blends precision with effect size, from a power calculation.
thus confusing two essential aspects of data interpretation. The unfortunate consequence of the focus on statistical
Measuring effect size and its precision as separate tasks is a significance testing has been to foster a dichotomous view
more direct and clearer approach to data interpretation. of relationships that are better assessed in quantitative
For research studies that aim to measure associations, and terms. This distinction is more than a nicety. Every day
infer whether they reflect causal connections, focusing on there are important, regrettable and avoidable misinterpre-
the magnitude of these associations ought to be the primary tations of data that results from the confusing fog of
1064 Rothman: Six Persistent Research Misconceptions JGIM

5. Grodstein F, Manson JE, Colditz GA, et al. A prospective, observation-

statistical significance testing. Most of these errors could be
al study of postmenopausal hormone therapy and primary prevention of
avoided if the focus were shifted from statistical testing to cardiovascular disease. Ann Intern Med. 2000;133:933–41.
estimation. 6. Varas-Lorenzo C, García-Rodríguez LA, Pérez-Gutthann S, et al.
Hormone replacement therapy and incidence of acute myocardial
infarction. Circulation. 2000;101:2572–8.
7. Hulley S, Grady D, Bush T, Furberg C, Herrington D, Riggs B,

CONCLUSION Vittinghoff E. Randomized trial of estrogen plus progestin for secondary

prevention of coronary heart disease in postmenopausal women. Heart
and Estrogen/progestin Replacement Study (HERS) Research Group.
Why do such important misconceptions about research JAMA. 1998;280:605–13. doi:10.1001/jama.280.7.605.
persist? To a large extent these misconceptions represent 8. Manson JE, Hsia J, Johnson KC, et al. Estrogen plus progestin and the
substitutes for more thoughtful and difficult tasks. It is simpler risk of coronary heart disease. N Engl J Med. 2003;349:523–34.
9. Taubes G. Do we really know what makes us healthy? New York Times,
to resolve a discrepancy between a trial and a nonexperimental September 16, 2007.
study in favor of the trial, without undertaking the laborious 10. Hernán MA, Alonso A, Logan R, Grodstein F, Michels K, Willett WC,
analysis that Hernan et al. did.10 It is easy to declare that a Manson JE, Robins JM. Observational studies analyzed like random-
ized experiments: an application to postmenopausal hormone therapy
result is not statistically significant, falsely implying that there and coronary heart disease. Epidemiology. 2008;19:766–79.
is no indication of an association, rather than to consider doi:10.1097/EDE.0b013e3181875e61
11. Concato J. Observational versus experimental studies: what’s the
quantitatively the range of associations that the data actually evidence for a hierarchy? NeuroRx. 2004;1:341–7.
support. These misconceptions involve taking the low road, 12. Vandenbroucke JP, Pearce N. Case–control studies: basic concepts. Int
but when that road is crowded with others taking the same J Epidemiol. 2012;41:1480–9. doi:10.1093/ije/dys147.
13. Rothman KJ. Chapter 5, Types of Epidemiologic Studies, in Epidemiol-
path, there may be little reason to question the route. Indeed, ogy, An Introduction, 2nd Edition. Oxford University Press, New York,
these misconceptions are often perpetuated in journals, 2012.
14. Doll R, Hill AB. The mortality of doctors in relation to their smoking
classrooms and textbooks. I believe that the best prospect for habits: a preliminary report. Br Med J. 1954;ii:1451–5.
improvement is to raise consciousness about the issues, with 15. Rothman KJ, Gallacher J, Hatch EE. Why representativeness should
reasoned debate. Max Planck once said, “A new scientific be avoided. Int J Epidemiol. 2013;42:1012–4. doi:10.1093/ije/dys223
16. Rothman KJ, Gallacher J, Hatch EE. When it comes to scientific
truth does not triumph by convincing its opponents and inference, sometimes a cigar is just a cigar. Int J Epidemiol.
making them see the light, but rather because its opponents 2013;42:1026–8. doi:10.1093/ije/dyt124
17. Rothman KJ. Chapter 11, Measuring Interaction, in Epidemiology, An
eventually die, and a new generation grows up that is familiar Introduction, 2nd Edition. Oxford University Press, New York, 2012.
with it.”31 To the extent that this cynical view is correct, we 18. Knol MJ, van der Tweel I, Grobbee DE, Numans ME, Geerlings MI.
can expect to see outmoded concepts fade away slowly at best. Estimating interaction on an additive scale between continuous deter-
minants in a logistic regression model. Int J Epidemiol. 2007;36:1111–8.
I hope that calling attention to these misconceptions will spark 19. Skali H, Parving HH, Parfrey PS, Burdmann EA, Lewis EF, Ivanovich
the needed debates and be a catalyst for change. P, Keithi-Reddy SR, McGill JB, McMurray JJ, Singh AK, Solomon SD,
Uno H, Pfeffer MA. TREAT Investigators: Stroke in patients with type 2
diabetes mellitus, chronic kidney disease, and anemia treated with
Darbepoetin Alfa: the trial to reduce cardiovascular events with Aranesp
therapy (TREAT) experience. Circulation. 2011;124:2903–8.
Acknowledgements: I received helpful criticism from Susana 20. Dietary Reference Intakes for Vitamin C, Vitamin E, Selenium, and
Perez, Andrea Margulis, Manel Pladevall, and Jordi Castellsague. Carotenoids. Institute of Medicine, The National Academies Press,
Washington, D. C., 2000.
Conflict of Interest: The author declares no conflict of interest. 21. Dudbridge F, Gusnanto A. Estimation of significance thresholds for
genomewide association scans. Genet Epidemiol. 2008;32:227–34.
22. Rothman KJ. No adjustments are needed for multiple comparisons.
Corresponding Author: Kenneth J. Rothman, DrPH; Research
Epidemiology. 1990;1:43–6.
Tr i a n g l e I n s t i t u t e , R e s e a rc h Tr i a n g l e P a r k , N C , U S A
23. Greenland S, Robins J. Empirical-Bayes adjustments for multiple
(e-mail: KRothman@[Link]).
comparisons are sometimes useful. Epidemiology. 1991;2:244–51.
24. Greenland S, Poole C. Empirical-Bayes and semi-Bayes approaches to
Open Access This article is distributed under the terms of the occupational and environmental hazard surveillance. Arch Environ
Creative Commons Attribution License which permits any use, Health. 1994;48:9–16.
distribution, and reproduction in any medium, provided the original 25. Rothman KJ. A show of confidence (editorial). N Engl J Med.
author(s) and the source are credited. 1978;299:1362–3.
26. Poole C. Beyond the confidence interval. Am J Public Hlth.
1987;77:195–9.

REFERENCES 27. Rothman KJ. Significance questing (Editorial). Ann Int Med.
1986;105:445–7.
1. Hernán MA, Hernández-Díaz S, Robins JM. Randomized trials ana- 28. Gelman A, Stern H. The difference between “Significant” and “Not
lyzed as observational studies. Ann Intern Med. 2013;159:560–2. Significant” is not itself statistically significant. Amer Statistician.
doi:10.7326/0003-4819-159-8-201310150-00709 2006;60:328–31.
2. Ioannidis JPA. Why most published research findings are false. PLoS 29. Uniform Requirements for Manuscripts Submitted to Biomedical
Med. 2005;2(8):e124. Journals, [Link] (accessed
3. Hiatt WR. Observational studies of drug safety–aprotinin and the May 2, 2013)
absence of transparency. N Engl J Med. 2006;355:2171–3. 30. Smith AH, Bates MN. Confidence limit analyses should replace power
4. Shaten BJ, Kuller LH, Kjelsberg MO, Stamler J, Ockene JK, Cutler calculations in the interpretation of epidemiologic studies. Epidemiology.
JA, Cohen JD. Lung cancer mortality after 16 years in MRFIT 1992;3:449–52.
participants in intervention and usual-care groups. Multiple Risk Factor 31. Planck M. Scientific Autobiography and Other Papers, Philosophical
Intervention Trial. Ann Epidemiol 1997;7:125–36. Library, New York, 1968, trans. F. Gaynor (New York, 1949), pp. 33–34