RESEARCH

Exploratory analyses in aetiologic research and considerations

BMJ: first published as 10.1136/bmj-2021-070113 on 3 May 2022. Downloaded from [Link] on 6 May 2022 by guest. Protected by copyright.
for assessment of credibility: mini-review of literature
Kim Luijken,1 Olaf M Dekkers,1 Frits R Rosendaal,1 Rolf H H Groenwold1,2
1
Department of Clinical Abstract (0-1467) for additional analyses. Six considerations
Epidemiology, Leiden University Objective were discussed that are important in assessing the
Medical Centre, Leiden, To provide considerations for reporting and credibility of exploratory analyses: research problem,
Netherlands
2 interpretation that can improve assessment of the protocol, statistical criteria, interpretation of findings,
Department of Biomedical
Data Sciences, Leiden University credibility of exploratory analyses in aetiologic completeness of reporting, and effect of exploratory
Medical Centre, Leiden, research. findings on future causal research.
Netherlands
Design Conclusions
Correspondence to: K Luijken
[Link]@[Link] Mini-review of the literature and account of exploratory Based on this mini-review, exploratory analyses in
(ORCID 0000-0001-5192-8368) research principles. aetiologic research were not always reported properly.
Additional material is published Setting Six considerations for reporting of exploratory
online only. To view please visit analyses in aetiologic research were provided to
the journal online. This study focuses on a particular type of causal
research, namely aetiologic studies, which investigate stimulate a discussion about their preferred handling
Cite this as: BMJ 2022;377:e070113
[Link] the causal effect of one or multiple risk factors on a and reporting. Researchers should take responsibility
bmj‑2021‑070113 particular health outcome or disease. The mini review for the results of exploratory analyses by clearly
Accepted: 22 March 2022 included aetiologic research articles published in reporting their exploratory nature and specifying
four epidemiology journals in the first issue of 2021: which findings should be investigated in future
American Journal of Epidemiology, Epidemiology, research and how.
European Journal of Epidemiology, and International
Journal of Epidemiology, specifically focusing Introduction
on observational studies of causal risk factors of Reports of aetiologic studies often have results
diseases. of multiple exploratory analyses, with the aim of
Main outcome measures identifying topics for future research. Although this
Number of exposure-outcome associations reported, form of reporting might seem reasonable, it is not
grouped by type of analysis (main, sensitivity, and without risk, because compared with the results of
additional). a confirmatory study, assessing the credibility of
exploratory findings is generally more complicated.
Results
The origin of exploratory data analysis can be traced
The journal articles reported many exposure-outcome
back at least to Tukey in the 1960s and 1970s1 2 who
associations: a mean number of 33 (range 1-120)
encouraged statisticians to develop visualisation
exposure-outcome associations for the primary
techniques for representing and capturing structures
analysis, 30 (0-336) for sensitivity analyses, and 163
in datasets to establish new research questions.
These new research questions should subsequently
be answered with independent datasets (often termed
What is already known on this topic confirmatory analysis). For example, when a new
Exploratory analyses in aetiologic research are initial steps towards solving biomarker is thought to be part of a known causal
a research problem and are often conducted in addition to planned primary pathway, performing a small preparatory exploratory
analyses of a study study before conducting a full blown large cohort
Exploratory analyses might lead to new discoveries in aetiologic research, but study seems worthwhile, because the cohort study is
effort is needed to accurately interpret the results because these analyses financially expensive and requires large investments
are often conducted with few data resources and insufficient adjusting for of resources. Similarly, if a known exposure-outcome
confounding effect is thought to vary across subgroups of the
population, exploring this idea first before embarking
Statistical properties of exploratory tests are less well known than those of
on confirmative analyses of the effect of heterogeneity
confirmatory tests
seems appropriate.
What this study adds Even when researchers consider an analysis to
This study focuses on a particular type of causal research, namely aetiologic be exploratory, a hypothesis is easily promoted to
studies, which investigate the causal effect of one or multiple risk factors on a a fact. For example, findings in journal articles can
particular health outcome or disease be exaggerated to more certain statements in press
releases and news articles.3 In medical science in
Six considerations for reporting of exploratory analyses in aetiologic research were
particular, where results are sometimes quickly
provided to stimulate a discussion about their preferred handling and reporting
implemented in clinical practice, researchers should
Researchers should take responsibility for results of exploratory analyses by
take responsibility for the results they report. The
clearly reporting their exploratory nature and specifying which findings should be
Hippocratic oath (“First, do not harm”) applies as well
investigated in future research and how
to medical research as it does to clinical practice.

the bmj | BMJ 2022;377:e070113 | doi: 10.1136/bmj-2021-070113 1

In this paper, we discuss issues that complicate the analyses, specifically the STROBE (Strengthening the
interpretation of exploratory analyses in causal studies. Reporting of Observational Studies in Epidemiology)

BMJ: first published as 10.1136/bmj-2021-070113 on 3 May 2022. Downloaded from [Link] on 6 May 2022 by guest. Protected by copyright.
Causal research can refer to different types of research, statement,6 RECORD (REporting of studies Conducted
such as randomised studies or intervention studies. using Observational Routinely collected health
We do not address these studies in our manuscript; Data) statement,7 STROBE-MR (Strengthening the
we focus on aetiologic research, in which causes Reporting of Observational Studies in Epidemiology
of disease are investigated. Specifically, the causal Using Mendelian Randomisation) for mendelian
effect of risk factors on a health outcome or disease randomisation studies,8 STREGA (Strengthening the
are studied, typically in an observational setting. We Reporting of Genetic Association Studies) for genome
provide practical pointers for researchers on how to association studies,9 and the CONSORT (Consolidated
report exploratory analyses in aetiologic research and Standards of Reporting Trials) extension to randomised
how to clarify what the exploratory results imply for pilot and feasibility trials.10
future research and implementation in practice. We
hope to encourage a discussion about the preferred Patient and public involvement
handling and reporting of these analyses. Involving patients or the public in the design, conduct,
reporting, or dissemination plans of our research was
Methods not appropriate or possible.
Exploratory analyses in aetiologic research
The term exploratory analysis typically refers to analyses Results
for which the hypothesis was not specified before the Mini-review
data analysis.4 Considering exploratory analyses in a The mini-review included 25 original aetiologic
broader sense, however, is probably more relevant in articles. These articles reported a mean number of
aetiologic research, because of the observational data 33 (range 1-120) exposure-outcome associations
and clustering of analyses within cohorts. We use the for the primary analysis, 30 (0-336) for sensitivity
term exploratory analyses here to indicate analyses analyses, and 163 (0-1467) for additional analyses,
that are initial and preliminary steps towards solving mainly concerning subgroup or interaction analyses
a research problem. Exploratory analyses are often (supplementary file). Most articles did not explicitly
conducted in addition to planned primary analyses report which analyses were prespecified, and only
of a study. We do not consider sensitivity analyses, one study referred to a publicly available protocol.11
where the main hypothesis is evaluated under different The methodological scrutiny of the subgroup analyses
assumptions, to be exploratory in this paper. We also do varied from thoughtful evaluations of exposure
not consider outcomes that are evaluated as a secondary effect heterogeneity in well established subgroups to
objective but are correlated with the primary outcome to evaluations of exposure effects across subgroups that
be exploratory, because these analyses contribute to the seemed to have been formed exhaustively across many
investigation of the primary research question. Genome- potential risk factors. Despite the fact that our review
wide association studies, where the exploratory nature included only a small sample of studies, the image that
of analyses is commonly accounted for by looking at arises from it is that many results were presented, and
multiple testing,5 are beyond the scope of this paper. insufficient information was reported to fully judge the
validity and merits of the results.
Mini-review and overview of existing reporting
guidance Existing reporting guidance
Before we discuss considerations about the reporting of The STROBE6 and RECORD7 statements provide
exploratory aetiologic studies, we wanted to illustrate checklists of items to report in observational studies
some of the aspects of exploratory studies that need that are relevant to exploratory analyses (table 1).
explicit reporting. Hence we performed a small review Extensions of STROBE, such as STROBE-MR8 and
of published aetiologic studies. We identified all articles STREGA,9 provide additional guidance for reporting
on original research in four journals in their first issue of of studies where many analyses are performed.
2021: American Journal of Epidemiology, Epidemiology, Guidance for reporting randomised trials also provides
European Journal of Epidemiology, and International helpful information for reporting exploratory analyses
Journal of Epidemiology. We excluded studies that did in aetiologic research, in particular the CONSORT
not look at an aetiologic research question, such as extension to randomised pilot and feasibility trials.10
prediction studies, studies on therapeutic interventions, Not all of these recommendations can be directly
and randomised trials. For each article, we counted applied to observational aetiologic studies, however,
the number of primary analyses, sensitivity analyses, because the procedures for generating and testing
and additional analyses that were performed. The unit of hypotheses are more established in randomised
of counting was the association estimator, where we studies than in observational settings.
counted only one association if the association was
reported on different scales (eg, absolute and relative Exploratory research principles
scales for binary endpoints). Inspired by the existing recommendations for
Also, we reviewed existing reporting guidance reporting, we list six considerations for reporting and
documents on aspects relevant to exploratory interpretation that can improve the assessment of

2 doi: 10.1136/bmj-2021-070113 | BMJ 2022;377:e070113 | the bmj

the credibility of exploratory analyses in aetiologic Nosek and colleagues22 have provided preliminary
research (table 1). The list is not exhaustive but we guidance on preregistration of analyses conducted

BMJ: first published as 10.1136/bmj-2021-070113 on 3 May 2022. Downloaded from [Link] on 6 May 2022 by guest. Protected by copyright.
hope it will encourage further discussion on the with existing data. These authors suggest that what
reporting of exploratory research. was known in advance about the dataset should
be transparently reported so that the credibility
Consideration 1: explicitly state the objective of all of statistical findings can be assessed, taking into
analyses, including exploratory analyses account analyses that have been performed previously.
Stating the objective of an aetiologic study clarifies Implementing this advice is probably challenging in
how to interpret the results. The objectives of large epidemiological cohort studies because of the
confirmatory aetiologic research ideally contain a well many analyses that might have been performed. But
defined targeted effect of a specific aetiologic factor on trying to clarify why and how an analysis is conducted
a specific outcome in a specific population.13 14 In early before observing the data is a laudable practice that
discovery research, objectives are not always rigorously can be implemented directly in aetiologic studies. This
defined but could be specified more generally (eg, practice is ideally accompanied by work on developing
understanding the origin of a particular outcome). guidance for preregistration of aetiologic studies that
An implication of stating the objective in general use existing data.
terms, however, is that the methodological handling Preregistration of analyses that are exploratory in
of the analysis becomes less clear and the number nature is even less common, possibly contradicting
of researchers’ degrees of freedom becomes large.15 the definition of exploration. We consider exploratory
Consequently, interpreting results without deriving analysis, however, as discovery work that serves to
spurious (causal) conclusions requires thought and motivate funding for larger studies that are, for example,
effort because the analysis does not necessarily provide better able to control confounding or to collect data
information towards a causal effect (see consideration rigorously. Given this important probing role, simply
4).16-18 The more general an objective is stated, the stating in a research protocol that certain relations
more provisional the analysis becomes. This caveat will be explored is not enough; time and effort must
includes machine learning approaches where no be invested in designing the analysis appropriately.
explicit causal modelling assumptions are made. Not every detail can be specified in advance, but
Because exploratory analyses in aetiologic interpretation of the results provided by data can be
research often aim to inform a future in-depth challenging and unintentionally overconfident when
causal analysis, reporting both the objective of the no question was clearly articulated before seeing the
provisional exploratory analysis and the (future) answer.
confirmatory analysis is important. This reporting is
in line with the CONSORT reporting checklist for pilot Consideration 3: do not base judgments on
randomised controlled trials which recommends that significance values only
researchers state the objective of the eventual trial in Only reporting the results of analyses that provided
the manuscript of a pilot study.10 The rationale and a P value below the prespecified α level (eg, 0.05) is
need for the exploratory analysis in aetiologic research discouraged throughout all scientific disciplines (for
should be outlined together with uncertainties that example, as discussed in a 2019 supplementary issue
need to be dealt with before performing an independent of The American Statistician).23 Avoiding selective
confirmative analysis of the causal mechanism. reporting based on significance values is particularly
Reporting the position of provisional analyses relative relevant to exploratory findings because the statistical
to future research clarifies the level of credibility of the properties of exploratory tests are less well known
findings from exploratory analyses. than those of confirmatory tests.24 For example, the
expected number of false positives (that is, the type I
Consideration 2: establish a study protocol before error rate) is probably increased when the choice for
data analysis and make the protocol available to a statistical test was based on pattens in the observed
readers data. Although procedures have been developed for
Preregistered protocols help distinguish which correction of multiple testing in confirmatory settings,
analyses were planned before observing the data and consensus on how to prevent false positive findings in
which analyses were performed post hoc, thereby exploratory settings has not yet been established.24-26
avoiding hypothesising after the results are known. For Increasing the number of exploratory analyses,
randomised trials, preregistration of the study protocol without correction for multiple testing, raises the
is considered the norm.19 Preregistration does not seem risk of deriving false positive conclusions, but too
as widespread in observational aetiologic research, strict corrections for multiple testing increases the
but is increasingly encouraged,20 21 and explicitly probability of false negative findings (that is, the type
recommended in the RECORD reporting checklist.7 II error rate).27 A raised type II error rate could occur,
Because aetiologic research often uses existing cohort for example, when an analysis of various positively
data that have been analysed for related research correlated hypotheses is corrected for multiple testing
questions, preregistration of aetiologic studies does as if all of the hypotheses were independent (eg,
not ensure the same level of credibility of statistical by applying a Bonferroni correction). The decision
evidence as preregistration before collecting the data. to statistically correct for multiple testing depends,

the bmj | BMJ 2022;377:e070113 | doi: 10.1136/bmj-2021-070113 3

Table 1 | Considerations for reporting of exploratory aetiologic research

BMJ: first published as 10.1136/bmj-2021-070113 on 3 May 2022. Downloaded from [Link] on 6 May 2022 by guest. Protected by copyright.
Considerations for reporting of exploratory aetiologic studies analyses
1. Explicitly state the objective of all analyses, including exploratory analyses State the objective of the definite trial and rationale for a pilot study (CONSORT
State the objective of an exploratory analysis to clarify how the results are to be interpreted. pilot and feasibility studies,10 items 2 and 6c)
Outline the objective of the definite analysis of interest and clarify why an exploratory
analysis should be conducted first
2. Establish a study protocol before data analysis and make the protocol available to Register a protocol (CONSORT statement12 and CONSORT pilot and feasibility
readers studies,10 items 23 and 24; RECORD,7 item 22)
Specify the objective, design, and analysis plan in a protocol, even when existing data are
analysed or when an analysis is considered exploratory
3. Do not base judgments on significance values only Consider adjustment for multiple comparisons (STROBE-MR,8 item 6e; STREGA,9
Avoid selective reporting of results based on significance values, particularly because items 12i and 16d)
exploratory analyses are commonly conducted with less rigorously collected data and
suboptimal ability to adjust for confounding. Also, statistical properties of exploratory tests
are less well known than those of confirmatory tests
4. Interpret findings in line with the nature of the analysis Distinguish prespecified from exploratory results in reporting (CONSORT pilot and
Be transparent about the exploratory aim of the analysis and avoid overstating the feasibility studies,10 items 21 and 22; CONSORT statement,12 items 3b and 6b)
credibility of findings. Minimise suggestions on generalisability and clinical relevance for Discuss if the data used for analysis might affect interpretation of findings
exploratory findings (RECORD,7 item 19)
5. Report (summarised) results of all exploratory analyses that were performed Report all study results (STROBE6 and RECORD,7 items 16 and 17; CONSORT
Report results of all exploratory analyses that were conducted (possibly in a supplementary statement12 and CONSORT pilot and feasibility studies,10 items 17 and 18)
file) to provide a transparent and honest account of the analysis that facilitates Report all study results when multiple (data driven) analyses are preformed
interpretation of findings (STROBE-MR,8 items 11 and 13; STREGA,9 items 17b and 17c)
Make analysis code available (RECORD,7 item 22)
6. Accompany exploratory analyses by a proposed research agenda Report which and how future confirmative studies can be informed by the conducted
Formulate a research agenda prioritising future research and how this research should be exploratory analyses (CONSORT pilot and feasibility studies,10 items 21 and 22)
set up. This process ensures researchers take responsibility for the presented exploratory
findings and follow-up research that should be performed
STROBE=Strengthening the Reporting of Observational Studies in Epidemiology statement; RECORD=REporting of studies Conducted using Observational Routinely collected health Data
statement; STROBE-MR=Strengthening the Reporting of Observational Studies in Epidemiology Using Mendelian Randomization; STREGA=Strengthening the Reporting of Genetic Association
Studies; CONSORT=Consolidated Standards of Reporting Trials.

among other issues, on the total number of tests methodology rather than overstepping the findings.
performed in the same dataset, correlation between the The specificity with which findings are interpreted
hypotheses being tested, and sample size. Reporting should match the generality with which the objective
each of these considerations clarifies the analytical is stated (see consideration 1).16-18 For example, when
context of findings and helps to assess the credibility various subgroup analyses are performed with the
of the results. This form of reporting is in line with general aim of identifying possible subgroups from the
the STROBE-MR8 and STREGA9 checklists which available data where an exposure effect was different,
recommend stating how multiple comparisons were researchers should report that many subgroups were
managed, although recommendations for the handling explored, including characterisation of the subgroups
of multiple testing seem more established in genome- and description of the presence or absence of effect
wide association studies than in clinical aetiologic heterogeneity, rather than discussing only one or two
cohort studies.5 specific subgroups where the effect size was extreme.
Furthermore, exploratory analyses often fail to support
Consideration 4: interpret findings in line with the strong conclusions. Recommendations for clinical
nature of the analysis practice or generalisations based on exploratory
Interpreting and communicating results in line with analyses should generally be avoided.
the exploratory nature of an analysis is challenging
because an accurate representation of the degree of Consideration 5: report (summarised) results of all
tentativeness of the results is required. Assessing this exploratory analyses that were performed
degree of tentativeness based on only the results of an When findings are selectively reported, especially
analysis (that is, based on the numerical estimates) is when reporting is guided by significant findings (see
complicated because seemingly convincing results can consideration 3), the credibility of reported findings
be misleading and a clinical explanation can be found is probably overstated.30 Reporting the results of all of
that does not follow from the statistical evidence.28 29 the exploratory analyses that were conducted (possibly
Cognitive biases, such as hindsight bias, can distort the in a supplementary file) provides a transparent and
interpretation of findings. honest report of the analysis and facilitates better
Reporting of findings from exploratory analyses interpretation of the findings. This approach is in
starts with indicating whether the analysis was line with the STROBE extension in STREGA, which
planned before or after observing the data, which recommends that all results of analyses should
is recommended in the CONSORT extension to be presented, even if numerous analyses were
randomised pilot and feasibility trials.10 Results of undertaken.9
exploratory analyses can be interpreted by focusing Reporting all analyses that have been conducted
on what is reported about the objectives and applied seems simple, but can be challenging in practice,

4 doi: 10.1136/bmj-2021-070113 | BMJ 2022;377:e070113 | the bmj

mainly because the process of performing a study is tempting. When quick test results are presented in a
typically iterative. A framework for initial data analysis research article, however, their interpretation might be

BMJ: first published as 10.1136/bmj-2021-070113 on 3 May 2022. Downloaded from [Link] on 6 May 2022 by guest. Protected by copyright.
by Huebner and colleagues could help keep track of ad hoc and unintentionally overconfident.
all subanalyses that are conducted as part of a main To show their full value, exploratory analyses
analysis.31 This framework distinguishes exploratory of aetiologic research need to be conducted
analyses that are part of a primary analysis from and interpreted correctly. We have provided six
additional exploratory analyses that require separate considerations for reporting of exploratory analyses to
reporting. Another helpful practice could be to have a encourage a discussion on exploratory analyses and
reflection period after performing analyses to establish how the credibility of these analyses is ideally assessed
whether the analyses look at (slightly) different in aetiologic research. Continuation of this discussion
research questions and to report separate analyses for will contribute to the understanding of inferences that
each research question. can be made from exploratory analyses in aetiologic
research and will help strike a balance between their
Consideration 6: accompany exploratory analyses by opportunities and risks.
a proposed research agenda Contributors: KL was involved in the conceptualisation, investigation,
The credibility of exploratory findings can be methodology, visualisation, and writing (original draft) of the
article. OMD was involved in the conceptualisation, investigation,
communicated through a research agenda prioritising methodology, and writing (review and editing) of the article. FRR
future research and how this research should be was involved in the conceptualisation, investigation, methodology,
set up. Reporting a research agenda is similar to and writing (review and editing) of the article. RHHG was involved in
the conceptualisation, investigation, methodology, supervision, and
the CONSORT extension to randomised pilot and writing (review and editing) of the article. KL, OMD, FRR, RHHG gave
feasibility trials that recommends reporting which final approval of the version to be published and are accountable for
and how future confirmative trials can be informed by all aspects of the work. KL is the main guarantor of this study. The
corresponding author attests that all listed authors meet authorship
the pilot study.10 criteria and that no others meeting the criteria have been omitted.
Formulating a research agenda allows researchers Funding: RHHG was supported by grants from the Netherlands
to take responsibility for the exploratory findings Organisation for Scientific Research (ZonMW, project 917.16.430)
presented and future research that should be and from Leiden University Medical Centre. The funders had no
role in considering the study design or in the collection, analysis,
performed, avoiding the empty statement that interpretation of data, writing of the report, or decision to submit the
“more research is needed”. In medical science in article for publication.
particular, where study results are sometimes quickly Competing interests: All authors have completed the ICMJE uniform
implemented into clinical practice, researchers are disclosure form at [Link]/disclosure-of-interest/ and declare:
support from the Netherlands Organisation for Scientific Research
encouraged to take responsibility for the results and Leiden University Medical Centre for the submitted work; no
they report by clearly explaining which exploratory financial relationships with any organisations that might have an
findings should be investigated in future research and interest in the submitted work in the previous three years; no other
relationships or activities that could appear to have influenced the
how. submitted work.
Ethical approval: Not required.
Discussion Data sharing: No additional data available.
Our mini-review showed that exploratory analyses The lead author (the manuscript’s guarantor) affirms that the
in aetiologic research were not always reported manuscript is an honest, accurate, and transparent account of the
optimally. The credibility of exploratory results is study being reported; that no important aspects of the study have
been omitted; and that any discrepancies from the study as planned
affected by a combination of the theoretical rationale (and, if relevant, registered) have been explained.
for the analysis, clarity of the defined research Dissemination to participants and related patient and public
problem, applied methodology, and degree to which communities: An abstract was submitted to the annual Dutch
analytical decisions are driven by the data. Choosing epidemiology conference ([Link]). The authors aim to share
their work with stakeholders at the annual Dutch epidemiology
a particular analysis based on observed patterns in the conference ([Link]), at institutional meetings, and will post a
data complicates statistical inferences. Moreover, the link with a plain language summary on their personal websites (www.
design and methods applied in an exploratory analysis [Link]).
might be less optimal than the primary analysis of Provenance and peer review: not commissioned; externally peer
reviewed.
the study, which further complicates interpretation of
This is an Open Access article distributed in accordance with the
exploratory analyses. Therefore, information on these Creative Commons Attribution Non Commercial (CC BY-NC 4.0)
aspects should be clearly reported. license, which permits others to distribute, remix, adapt, build upon
Exploration is essential to the progress of science. this work non-commercially, and license their derivative works on
different terms, provided the original work is properly cited and the
Strict confirmatory studies are a powerful mechanism use is non-commercial. See: [Link]
for final evaluations before implementation in by-nc/4.0/.
clinical practice, but will probably not stimulate new
1 Tukey JW. The future of data analysis. Ann Math Stat 1962;33:1-67.
ideas.32 33 Open minded exploratory analyses can lead 2 Tukey JW. Exploratory data analysis. 1977.
to unexpected discoveries and resourceful innovations 3 Sumner P, Vivian-Griffiths S, Boivin J, et al. The association between
of epidemiological science, but effort is required to exaggeration in health related science news and academic press
releases: retrospective observational study. BMJ 2014;349:g7015.
accurately interpret the results. Because exploratory 4 Wagenmakers E-J, Wetzels R, Borsboom D, van der Maas HL, Kievit
analyses are usually done to generate new research RA. An agenda for purely confirmatory research. Perspect Psychol
Sci 2012;7:632-8.
questions, quickly performing a statistical test (or 5 Sham PC, Purcell SM. Statistical power and significance testing in
multiple tests) to get the first answer to the problem is large-scale genetic studies. Nat Rev Genet 2014;15:335-46.

the bmj | BMJ 2022;377:e070113 | doi: 10.1136/bmj-2021-070113 5

6 von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, 18 Westreich D, Greenland S. The table 2 fallacy: presenting
Vandenbroucke JP, STROBE Initiative. Strengthening the Reporting and interpreting confounder and modifier coefficients. Am J
of Observational Studies in Epidemiology (STROBE) statement: Epidemiol 2013;177:292-8.

BMJ: first published as 10.1136/bmj-2021-070113 on 3 May 2022. Downloaded from [Link] on 6 May 2022 by guest. Protected by copyright.
guidelines for reporting observational studies. BMJ 2007;335:806-8. 19 Zarin DA, Tse T, Williams RJ, Rajakannan T. Update on trial
doi:10.1136/[Link]. registration 11 years after the ICMJE policy was established. N Engl J
7 Benchimol EI, Smeeth L, Guttmann A, et al, RECORD Working Med 2017;376:383-91.
Committee. The REporting of studies Conducted using Observational 20 Loder E, Groves T, Macauley D. Registration of observational studies.
Routinely-collected health Data (RECORD) statement. PLoS BMJ 2010;340:c950. doi:10.1136/bmj.c950
Med 2015;12:e1001885. 21 Hemingway H, Riley RD, Altman DG. Ten steps towards improving
8 Skrivankova VW, Richmond RC, Woolf BAR, et al. Strengthening prognosis research. BMJ 2009;339:b4184.
the Reporting of Observational Studies in Epidemiology 22 Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration
Using Mendelian Randomization: The STROBE-MR Statement. revolution. Proc Natl Acad Sci U S A 2018;115:2600-6.
JAMA 2021;326:1614-21. 23 American Statistician. Statistical inference in the 21st century: A
9 Little J, Higgins JP, Ioannidis JP, et al. Strengthening the reporting world beyond p<0.05. Am Stat 2019;73; suppl 1. [Link]
of genetic association studies (STREGA): an extension of the [Link]/toc/utas20/73/sup1
strengthening the reporting of observational studies in epidemiology 24 Goeman JJ, Solari A. Multiple testing for exploratory research. Stat
(STROBE) statement. J Clin Epidemiol 2009;62:597-608.e4. Sci 2011;26:584-97.
10 Eldridge SM, Chan CL, Campbell MJ, et al, PAFS consensus group. 25 Heller R. Discussion of “Multiple testing for exploratory research” by JJ
CONSORT 2010 statement: extension to randomised pilot and Goeman and A Solari. Stat Sci 2011;598-600.
feasibility trials. BMJ 2016;355:i5239. doi:10.1136/bmj.i5239 26 Westfall PH. Discussion of “Multiple testing for exploratory research”
11 Keys MT, Serra-Burriel M, Martínez-Lizaga N, et al. Population- by JJ Goeman and A Solari. Stat Sci 2011;26:604-7.
based organized screening by faecal immunochemical testing 27 Groenwold RHH, Goeman JJ, Cessie SL, Dekkers OM. Multiple testing:
and colorectal cancer mortality: a natural experiment. Int J when is many too much?Eur J Endocrinol 2021;184:E11-4.
Epidemiol 2021;50:143-55. 28 Goldacre B, Drysdale H, Marston C, et al. COMPare: Qualitative
12 Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation analysis of researchers’ responses to critical correspondence on a
and elaboration: updated guidelines for reporting parallel group cohort of 58 misreported trials. Trials 2019;20:124.
randomised trials. BMJ 2010;340:c869. 29 Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values,
13 Shmueli G. To explain or to predict?Stat Sci 2010;25:289-310. confidence intervals, and power: a guide to misinterpretations. Eur J
14 Goetghebeur E, le Cessie S, De Stavola B, Moodie EE, Waernbaum I, Epidemiol 2016;31:337-50.
“on behalf of” the topic group Causal Inference (TG7) of the STRATOS 30 Ioannidis JP. Why most discovered true associations are inflated.
initiative. Formulating causal questions and principled statistical Epidemiology 2008;19:640-8.
answers. Stat Med 2020;39:4922-48. 31 Huebner M, le Cessie S, Schmidt CO, et al. A contemporary conceptual
15 Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: framework for initial data analysis. Obs Stud 2018;4:171-92.
undisclosed flexibility in data collection and analysis allows 32 Vandenbroucke JP. Observational research, randomised trials, and
presenting anything as significant. Psychol Sci 2011;22:1359-66. two views of medical science. PLoS Med 2008;5:e67.
16 Conroy S, Murray EJ. Let the question determine the methods: 33 Tukey JW. We need both exploratory and confirmatory. Am
descriptive epidemiology done right. Br J Cancer 2020;123: Stat 1980;34:23-5.
1351-2.
17 Grimes DA, Schulz KF. Descriptive studies: what they can and cannot
do. Lancet 2002;359:145-9. Web appendix: Supplementary file

6 doi: 10.1136/bmj-2021-070113 | BMJ 2022;377:e070113 | the bmj