IDEAL Reporting Guidelines for Surgery
IDEAL Reporting Guidelines for Surgery
clinical decision, and included in a systematic review. There are now identifying existing relevant reporting guidelines and those in devel-
>400 checklists on the EQUATOR Network’s website, including 42 opment registered on the EQUATOR Network website (http://
that are surgery related ([Link] [Link]), a survey to establish which stages of
guidelines/). Among these existing reporting guidelines applicable to IDEAL, if any, participants considered in need of new reporting
surgery SCARE,13 PROCESS,14 STROCSS,15 and CONSORT guidelines, the development of draft checklists for Stages 1, 2a, 2b,
NPT16 aim to improve the reporting of specific, well-recognized and 4 following establishment of need based on consensus, 2 rounds
study types (case reports, case series, cohort studies, and randomized of a Delphi consensus survey to refine each of the checklists, and a
controlled trials). Additional guidance exists for specific aspects of consensus meeting via teleconference with the working group to
surgical evaluative study designs. For example TIDIER 17 has been finalize the checklists.
developed as an extension of CONSORT to improve reporting of
intervention details, and the COHESIVE18 guidelines are being Participants
developed for a generic core outcome set in studies of early phase Using our MailChimp e-newsletter, we contacted 445 indi-
surgery and medical devices. A CONSORT extension for RCTs using viduals who met at least 1 of 4 criteria: member of the original
cohorts and routinely collected health data is also underway.19 IDEAL steering group; a wider member of the IDEAL Collaboration
Additionally, the National Evaluation System for Health Technology signed up to receive newsletters; author or co-author of any key core
Coordinating Center (NESTcc) Methods Framework has been devel- IDEAL publication; attended 1 of the 2016, 2017, and 2018 IDEAL
oped to define the key components of study design, protocols, and conferences. From this group, 119 expressed interest in participating,
evidence gathering for the evaluation of medical devices20 and those individuals were invited to complete an online survey about
All these guidelines draw upon principles of good research the need for reporting guidelines for any stage of IDEAL, as defined
methodology, as does the IDEAL framework. However the IDEAL by the EQUATOR Network’s suggested criteria.24 The 61 respond-
reporting guidelines have a unique and specific focus on stepwise ents recommended that new reporting guidelines should be devel-
evaluation of surgical innovation through all stages of its develop- oped for Stage 1 (69%), Stage 2a (78%), Stage 2b (75%), and Stage 4
ment, from pre-clinical through first in human evidence, feasibility (66%) (Supplementary Figure 1, [Link]
and randomized trials to long-term follow-up including use of real- IDEAL Stage 3 was considered adequately covered by existing
world data. Throughout these stages, IDEAL prescribes specific guidelines (CONSORT, CONSORT NPT and other relevant CON-
novel study formats, focusing on clarity about key elements of SORT extensions) by 68% of respondents. Characteristics of
surgical innovation, with which many researchers may be unfamiliar. responding participants are shown in Table 1. Fourteen volunteers
This likely explains the inaccuracy in reporting noted in studies from groups (1) and (3) formed a working group to agree the final
attempting to follow the IDEAL recommendations and suggests a checklists during the final consensus call.
pressing need for IDEAL reporting guidelines.
The development of such IDEAL-specific guidelines would Development of Checklists
serve to improve the usability and impact of IDEAL. As Altman and The initial drafts of checklists for the 4 stages were developed
Moher state, ‘‘reporting problems affect journal articles in 2 main using a combination of approaches to identify what key items may be
ways. First, the study methods are frequently not described in adequate needed. These included ongoing monitoring of the literature using or
detail. Second, the study findings are presented ambiguously, incom- commenting on the usability of IDEAL, identified from Google
pletely, or selectively. The cumulative effect of these problems is to Scholar and Web of Science Alerts of articles citing the 2009 and
render many reports of research unusable or even harmful; at the very 2019 IDEAL Statement articles, assessing gaps in existing reporting
least, such articles certainly represent a waste of resources’’.21 Chalm- guidelines, and accumulated refinements of IDEAL principles devel-
ers and Glasziou have highlighted the detrimental effects of waste in oped from issues repeatedly raised in recent years during educational
the production and reporting of research evidence and call for seminars and lectures on IDEAL. These checklists were evaluated in
increased use of reporting guidelines as part of the solution.22 2 rounds of a Delphi consensus survey using a 9-point Likert scale,
This article describes the development of a set of IDEAL (as recommended by GRADE), to rate agreement on the importance
reporting guidelines which aim to improve the quality, transparency, of each element and elicit free-text comments on the wording of each
consistency, and utility of the surgical innovation evidence base. The item.25,26 In the Likert scale, a score of 1 to 3 signified limited
protocol for the project was published in 2018.23 importance of the item, 4 to 6 important but not critical, and 7 to 9
critical. If 70% of respondents scored an item 7 to 9 and >15%
METHODS scored it 1 to 3, then that item was included in the reporting guideline.
Conversely, consensus that an item should not be included was
Research Design defined as 70% scoring it 1 to 3 and 15% scoring it 7 to 9.
We developed the IDEAL reporting guidelines through a 5- Fifty-four participants completed Round 1 of the Delphi. Two
phase process between October 2018 and May 2019. This comprised reviewers (N.B., A.H.) reviewed the responses and modified the
checklist drafts based on the participants scoring and feedback, modifications and monitoring of learning curves. Because of this,
editing phrasing and wording of checklist items, and merging the diligent use of other guidelines cannot help investigators to report
redundant items, but ensuring meaning was not significantly IDEAL studies appropriately, and an IDEAL-specific guideline is
changed. Round 2 of the Delphi survey presented the results of required. For example comparing the SCARE guidelines for surgical
round one to participants, addressed clarifications, and served to case reports13 and the IDEAL Stage 1 recommendations, IDEAL
refine the checklist items where consensus was not yet reached. specifies requirements absent from SCARE (Applicability only to
Participants were again asked to score the Round 2 checklist items on first-in-human studies, justify need and refer to pre-clinical devel-
a 1 to 9 Likert Scale. Forty-seven participants completed Round 2 opment, detailed account of patient selection, requirement for fully
and their responses were analyzed by 2 researchers (N.B., A.H.) and informed consent for treatment, requirement for explanation of
presented to the working group during the final teleconference procedure detailed enough to allow reproduction).
consensus meeting. The IDEAL checklists are intended to provide a minimum list
of concepts authors should include in a report of an IDEAL format
Consensus Teleconference study. They have been created for wide-spread adoption across all
A consensus call held in May 2019 was attended by 11 of the interventional specialties, including therapeutic devices, for which
14 members of the working group. Three participants who were IDEAL has published a modified framework, IDEAL-D.5 The
unable to attend submitted written contributions via email. A draft of checklists are equally usable for IDEAL and IDEAL-D studies,
the methods, checklist drafts for each stage, and a provisional list of most items pertaining to both surgical and device innovation.
items where the Delphi process had not yielded a definitive result Although not designed expressly for these purposes, the guidelines
were prepared in advance and sent to the call participants. The items can also be used both prospectively to help plan a study and
that remained controversial were discussed and modifications were retrospectively to assist in appraisal. They can also be used alongside
made once the group reached agreement. All checklist items for each other reporting guidelines to provide additional useful detail (eg,
stage were then reviewed and revised as needed to ensure clarity. The COHESIVE for core outcomes reporting, CONSORT extension for
group also decided on a strategy to disseminate the checklists. The routinely collected health data for Stage 4).18,19
revisions were then sent out to the participants for final comment
and approval. Dissemination and Evaluation of IDEAL Checklists
Checklists are not effective unless used, so dissemination
RESULTS efforts are important. The EQUATOR network27 recommends 10
The citation alerts and search of the EQUATOR Network list specific actions to aid dissemination: making the checklist freely
of reporting guidelines revealed no specific guidelines dealing with accessible in editable format; publishing and presenting editorials,
the IDEAL Recommendations as an entity either as the sole focus of blogs, website pages, conference papers and webinars, co-operating
study or as part of a larger ambit, and no guidelines which specifi- with EQUATOR, seeking journal endorsements and running training
cally described how to report IDEAL format studies. Numerous courses. We intend to follow this guidance utilizing our website and
guidelines contained generic recommendations also included by Twitter accounts ([Link] and @IDEALCol-
IDEAL, for example, use of standard outcome definitions, publica- lab). Some specific elements of our strategy will be the use of social
tion of protocols, and full informed consent, but none referenced media to establish a feedback dialogue with users, presentation of the
IDEAL as a special case.14– 19 guidelines at conferences and scientific meetings, editorials and
Supplementary Tables 1, [Link] and blogs, and translation into Chinese and Spanish to maximize inter-
2, [Link] show the distribution of scores for national readership. In line with EQUATOR guidance, we will pilot
each checklist item in Rounds 1 and 2 respectively of the Delphi the checklist in ongoing IDEAL format studies, for which we will
process. The 4 unresolved issues following Round 2, discussed shortly be introducing an advisory service, and will publish an
during the final consensus call were: Inclusion of IDEAL and the updated version with any modifications resulting from both feedback
stage number in titles; inclusion of ‘‘first-in-human’’ in the title of and piloting. A pre-post guideline publication survey of reporting of
Stage 1 studies; mandatory publication of a previous protocol for all IDEAL format studies to evaluate the effectiveness of these efforts
studies; and how to resolve tensions between commercial confidenti- will be developed at a future date. Endorsement by authoritative
ality and ethical demands for transparency in early-stage studies bodies and efforts by editors to encourage or enforce compliance
involving devices. The solutions to each are incorporated in the final were features of the development and spread of the CONSORT
full checklists presented here. guidelines, which appear to be the best complied with to date, so we
will continue to seek support in these sectors.
IDEAL Checklists Limitations
IDEAL is based on ethical and methodological principles Like all consensus processes, our Delphi study was dependent
which apply to all stages and types of medical research, and many on the representativeness of participants. We invited participants with
of which are held in common with other methodological paradigms a good understanding of IDEAL, as a naı̈ve group would be unlikely
and checklists. Supplementary Table 3, [Link] to be able to appreciate some of the issues or achieve consensus. We
C820 presents the full checklists for IDEAL Stages 1, 2a, 2b, and 4 attempted to include a wide range of stakeholders but were not
and highlights key or unique IDEAL items with an example from the universally successful. The number of journal editors, for example,
literature, which illustrates appropriate use. We have not provided was small. A different group might have come up with different
examples for more familiar items common to other reporting check- recommendations, but no major differences in responses could be
lists. The article quoted does not necessarily demonstrate good detected according to background in our sample. Although consensus
reporting for other IDEAL recommendations. was achieved by GRADE criteria on the need for a guideline in all
stages except Stage 3, the minority which disagreed was substantial
DISCUSSION (around 25%) in each case. Informal feedback indicates that this may
The IDEAL Recommendations include several innovations in have been because participants familiar with IDEAL had come to see
study design for clinical research, such as reporting of iterative reporting of IDEAL stage items as straightforward, and not needing
guidance. Some participants also mentioned the proliferation of oesophageal sphincter for gastro-oesophageal reflux disease. Br J Surg.
2020;107:44–55.
reporting guidelines as a reason for caution.
12. EQUATOR Network website. What is a Reporting Guideline? Available at:
[Link]
CONCLUSIONS (Accessed March 30, 2020)
13. Agha RA, Borrelli MR, Farwana R, et al. For the SCARE Group. The SCARE
The consensus-based checklists developed through this Delphi 2018 Statement: Updating Consensus Surgical CAse REport (SCARE) Guide-
process have the potential to improve the standards of reporting of lines. Int J Surg. 2018;60:132–136.
early-stage innovation and surgical trials and thereby advance the 14. Agha RA, Borrelli MR, Farwana R, et al. For the PROCESS Group. The
quality of research on surgery and complex interventions. Surgical PROCESS 2018 Statement: Updating Consensus Preferred Reporting of CasE
innovation and research that follows a transparent, sequential frame- Series in Surgery (PROCESS) Guidelines. Intl J Surg. 2018;60:279–282.
work such as IDEAL and is reported with these guidelines can also 15. Agha R, Abdall-Razak A, Crossley E, et al. STROCSS 2019 Guideline:
strengthening the reporting of cohort studies in surgery. Int J Surg.
aid reproducibility, reduce research waste, and improve evidence- 2019;72:156–165.
based practice for the benefit of patients. 16. Boutron I, Altman DG, Moher D, et al. CONSORT NPT Group. CONSORT
statement for randomized trials of nonpharmacologic treatments: a 2017
Contributors update and a CONSORT extension for nonpharmacologic trial abstracts.
IDEAL Collaboration Reporting Guidelines Working Group: Ann Intern Med. 2017;167:40–47.
Jane Blazeby, Maroeska Rovers, Christopher Pennell, Joel Horovitz, 17. Hoffmann T, Glasziou P, Boutron I, et al. Better reporting of interventions:
template for intervention description and replication (TIDieR) checklist and
Angelos Kolias, Janet Martin, Tom Lewis, Riaz Agha, Josh Feinberg, guide. BMJ. 2014;348:g1687.
in addition to the authors. 18. Avery K, Blazeby J, Wilson N, et al. Development of reporting guidance and
core outcome sets for seamless, standardised evaluation of innovative surgical
REFERENCES procedures and devices: a study protocol for content generation and a Delphi
consensus process (COHESIVE study). BMJ Open. 2019;9:e029574.
1. Ergina PL, Cook JA, Blazeby JM, et al. Challenges in evaluating surgical
innovation. Lancet. 2009;374:1097–1104. 19. Kwakkenbos L, Juszczak E, Hemkens LG, et al. Protocol for the development
of a CONSORT extension for RCTs using cohorts and routinely collected
2. McCulloch P, Altman DG, Campbell WB, et al. No surgical innovation health data. Res Integr Peer Rev. 2018;3:1–9.
without evaluation: the IDEAL recommendations. Lancet. 2009;374:1105–
1112. 20. National Evaluation System for health Technology Coordinating Center
(NESTcc) Frameworks Feb 2020. Available at: [Link]
3. Pennell CP, Hirst AD, Campbell WB, et al. Practical guide to the idea, nestcc-data-quality-framework/and [Link]
development and exploration stages of the IDEAL framework and recom- ods-framework/(Accessed March 30, 2020).
mendations. Br J Surg. 2016;103:607–615.
21. Altman DG, Moher D. Guidelines for Reporting Health Research: A User’s
4. Hirst A, Philippou Y, Blazeby J, et al. No surgical innovation without Manual. In: Moher, D., A: [Link]
evaluation: evolution and further development of the IDEAL framework 9781118715598.
and recommendations. Ann Surg. 2019;269:211–220.
22. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of
5. Sedrakyan A, Campbell B, Merino JG, et al. IDEAL-D: A rational framework research evidence. Lancet. 2009;374:86–89.
for evaluating and regulating the use of medical devices. BMJ.
2016;353:i2372. 23. Agha RA, Hirst A, Khachane A, et al. A protocol for the development of
reporting guidelines for IDEAL stage studies. Int J Surg Protoc. 2018;9:11–
6. Hamilton D, Beard D, McCulloch P, et al. Evidence-based evaluation of 14.
practice and innovation in rehabilitation using the IDEAL-physio framework.
Phys Ther. 2018;98:108–121. 24. EQUATOR Network - Identifying the need for a new reporting guideline.
Available at: [Link]
7. Khachane A, Philippou Y, Hirst A, et al. Appraising the uptake and use of the ing-guideline/identifying-the-need-for-a-new-reporting-guideline/(Accessed
IDEAL framework and recommendations: a review of the literature. Int J Surg. March 30, 2020).
2018;57:84–90.
25. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on
8. Tradewell MB, Albersheim J, Dahm P. Use of the IDEAL framework in the rating quality of evidence and strength of recommendations. BMJ. 2008;336.
urological literature: where are we in 2018? BJU Int. 2019;123:1078–1085. 924 LP-926.
9. Currie A, Brigic A, Blencowe NS, et al. Systematic review of surgical 26. Jaeschke R, Guyatt GH, Dellinger P, et al. Use of GRADE grid to reach
innovation reporting in laparoendoscopic colonic polyp resection. Br J Surg. decisions on clinical practice guidelines when consensus is elusive. BMJ.
2015;102:e108–e116. 2008;337.
10. Barkun JS, Dimick JB, Clavien PA. Surgical research in patients: ideal time for 27. EQUATOR Network – Developing and disseminating your reporting guide-
an IDEAL checklist. Ann Surg. 2019;269:208–210. line. Available at: [Link]
11. Kirkham EN, Main BG, Jones KJB, et al. Systematic review of the reporting-guideline/disseminating-your-reporting-guideline/(Accessed
introduction and evaluation of magnetic augmentation of the lower March 30, 2020).