Research

Evidence summaries tailored to health policy-makers in

low- and middle-income countries
Sarah E Rosenbaum,a Claire Glenton,b Charles Shey Wiysonge,c Edgardo Abalos,d Luciano Mignini,d Taryn Young,e
Fernando Althabe,f Agustín Ciapponi,f Sebastian Garcia Marti,f Qingyue Meng,g Jian Wang,g Ana Maria De la Hoz
Bradford,h Suzanne N Kiwanuka,i Elizeus Rutebemberwa,i George W Pariyo,i Signe Flottorpa & Andrew D Oxmana

Objective To describe how the SUPPORT collaboration developed a short summary format for presenting the results of systematic
reviews to policy-makers in low- and middle-income countries (LMICs).
Methods We carried out 21 user tests in six countries to explore users’ experiences with the summary format. We modified the
summaries based on the results and checked our conclusions through 13 follow-up interviews. To solve the problems uncovered by
the user testing, we also obtained advisory group feedback and conducted working group workshops.
Findings Policy-makers liked a graded entry format (i.e. short summary with key messages up front). They particularly valued the section
on the relevance of the summaries for LMICs, which compensated for the lack of locally-relevant detail in the original review. Some
struggled to understand the text and numbers. Three issues made redesigning the summaries particularly challenging: (i) participants
had a poor understanding of what a systematic review was; (ii) they expected information not found in the systematic reviews and (iii)
they wanted shorter, clearer summaries. Solutions included adding information to help understand the nature of a systematic review,
adding more references and making the content clearer and the document quicker to scan.
Conclusion Presenting evidence from systematic reviews to policy-makers in LMICs in the form of short summaries can render the
information easier to assimilate and more useful, but summaries must be clear and easy to read or scan quickly. They should also
explain the nature of the information provided by systematic reviews and its relevance for policy decisions.

Abstracts in ‫عريب‬, 中文, Français, Pусский and Español at the end of each article.

Introduction The Supporting Policy-relevant Reviews and Trials (SUP-

PORT) project was an international collaboration funded from
To maximize the use of available resources, health policy-makers 2006 to 2010 by the European Commission´s 6th Framework
need reliable up-to-date evidence about “what works”.1–3 In low- Programme and by the Global Health Research Initiative of
and middle-income countries (LMICs) the pressure to extract the Canadian Institutes of Health Research. Its objective was
the most out of funds is particularly great, as the gap between to provide training and support to encourage researchers and
the resources available and those that are needed to address the policy-makers to undertake and use policy-relevant research. The
burden of preventable diseases is much larger than elsewhere.4 consortium had 10 partners in nine countries in Africa, South
Systematic reviews cover not only clinical interventions, but America and Europe. In this article we report on the SUPPORT
also arrangements for delivering, financing and managing health collaboration’s development of summaries of systematic reviews
for policy-makers in LMICs. Our objective was to tailor a sum-
services. Insofar as they are based on an exhaustive search for
mary format that was sensitive to the needs of this audience.
and appraisal of the relevant studies available, they are valuable
sources of research evidence and reduce the chances of being
misled by biased information.5 They make finding and apprais- Methods
ing the evidence much easier and faster and they illuminate areas Selecting reviews and developing content
where no evidence exists.3,6–8
We screened references (up to 2009) from the Cochrane
Systematic reviews very often contain findings that are rel- Library, MEDLINE and EMBASE to identify systematic
evant for LMICs. However, most are written largely for scientific reviews relevant for SUPPORT, based on topic and methods
audiences and are not well tailored to the information needs of documentation. For this study we selected five of these.26–30
policy-makers.9 Table 1 presents what we know about the type We extracted data, assessed review quality using a checklist and
of information policy-makers need. assessed evidence quality using GRADE.31,32 Based on earlier

a
Norwegian Knowledge Centre for the Health Services, Boks 7004, St Olavs Plass, N/0130 Oslo, Norway.
b
Sintef, Trondheim, Norway.
c
School of Child and Adolescent Health, University of Cape Town, Cape Town, South Africa.
d
Centro Rosarino de Estudios Perinatales, Santa Fe, Argentina.
e
South African Cochrane Centre, Cape Town, South Africa.
f
Institute for Clinical Effectiveness and Health Policy, Buenos Aires, Argentina.
g
Centre for Health Management and Policy, Shandong University, Jinan, China.
h
Department of Clinical Epidemiology and Biostatistics, Pontificia Universidad Javeriana, Bogota, Colombia.
i
Health Policy Planning and Management, Makerere University School of Public Health, Kampala, Uganda.
Correspondence to Sarah E Rosenbaum (e-mail: sarah@[Link]).
(Submitted: 2 January 2010 – Revised version received: 15 September 2010 – Accepted: 17 September 2010 – Published online: 24 November 2010 )

54 Bull World Health Organ 2011;89:54–61 | doi:10.2471/BLT.10.075481

 Research
Sarah E Rosenbaum et al. Evidence summaries for health policy-makers

Table 1. Information needs of policy-makers with respect to the evidence

General topic What is already known

Retrieval Timely retrieval of relevant research facilitates use.10,11
Time scale for commissioning new research fits poorly into time frame for policy-making.12
Research is often published in academic sources poorly accessible to policy-makers.13
Relevance for LMICs LMIC policy-makers may have limited access to subscription-based information or to the Internet.14,15
Research carried out in high-income countries may have limited applicability to LMICs.16–18
Content Systematic reviews sometimes answer too narrow a question.19
Policy-makers want not just information about “what works”, but also clearly articulated implications for policy,
such as costs, applicability, impacts on equity.1,3,10,13,20–22
Design/ease of use Length is a barrier; short summaries (with key messages highlighted) are strongly preferred.10,13,20,23
The perceptions that reviews facilitate the critical appraisal of evidence and are easy to use are strongly associated
with use.6
Correct understanding of evidence and its quality in full-text format may be difficult for non-researchers. Tables that
summarize findings may help.24
Use of familiar, jargon-free “plain language” is recommended.13,25
LMCIs, low- and middle-income countries.

research on the issues of greatest impor- experts in evidence dissemination (CG), ability”, credibility, usability, usefulness,
tance to policy-makers, we added content information design (SR) and epidemiol- desirability and value.35 Finally, the inter-
regarding applicability, equity, cost and ogy (ADO), during which ideas were viewer asked participants for suggestions
the future research needed.3,13 Research- generated based on an analysis of user test- and additional comments.
ers and policy-makers in LMICs, the lead ing and advisory group feedback. Finally, The interviewers audio-taped and
author of the systematic review being we designed new summary versions based transcribed each session and arranged
summarized and peers of the authors on the above. for the transcriptions and notes to be
with expertise in the review topic assessed translated (when necessary) and sent to
the completed summaries. An in-depth Testing the summaries the working group. They also erased audio
description of selection criteria, quality The working group conducted three pilot tapes and removed participant identity
assessments and content development can user tests with participants from Norwe- from the compiled results. Two research-
be found on the SUPPORT web site.33 gian government agencies involved in ers from the working group performed
LMIC development projects and made separate analyses and identified barriers
Developing summary format or facilitators to favourable experiences
further improvements based on these
of the summary according to the user ex-
As a starting point, we adopted a graded results. The advisory group then tested
perience framework. We then compared
entry format consisting of key mes- the summaries with 18 policy-makers in
and reconciled analyses and sorted the
sages followed by a short abstract. 13,34 Argentina (6), China (3), Colombia (3),
findings according to summary section
The abstract deviated from a traditional South Africa (3) and Uganda (3). The
(e.g. front page) or general theme (e.g.
academic format in that we replaced the group used Spanish-language versions in
language).
methods section with a short description Argentina and Colombia and English-
We used the results of the analysis to
of review characteristics. In addition, language versions in the remaining
make both content and design changes,
we replaced the discussion section with countries. The advisory group purposively
after which we presented both the user
one describing different aspects of the sampled participants, including health test analysis results and the new summary
information’s relevance for LMICs. This policy-makers and managers at different to the advisory group in a telephone
section included the applicability of the levels, and recruited them by e-mail and meeting. This group suggested only
evidence to LMIC settings, the impact telephone. minor changes.
of the approach or intervention on eq- The testing method was a think- After this redesign we sent the partic-
uity, the costs and other considerations aloud protocol using a semi-structured ipants both a brief outline of our findings
involved in scaling up the intervention, interview guide. Individual sessions lasted and the old and new summaries by post or
and the need for further evaluation. one hour and included one participant, e-mail. We asked them to indicate which
We used four methods in repeated one interviewer and one note-taker. version of the summary they preferred and
cycles to further develop this preliminary Introductory questions covered the par- why and to comment on the accuracy of
format. First we carried out user testing ticipants’ education, employment and our findings.
with LMIC policy-makers to inform familiarity with research and systematic
summary development from a user reviews. Participants chose one of five
perspective. We then elicited advisory possible summaries to read at their own Results
group feedback from multi-disciplinary pace. The interviewer then guided them Since the pilot test results did not devi-
LMIC researchers (prospective summary through each part of the document, ate from the rest, we pooled together
authors) to inform summary development prompting them to think aloud. The in- all the results. One test participant had
from an author’s perspective. This was fol- terview guide was based on a framework full-time medical school employment;
lowed by working group workshops with for user experience with six facets: “find- the other 20 were primarily senior staff

Bull World Health Organ 2011;89:54–61 | doi:10.2471/BLT.10.075481 55

 Research
Evidence summaries for health policy-makers Sarah E Rosenbaum et al.

members involved in national or inter- Eight participants found the tables Desirability
national health service or policy-related difficult or confusing, and nine said that Fourteen participants said they liked the
work in health departments, national the concepts presented in them, including summary, particularly the front page with
insurance programmes, hospitals or aid those that showed the GRADE assess- key messages and the section on the rel-
organizations. Seventeen participants ment and different levels of risk, were evance of the evidence and the interven-
said that they used research in their not clear. tion for LMICs. Seven reacted positively
work, though several seemed to define to the table describing the characteristics
“research” as any information-gathering “This section [summary of the findings] of the reviews:
on a topic; 18 said that they knew what would be very difficult to understand by
a systematic review was, but six of these people not trained in evidence-based “[I] like this chart; it makes clear what
participants were unfamiliar with Co- medicine. Words like ‘sample size’ and the review was looking for.”
chrane reviews. ‘relative risk’ would be difficult to in-
Of the six facets we explored from terpret…” Five participants said that they liked
the user experience framework, those the framing of the title as a question (e.g.
pertaining to usefulness, usability and Some participants felt that tables “Does pay-for-performance improve the
credibility yielded the most important running over two pages were cumber- quality of health care?”).
findings. some to read and that the abbreviations
caused confusion. Participants also “Findability”
Usefulness compared the numbers in the text with When asked where they would expect to
Sixteen participants reported that the evi- those in the tables and became confused find these summaries, seven participants
dence summary would be useful to them if if they did not correspond precisely. The answered “in face-to-face meetings”. Many
they had to make a decision on the subject use of jargon and/or unfamiliar vocabu- mentioned the web sites of the World
treated in the summary. The graded-entry lary posed a barrier (e.g. “scaling up” was Health Organization, the Pan American
format with key messages up front was not understood to include financial Health Organization, the Cochrane
perceived as particularly useful because it considerations). Collaboration, health ministries and
offered concision. However, many still felt universities.
a mismatch between the type of content Credibility
offered and their information needs: Early in the interview participants were Three challenging findings
asked if they would trust the summary. Several of our findings pointed to obvious
“[The summary] explains that there is Two responded affirmatively because it solutions that we adopted. These included
a high degree of satisfaction with what was “well written”. Twelve answered that simplifying the text and tables; limiting
the nurse practitioners are doing com- they would trust it because they perceived the number of tables and not letting
pared to the doctors. But it doesn’t say it as coming from credible sources: them break across pages; ensuring that
… whether they are supposed to cover the results in the text matched those in the
what the medical doctor or practitioner tables; eliminating abbreviations; using
“I would trust a report like this. It uses
usually covers. And what sort of services? consistent language and standard phrases
systematic reviews as sources of infor-
Is it general practice, is it in a hospital to describe effect sizes and the quality of
mation and I know that this kind of
ward or where?” the evidence36; replacing unfamiliar terms
information is of high quality.”
or adding definitions; moving partner
Some respondents expressed unmet logos and the summary publication date
expectations probably stemming from “The references are clear as well as the
source. That’s the most important thing.” to the front page.
a poor understanding of the nature of However, we found three larger is-
a systematic review. Specifically, they sues more challenging: (i) participants’
expected content lying outside the scope However, not everybody understood
that the summary stemmed from a sys- poor conceptual understanding of sys-
of a review: recommendations, outcome tematic reviews; (ii) participants’ expecta-
measurements not usually included in a tematic review. Some expressed confusion
about authorship (partner logos appeared tions that they would receive information
review, detailed information about local not found in the systematic reviews; and
applicability or costs and a broader fram- on the last page). Some also expressed
(iii) participants’ expressed desire for
ing of the research enquiry. reduced interest in the content when they
shorter, clearer summaries.
discovered that the quality of the evidence
To address the poor understanding of
Usability was low, that no evidence for important
the nature of systematic reviews and the
Five participants felt that the sum- outcomes existed or that the studies were
type of information they can provide, we
mary was not comprehensive enough. old. One participant was confused about
added “information about the informa-
However, six wanted a shorter, clearer how a high-quality review could be com- tion” or meta-information in the form of
presentation: patible with low-quality evidence. boxes placed throughout the summaries.37
To help satisfy participants’ expecta-
“Operational managers will be petrified. Value tion of being provided with information
When I think summary, I think one page Seventeen participants felt that summa- not found in the systematic reviews, we
... I would not have time to read a long ries of the kind presented to them would replaced the section for references with a
document even though I would want my be valuable to policy-makers holding section for “additional information”. We
work to be evidence-based.” positions similar to theirs. broadened the scope of this section and

56 Bull World Health Organ 2011;89:54–61 | doi:10.2471/BLT.10.075481

 Research
Sarah E Rosenbaum et al. Evidence summaries for health policy-makers

included not only research references their needs. The sections of the sum- messages rendered health technology as-
but also information that was helpful for mary on key messages and relevance for sessment reports more useful.44 An evalu-
understanding the problem, that provided LMICs proved to be the most interest- ation of Evidence Aid summaries showed
details about the interventions or that ing to participants, who had difficulty that the summaries would be more useful
put the results of the review in a broader understanding the risks presented in the if their coverage were not restricted to a
context. tables and were often frustrated with text single review and that language should
The third change we made addressed that seemed too long and complicated. be tailored to non-clinical audiences.45
the need for shorter, clearer summaries. Some did not seem to understand what In both studies, content that helped us-
Since each summary was already ex- a systematic review was and expected or ers to contextualize the evidence (e.g. a
tremely condensed, making the text even wanted information not usually found in discussion of applicability) was found to
shorter proved difficult. Instead, we facili- one. Some were also confused about the be particularly valuable.
tated rapid scanning of the document by source of the summaries. We addressed
reformatting the text to make it easier to these issues by altering the template’s Shorter messages or rapidly
pick out important parts. We reformatted content and design, especially by adding scannable texts?
the findings in the text as bullet point meta-information and reformatting the One of our overriding findings was a clear
items highlighted with blue arrows; we text to make it easier to scan. The advisory strong preference for short messages, also
divided the part on relevance into a table group and the participants agreed with found in other studies of policy-maker
placed between the findings and the sec- our analysis and supported our subse- preferences in research presentation.10,13,20
tion on the authors’ interpretations; we quent changes. There is, however, a limit to how much
moved the table with the characteristics information can be condensed before it
of the review to the background section, Study strengths and weaknesses loses value and credibility. When these
making it possible to restrict background Our study derived strength from the limits are reached, editing the text does
text to key information; and we used a participation of a wide range of policy- not suffice and methods such as graded-
narrower font to reduce document length. makers from different countries who entry structuring of the text and front-
To help summary authors in their represented different levels of decision- page summaries of key messages must be
efforts to create short, pertinent texts, making and familiarity with research used. In recent years, research on the use
we developed explicit instructions evidence. It was also strengthened by the of web sites has taught us much about how
about what information to include and presence of a multi-disciplinary advisory people visually scan texts, rather than read
exclude (template and guidelines avail- group of researchers and summary authors them. This knowledge can be applied to
able from authors). from LMICs. However, the translation of improve information delivery in policy
interview transcripts from Spanish and contexts where readers have limited time.
Follow-up interviews Chinese into English may have affected Bulleted lists, shorter paragraphs and
the correct interpretation of participants’ judicious use of headings are known to
Thirteen participants responded to the
feedback. Additionally, participants’ make scanning a text easier.46
follow-up questions. All preferred the
new format and said that they found it awareness that interviewers were involved
easier to read, primarily because of the in preparing the summaries may have af- Supporting better comprehension
new front-page design and the addition fected their responses. Finally, summary We uncovered several problems linked to
of the meta-information boxes: topics were pre-selected and not neces- poor comprehension of numbers and sta-
sarily matched to participants’ interests, tistics. Other studies have also shown that
“The content is presented in simple, and as a result reading motivation and even highly educated people struggle to
easy-to-understand language, especially understanding of the material may have understand numerical risk.47 This can re-
the first page … The reference box on been undermined. sult in frustration and in a failure to fully
the right, on page one, is perfect as it understand the main messages. However,
tells you what to expect.” Other summaries and evaluations correct comprehension depends not only
There are various products for effectively on the skills and knowledge of the reader,
There was general agreement that our conveying the results of systematic reviews but also on the way the information is pre-
analysis of the problems was precise and that are targeted to policy-makers, 38,39 sented.48 By assuming a weak background
that the new summary resolved the main and several of them target policy-makers knowledge (e.g. of scientific language or of
issues. Two participants repeated earlier in LMICs in particular. For instance, the the nature of systematic reviews) and low
misgivings about missing content outside Evidence Aid project40 provides summa- “statistical literacy”,48 summary authors
the scope of a systematic review. One ries of Cochrane reviews for emergency can add information to help readers better
participant felt that the tables remained settings. Sources of relevant evidence understand the strengths and limitations
confusing because “relative risk” was still summaries from high-income countries of the scientific evidence being sum-
not defined. include the Rx for Change database,41 marized. Adding meta-information that
Evidence Boost42 and the Policy Liaison explains concepts such as the quality of
Initiative.43 However, we uncovered few the evidence may help eliminate frustra-
Discussion studies reporting on the evaluations of tion and trigger reflection.
Policy-makers participating in user tests summary formats for policy-makers and
indicated that the graded entry format those we did identify support our own Future research
(one page of key messages followed by findings. Lavis et al. found that a graded- Systematic reviews attempt to answer nar-
a short summary) was well suited to entry format and up-front take-home rowly-defined scientific questions, such

Bull World Health Organ 2011;89:54–61 | doi:10.2471/BLT.10.075481 57

 Research
Evidence summaries for health policy-makers Sarah E Rosenbaum et al.

as whether or not an intervention has an geared towards specific settings cannot be unfamiliar with them and they are not
impact on specific outcomes. But policy- provided. According to earlier studies, re- easily accessible. Summaries of systematic
makers’ questions go far beyond whether search findings may be conceptually use- reviews can help address these problems
an intervention merely works; they also ful though not necessarily instrumental as long as they are clear and easy to read
address, among other things, whether for policy-making.11,21,49,50 When evidence or scan quickly. They should also help
it will work in a particular setting, how quality is too weak to provide conclusive to clarify the nature of the information
much it will cost and what consequences answers or when decision-makers’ settings provided by a systematic review and its
it may bring. The answers to these ques- vary greatly from those portrayed in the applicability to policy decisions. The
tions will vary from setting to setting and studies, evidence coupled with this kind SUPPORT summary format can make
cannot be provided by a single, generic of complementary content may still be the evidence gathered through systematic
summary. But summaries can support helpful in understanding the nature of reviews more useful for policy-making in
policy-makers by including content that the problem at hand, a possibility that health. ■
maps out the main issues they may need to future research should explore. In ad-
dition, studies should be conducted to Funding: The SUPPORT collaboration
consider in their own contexts (e.g. those was funded by the European Commis-
determine how SUPPORT summaries
findings and interpretations that relate to sion’s 6th Framework Programme Priority
compare with full reviews in terms of
applicability, equity and cost and hence FP6-2004-INCO-DEV-3. SUPPORT
their effect on understanding, time spent
to the relevance of the intervention for summaries are freely available at http://
reading and user satisfaction. If conducted
LMICs). Despite the lack of local detail in real-life contexts, such studies could [Link]
in the texts they were given, policy-makers further inform summary development.
in our study found this general type of Competing interests: None declared.
information very useful.
The favourable responses we ob-
Conclusion
served suggest that there is value in Systematic reviews are an important
mapping out these issues even if answers resource, but policy-makers are often

‫ملخص‬
‫مواءمة ملخصات الب ّينات حسب احتياجات راسمي السياسات الصحية يف البلدان املنخفضة واملتوسطة الدخل‬
‫) سوء‬1( :‫الخصوص جعلت إعادة تصميم امللخصات مثرية للتحدي وهي‬ ‫الغرض وصف كيف أَع َّد تنسيق مرشوع املراجعات والتجارب الداعمة‬
‫) املشاركون تو ّقعوا معلومات‬2( ‫إدراك املشاركني مباهية املراجعة املنهجية؛‬ ‫ منوذجاً ملخصاً موجزاً لتقديم نتائج املراجعات‬SUPPORT ‫للسياسات‬
‫) أراد املشاركون ملخصات أصغر‬3( ‫مل تكن موجودة يف املراجعات املنهجية؛‬ .‫املنهجية إىل راسمي السياسات يف البلدان املنخفضة واملتوسطة الدخل‬
‫ وتضمنت الحلول إضافة معلومات للمساعدة عىل فهم‬.ً‫حج ًام وأكرث وضوحا‬ ‫لخص يف ستة‬ َّ ‫ اختباراً عىل مستخدمي النموذج ا ُمل‬21 ‫الطريقة أجرى الباحثون‬
‫ وجعل املحتوى أكرث‬،‫ وإضافة املزيد من املراجع‬،‫طبيعة املراجعة املنهجية‬ َّ ُ
‫ وعدّل الباحثون امللخصات‬.‫لخص‬ َّ ‫بلدان الستكشاف خرباتهم حول النموذج ا ُمل‬
.‫ والوثيقة أرسع لالطالع‬،ً‫وضوحا‬ ‫ ولحل‬.‫ مقابلة للمتابعة‬13 ‫وفقاً للنتائج وراجعوا استنتاجاتهم عن طريق‬
‫االستنتاج إن تقديم ب ّينات املراجعات املنهجية لراسمي السياسات يف البلدان‬ ‫ أحرض الباحثون أيضاً مجموعة‬،‫املشاكل التي اكتشفها مستخدمو االختبار‬
‫املتوسطة واملنخفضة الدخل يف شكل ملخصات قصرية ميكن أن يجعل‬ .‫استشارية الستقصاء اآلراء وأجروا حلقات عمل ملجموعة العمل‬
‫ ولكن يجب أن تكون‬،‫املعلومات أكرث سهولة يف االستيعاب واالستفادة‬ ‫فضل راسمو السياسات منوذج إدخال متدرج (أي ملخص قصري مع‬ ّ ‫النتائج‬
ً‫ ويجب أن ترشح أيضا‬.‫واالطالع الرسيع‬ّ ‫امللخصات واضحة وسهلة للقراءة‬ ‫ واستحسنوا عىل وجه الخصوص القسم الخاص‬.)‫رسائل أساسية عىل الواجهة‬
‫طبيعة املعلومات التي تقدمها املراجعات املنهجية وعالقتها مبتخذي القرار‬ ‫لخصات براسمي السياسات يف البلدان املنخفضة واملتوسطة‬ َّ ‫بارتباط ا ُمل‬
.‫السيايس‬ ‫ وجد‬.‫ والذي ع ّوض نقص التفاصيل املحلية الهامة يف املراجعة األصلية‬،‫الدخل‬
‫ وهناك ثالثة مواضيع عىل وجه‬.‫البعض صعوبة يف فهم النصوص واألرقام‬

概要
针对中低收入国家卫生决策者的证据概要
目的 旨在描述SUPPORT协作如何开发出一个简短的概要 足。一些人则努力了解文本和数字。下述三个问题使得重
格式，从而为中低收入国家（LMICs）决策者呈现系统综 新设计概要特别具有挑战性：（1）参与者对系统综述缺
述结果。 乏了解；（2）他们期待的信息不能在系统综述中找到；
方法 我们在六个国家进行了21个用户测试，以探讨用户 且（3）他们想要更简短、更清晰的概要。解决方案包括
的概要格式经验。我们基于调查结果修改了概要并通过 增加信息以帮助了解系统综述的性质、增加更多参考文献
13个后续采访检验了结论。为了解决用户测试未曾涉及 并使内容更清晰、使文件更易于浏览。
的问题，我们还获得了顾问组的反馈并开展了工作小组 结论 以简短概要的形式为中低收入国家的决策者呈现系统
研讨会。 综述得出的证据，能够使信息更易吸收且更有用，但是概
结果 决策者喜欢一种分级登录格式（即在前面有关键信 要必须清楚且易于阅读和浏览。其还应该解释系统综述所
息的简短概要）。他们特别重视有关中低收入国家概要关 提供的信息的性质和其与决策的关联性。
联性的部分，这弥补了原始综述中缺乏本地相关细节的不

58 Bull World Health Organ 2011;89:54–61 | doi:10.2471/BLT.10.075481

 Research
Sarah E Rosenbaum et al. Evidence summaries for health policy-makers

Résumé
Synthèses de preuves adaptées aux décideurs de la santé dans les pays à revenu faible ou intermédiaire
Objectif Décrire la façon dont la collaboration SUPPORT a développé un et les chiffres. Trois problèmes ont fait de la reconception des synthèses
format synthétique permettant de présenter les résultats d’évaluations une véritable gageure : (i) les participants avaient une compréhension
systématiques aux décideurs de la santé dans les pays à revenu faible médiocre de l’évaluation systémique; (ii) ils attendaient des informations
ou intermédiaire. que les évaluations systématiques ne contenaient pas et (iii) ils souhaitaient
Méthodes Nous avons réalisé 21 tests utilisateur dans six pays afin des synthèses plus courtes et plus précises. Les solutions comprenaient
d’étudier l’expérimentation du format synthétique par les utilisateurs. l’ajout d’informations afin de mieux saisir la nature d’une évaluation
Nous avons modifié les synthèses en fonction des résultats et vérifié nos systématique, un surcroît de références, la rédaction d’un contenu plus
conclusions par le biais de 13 entretiens de suivi. Afin de résoudre les clair ainsi qu’une analyse plus rapide du document.
problèmes dont les tests utilisateur ne traitaient pas, nous avons également Conclusion La présentation d’éléments de preuve provenant d’évaluations
recueilli les commentaires du groupe consultatif et organisé des ateliers systématiques aux responsables de la santé dans les pays à revenu
de groupe de travail. faible ou intermédiaire sous forme de brèves synthèses peut faciliter
Résultats Les responsables appréciaient le format d’entrée progressive l’assimilation des informations et les rendre plus utiles, mais ces synthèses
(par ex. une synthèse courte introduite par des messages clés). Ils doivent être claires, faciles à lire ou rapidement analysables. Elles doivent
avaient une préférence notable pour la section dédiée à l’importance des également expliquer la nature des informations fournies par les évaluations
synthèses pour les PRFI car elle compensait l’absence de détails locaux systématiques et leur pertinence en matière de décision de politiques.
essentiels de l’évaluation d’origine. Certains ont eu du mal à saisir le texte

Резюме
Типовая форма отчета, адаптированная к потребностям разработчиков политики в области
здравоохранения в странах с низким и средним доходом
Цель Рассказать о том, как организация SUPPORT цифры. Три проблемы создавали особые трудности при
разработала краткую форму отчета для представления доработке резюме: (i) участники плохо понимали, что такое
результатов систематических обзоров разработчикам систематический обзор; (ii) они ожидали информации,
политики в странах с низким и средним доходом (СНСД). которая не содержится в систематических обзорах и (iii)
Методы Мы провели 21 тест с пользователями в восьми они хотели, чтобы резюме были короче и изложены более
странах с целью обобщить опыт использования ими типовой понятным языком. Для решения этих проблем в текст была
формы отчета. На основе результатов мы внесли изменения добавлена информация, помогающая понять особенности
в отчеты и проверили наши выводы путем проведения 13 систематического обзора, увеличено число ссылок,
интервью на этапе последующего наблюдения. Для решения содержание упрощено, и документ был приспособлен для
проблем, не учтенных при тестировании пользователей, быстрого просмотра.
мы также обратились за помощью к экспертам и провели Вывод Представление данных систематических обзоров
семинары для рабочих групп. разработчикам политики в СНСД в форме кратких резюме
Результаты Разработчикам политики понравилась может облегчить восприятие информации и сделать ее
нумерованная форма входящего документа (т. е. краткое более полезной, но резюме должны быть понятными и
резюме с основными выводами, вынесенными в начало). удобными для быстрого прочтения или просмотра. В них
Они особенно высоко оценили раздел о возможностях также должны быть объяснены характер информации,
применения резюме в СНСД, который компенсировал содержащейся в систематических обзорах, и ее полезность
отсутствие привязки к местным условиям в первоначальном для принятия политических решений.
обзоре. Некоторые с трудом воспринимали текст и

Resumen
Resúmenes de datos diseñados para los responsables políticos sanitarios de los países de ingresos medios y
bajos
Objetivo Describir cómo la colaboración SUPPORT ha desarrollado un Resultados A los responsables políticos les gustó el formato de entradas
formato de resumen breve que presenta los resultados de las revisiones clasificadas (es decir, un resumen breve con los mensajes clave al
sistemáticas para los responsables políticos en los países de ingresos principio). Valoraron positivamente el apartado sobre la importancia de
medios y bajos (PIMB). los resúmenes para los PIMB, que compensaba la falta de información
Métodos Hemos llevado a cabo 21 pruebas de usuario en seis países más detallada con interés local en la revisión original. A algunos les
distintos, con el fin de conocer las experiencias de los usuarios con este costó trabajo entender el texto y los números. La reestructuración de
formato de resumen. Hemos modificado los resúmenes en función de los resúmenes resultó especialmente compleja por tres cuestiones en
los resultados y hemos comprobado nuestras conclusiones mediante particular: (a) los participantes contaban con escasos conocimientos sobre
13 entrevistas de seguimiento. Con el fin de solucionar los problemas que lo que era una revisión sistemática; (b) esperaban disponer de información
las pruebas de usuario evidenciaron, hemos consultado la opinión de un que no se incluye en las revisiones sistemáticas y (c) preferían resúmenes
grupo de asesores y hemos llevado a cabo talleres de grupos de trabajo. más cortos y claros. Entre las posibles soluciones se encontraban la

Bull World Health Organ 2011;89:54–61 | doi:10.2471/BLT.10.075481 59

 Research
Evidence summaries for health policy-makers Sarah E Rosenbaum et al.

de añadir información para ayudar a entender el concepto de revisión resúmenes breves puede contribuir a que la información resulte más
sistemática, añadir más referencias y hacer que el contenido resultara fácil de asimilar y más útil, si bien los resúmenes deben ser claros, así
más claro y que el documento pudiera escanearse más rápido. como fáciles y rápidos de leer. También deben explicar la naturaleza de la
Conclusión La presentación de los datos extraídos de las revisiones información que se facilita en las revisiones sistemáticas y su relevancia
sistemáticas a los responsables sanitarios de los PIMB en forma de para la toma de decisiones sobre políticas sanitarias.

References
1. Task Force on Health Systems Research. Informed choices for attaining the 19. Lomas J. Using research to inform healthcare managers’ and policy
Millennium Development Goals: towards an international cooperative agenda makers’ questions: from summative to interpretive synthesis. Healthc Policy
for health-systems research. Lancet 2004;364:997–1003. doi:10.1016/ 2005;1:55–71. PMID:19308103
S0140-6736(04)17026-8 PMID:15364193 20. Dobbins M, Jack S, Thomas H, Kothari A. Public health decision-makers’
2. Travis P, Bennett S, Haines A, Pang T, Bhutta Z, Hyder AA et al. Overcoming informational needs and preferences for receiving research evidence.
health-systems constraints to achieve the Millennium Development Worldviews Evid Based Nurs 2007;4:156–63. doi:10.1111/j.1741-
Goals. Lancet 2004;364:900–6. doi:10.1016/S0140-6736(04)16987-0 6787.2007.00089.x PMID:17850496
PMID:15351199 21. Lomas J. Improving research dissemination and uptake in the health sector:
3. Lavis JN, Posada FB, Haines A, Osei E. Use of research to inform beyond the sound of one hand clapping. Hamilton: McMaster University
public policymaking. Lancet 2004;364:1615–21. doi:10.1016/S0140- Centre for Health Economics and Policy Analysis; 1997.
6736(04)17317-0 PMID:15519634 22. Jewell CJ, Bero LA. “Developing good taste in evidence”: facilitators of and
4. The Millenium Development Goals report 2006. New York: United Nations; hindrances to evidence-informed health policymaking in state government.
2008. Milbank Q 2008;86:177–208. doi:10.1111/j.1468-0009.2008.00519.x
5. Greenhalgh T. Papers that summarise other papers (systematic reviews and PMID:18522611
meta-analyses). BMJ 1997;315:672–5. PMID:9310574 23. Dobbins M, Thomas H, O’Brien MA, Duggan M. Use of systematic reviews
6. Dobbins M, Cockerill R, Barnsley J. Factors affecting the utilization of in the development of new provincial public health policies in Ontario.
systematic reviews. A study of public health decision makers. Int J Technol Int J Technol Assess Health Care 2004;20:399–404. doi:10.1017/
Assess Health Care 2001;17:203–14. [research article] doi:10.1017/ S0266462304001278 PMID:15609787
S0266462300105069 PMID:11446132 24. Rosenbaum SE, Glenton C, Oxman AD. Summary-of-findings tables
7. Davies HTO, Nutley SM, Smith PC, editors. What works? Evidence-based in Cochrane reviews improved understanding and rapid retrieval of
policy and practice in public services. Bristol: The Policy Press; 2000. key information. J Clin Epidemiol 2010;63:620–6. doi:10.1016/j.
8. Lavis JN, Davies HTO, Gruen RL, Walshe K, Farquhar CM. Working within and jclinepi.2009.12.014 PMID:20434024
beyond the Cochrane Collaboration to make systematic reviews more useful 25. Dobbins M, Rosenbaum P, Plews N, Law M, Fysh A. Information transfer:
to healthcare managers and policy makers. Healthc Policy 2006;1:21–33. what do decision makers want and need from researchers? Implement Sci
PMID:19305650 2007;2:20. doi:10.1186/1748-5908-2-20 PMID:17608940
9. Sheldon TA. Making evidence synthesis more useful for management 26. Waters H, Hatt L, Peters D. Working with the private sector for child
and policy-making. J Health Serv Res Policy 2005;10(Suppl 1):1–5. health. Health Policy Plan 2003;18:127–37. doi:10.1093/heapol/czg017
doi:10.1258/1355819054308521 PMID:16053579 PMID:12740317
10. Petticrew M, Whitehead M, Macintyre SJ, Graham H, Egan M. Evidence 27. Briggs CJ, Garner P. Strategies for integrating primary health services in
for public health policy on inequalities: 1: the reality according to middle- and low-income countries at the point of delivery. Cochrane Database
policymakers. J Epidemiol Community Health 2004;58:811–6. doi:10.1136/ Syst Rev 2006;19:CD003318. PMID:16625576
jech.2003.015289 PMID:15365104 28. Gosden T, Forland F, Kristiansen IS, Sutton M, Leese B, Giuffrida A et al.
11. Innvaer S, Vist GE, Trommald M, Oxman AD. Health policy-makers’ Impact of payment method on behaviour of primary care physicians:
perceptions of their use of evidence: a systematic review. J Health Serv a systematic review. J Health Serv Res Policy 2001;6:44–55.
Res Policy 2002;7:239–44. doi:10.1258/135581902320432778 doi:10.1258/1355819011927198 PMID:11219360
PMID:12425783 29. Beney J, Bero LA, Bond C. Expanding the roles of outpatient pharmacists:
12. Elliott H, Popay J. How are policy makers using evidence? Models of research effects on health services utilisation, costs, and patient outcomes. Cochrane
utilisation and local NHS policy making. J Epidemiol Community Health Database Syst Rev 2000;3:CD000336. PMID:10908471
2000;54:461–8. doi:10.1136/jech.54.6.461 PMID:10818123 30. Lewin SA, Dick J, Pond P, Zwarenstein M, Aja G, van Wyk B et al. Lay health
13. Lavis J, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. workers in primary and community health care. Cochrane Database Syst Rev
Towards systematic reviews that inform health care management and 2005;1:CD004015.[PMID:15674924] PMID:15674924
policy-making. J Health Serv Res Policy 2005;10(Suppl 1):35–48. 31. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S et al.;
doi:10.1258/1355819054308549 PMID:16053582 GRADE Working Group. Grading quality of evidence and strength of
14. Smith R. Closing the digital divide. [editorial]BMJ 2003;326:238. recommendations. BMJ 2004;328:1490. doi:10.1136/bmj.328.7454.1490
doi:10.1136/bmj.326.7383.238 PMID:12560254 PMID:15205295
15. Edejer TT-T. Disseminating health information in developing countries: the role 32. Cochrane IMS [Internet]. GRADEpro (GRADEprofiler) is the software used to
of the internet. BMJ 2000;321:797–800. doi:10.1136/bmj.321.7264.797 create summary of findings (SoF) tables in Cochrane systematic reviews.
PMID:11009519 Oxford: The Cochrane Collaboration. Available from: [Link]
16. Isaakidis P, Swingler GH, Pienaar E, Volmink J, Ioannidis JP. Relation revman/gradepro/gradepro. [accessed 8 November 2010].
between burden of disease and randomised evidence in sub-Saharan Africa: 33. SUPPORT Collaboration [Internet]. How SUPPORT summaries are prepared.
survey of research. BMJ 2002;324:702. doi:10.1136/bmj.324.7339.702 SUPPORT. Available from: [Link]
PMID:11909786 [Link] [accessed 8 November 2010].
17. Reinar LM, Greenhalgh T, editors. Working in partnership with people in the 34. Reader-friendly writing - 1:3:25. Ottawa: Canadian Health Services Research
developing world. London: Cochrane Colloquium Abstracts; 2005. Available Foundation; 2001. Available from: [Link]
from: [Link] pdf/cn-1325_e.pdf [accessed 12 November 2010].
view/1229 [accessed 8 November 2010]. 35. Morville P. User experience design. Ann Arbor: Semantic Studios LLC;
18. Haines A, Kuruvilla S, Borchert M. Bridging the implementation gap between 2004. Available from: [Link]
knowledge and action for health. Bull World Health Organ 2004;82:724–31, semantics/[Link] [accessed 8 November 2010].
discussion 732. PMID:15643791

60 Bull World Health Organ 2011;89:54–61 | doi:10.2471/BLT.10.075481

 Research
Sarah E Rosenbaum et al. Evidence summaries for health policy-makers

36. Glenton C, Santesso N, Rosenbaum S, Nilsen ES, Rader T, Ciapponi A et al. 43. Policy Liaison Initiative [Internet]. Linking health policy to the latest evidence.
Presenting the results of Cochrane Systematic Reviews to a consumer Melbourne: Australasian Cochrane Centre. Available from: [Link].
audience: a qualitative study. Med Decis Making 2010;30:566–77. au/ebpnetwork/ [accessed 8 November 2010].
doi:10.1177/0272989X10375853 PMID:20643912 44. Lavis JN, Wilson MG, Grimshaw JM, Haynes RB, Ouimet M, Raina P et al.
37. SUPPORT Collaboration [Internet]. Supporting policy relevant reviews and Supporting the use of health technology assessments in policy making
trials. Support summaries. SUPPORT. Available from: [Link] about health systems. Int J Technol Assess Health Care 2010;26:405–14.
support/[Link]. [accessed 8 November 2010]. doi:10.1017/S026646231000108X PMID:20923592
38. Lavis JN. How can we support the use of systematic reviews in policymaking? 45. Turner T, Green S, Harris C. Supporting evidence-based health care in crises:
PLoS Med 2009;6:e1000141. doi:10.1371/[Link].1000141 What information do humanitarian organisations need? Disaster Med Public
PMID:19936223 Health Prep Forthcoming
39. Robeson P, Dobbins M, DeCorby K, Tirilis D. Facilitating access to 46. [Link]: Jakob Nielsen’s website [Internet]. Morkes J, Nielsen J. Concise,
pre-processed research evidence in public health. BMC Public Health scannable, and objective: how to write for the web; 1997. Available from:
2010;10:95. doi:10.1186/1471-2458-10-95 PMID:20181270 [Link] [accessed 8 November
40. The Cochrane Collaboration [Internet]. Evidence aid summaries. London: 2010].
Evidence Aid Project, The Cochrane Collaboration. Available from: www. 47. Lipkus IM, Samsa G, Rimer BK. General performance on a numeracy scale
[Link]/evidenceaid/[Link]. [accessed 8 November 2010]. among highly educated samples. Med Decis Making 2001;21:37–44.
41. Rx for change [Internet]. Ottawa: Canadian Agency for Drugs and doi:10.1177/0272989X0102100105 PMID:11206945
Technologies in Health. Available from: [Link]/[Link]/en/compus/ 48. Ancker JS, Kaufman D. Rethinking health numeracy: a multidisciplinary
optimal-ther-resources/interventions/ [accessed 8 November 2010]. literature review. J Am Med Inform Assoc 2007;14:713–21. doi:10.1197/
42. EvidenceBoost [Internet]. Ottawa: Canadian Health Services Research jamia.M2464 PMID:17712082
Foundation. Available from: [Link]/mythbusters/eb_e.php [accessed 8 49. Weiss CH. The many meanings of research utilization. Public Adm Rev
November 2010]. 1979;39:426–31. doi:10.2307/3109916
50. Nutley SM, Walter I, Davey HTO. Using evidence: how research can inform
public services. Bristol: The Policy Press; 2007.

Bull World Health Organ 2011;89:54–61 | doi:10.2471/BLT.10.075481 61