Critical appraisal checklist for a questionnaire study
(Selecting, designing, and developing your questionnaire Petra M Boynton, lecturer in
health services research1 and Trisha Greenhalgh. BMJ. 2004 May 29; 328(7451): 1312–1315.)
Research question and study design
What information did the researchers seek to obtain?
Was a questionnaire the most appropriate method and if not, what
design might have been more appropriate?
Were there any existing measures (questionnaires) that the
researchers could have used? If so, why was a new one developed and
was this justified?
Were the views of consumers sought about the design, distribution,
and administration of the questionnaire?
Validity and reliability
What claims for validity have been made, and are they justified? (In
other words, what evidence is there that the instrument measures what
it sets out to measure?)
What claims for reliability have been made, and are they justified? (In
other words, what evidence is there that the instrument provides
stable responses over time and between researchers?)
Format
Was the title of the questionnaire appropriate and if not, what were its
limitations?
What format did the questionnaire take, and were open and closed
questions used appropriately?
Were easy, non-threatening questions placed at the beginning of the
measure and sensitive ones near the end?
Was the questionnaire kept as brief as the study allowed?
Did the questions make sense, and could the participants in the
sample understand them? Were any questions ambiguous or overly
complicated?
Instructions
Did the questionnaire contain adequate instructions for completion—
eg example answers, or an explanation of whether a ticked or written
response was required?
Were participants told how to return the questionnaire once
completed?
Did the questionnaire contain an explanation of the research, a
summary of what would happen to the data, and a thank you
message?
Piloting
Was the questionnaire adequately piloted in terms of the method and
means of administration, on people who were representative of the
study population?
How was the piloting exercise undertaken—what details are given?
In what ways was the definitive instrument changed as a result of
piloting?
Sampling
What was the sampling frame for the definitive study and was it
sufficiently large and representative?
Was the instrument suitable for all participants and potential
participants? In particular, did it take account of the likely range of
physical/mental/cognitive abilities, language/literacy, understanding
of numbers/scaling, and perceived threat of questions or questioner?
Distribution, administration and response
How was the questionnaire distributed?
How was the questionnaire administered?
Were the response rates reported fully, including details of
participants who were unsuitable for the research or refused to take
part?
Have any potential response biases been discussed?
Coding and analysis
What sort of analysis was carried out and was this appropriate? (eg
correct statistical tests for quantitative answers, qualitative analysis
for open ended questions)
What measures were in place to maintain the accuracy of the data,
and were these adequate?
Is there any evidence of data dredging—that is, analyses that were not
hypothesis driven?
Results
What were the results and were all relevant data reported?
Are quantitative results definitive (significant), and are relevant non-
significant results also reported?
Have qualitative results been adequately interpreted (e.g. using an
explicit theoretical framework), and have any quotes been properly
justified and contextualised?
Conclusions and discussion
What do the results mean and have the researchers drawn an
appropriate link between the data and their conclusions?
Have the findings been placed within the wider body of knowledge in
the field (eg via a comprehensive literature review), and are any
recommendations justified?