STEPS IN INSTRUMENT

DEVELOPMENT
Dr. Akila Ganesh, MDS, PhD,
Professor and Head
Department of Public Health Dentistry
Sri Ramachandra Dental College and Hospital
Sri Ramachandra Institute of Higher Education and Research
 CONTENTS
1
• Tool - introduction

2 • Types of questions

3 • Steps in tool development

4 • Common errors

5 • Writing good questions

6
• Key points

7
• Conclusion
INTRODUCTION - QUESTIONNAIRE
• Set of questions trying to measure different
aspects of the same trait

• Inventory – list of choices

• Scale – set of possible answers to a question

(on a scale of 1 to 10)

• Information that cannot be assessed by physical

examination
QUESTIONNAIRE – INFORMATION
MEASURED?

Psychological Knowledge/ Behaviors

constructs awareness/ (practices)
attitudes/beliefs
QUESTIONNAIRE - OPTIONS

Use an existing Modify an Develop a new

questionnaire existing questionnaire
questionnaire
A WELL DESIGNED QUESTIONNAIRE

Meet the objectives of the study

Obtain the most complete and accurate information

possible

To be done within limits of available time and

resources
METHODS OF ADMINISTRATION

Rater administered

Self administered

Telephone interview

Internet
 Questions

Open ended Close ended

Close ended Close ended Partially close

with ordered with unordered ended
choices choices
CLOSE ENDED QUESTIONS - SCENARIOS

Interview Less Partially

quick & demanding One wants close ended
easy for for spontaneous – generates
statistician reply new
respondents information
OPEN ENDED QUESTIONS - SCENARIOS

Very
demanding for
Too many respondents Researcher
categories to be & wants to Well suited
listed provide an when a precise
Rarely provide opportunity for piece of
& accurate respondents to information is
Least amount of measurement or express needed
effort to write consistent/ themselves
comparable
information
Open ended questions Close ended questions
Advantages
Advantages
-Respondent answers using own words
-Greater uniformity of results
-Respondent not influenced by any
specific alternative suggested by -Easier to code, process and analyse
interviewer -Used for comparing responses across
-Reveals what is most salient to different groups
respondent
-Suggest answers that may not been
thought of
Disadvantages
-Lot of irrelevant material Disadvantages
-Individuals differ in their ability to -Could force respondent into selecting
articulate inappropriate options
-Analysis – tedious and demanding -May not permit expression of exact
shades of opinion
CATEGORY ORDER EFFECT
In questions with a long list of unordered responses

Mail – select first response

Telephone/face to face survey – select last responses

Increases measurement error

 STEPS IN TOOL DEVELOPMENT
Cognitive
Choose title interviewing Pre testing

Formation of
Focus group
a conceptual discussion Validity
framework

Item pool Refinement of Reliability

generation item pool

Vaughn AE, Tabak RG, Brynt MJ, Ward DS. Measuring parent food practices: a systematic review of existing
measures and examination of instruments. Int J Behav Nutr Phys Act 2013;10:61.
FEAST SCALE
Feeding At Sleep Time (FeAST) scale
Night time feeding practices – risk factor in ECC
Sleep time feeding practices – must be considered
Lack of such a scale
FORMATION OF A CONCEPTUAL
FRAMEWORK

Literature Existing One’s

search questionnaire knowledge and
opinion
FEEDING AT SLEEP TIME (FEAST) SCALE

Conceptual
framework

Breast feeding Bottle feeding Other feeding

construct construct modes
 ITEM POOL GENERATION
Items must address the research
question

Must be suitable to method of

administration

Phrased at participant’s level of

understanding
Parmenter K, Wardle J. Development of a general nutrition knowledge questionnaire for adults. Eur J
Clin Nutr 1999;53:298-308.
 ITEM POOL GENERATION - SOURCES

1 2 3 4
Clinical Expert
observation Research Theory
opinion
ITEM POOL GENERATION - SELECTING ITEMS

Reading level • Use short sentences

Ambiguity • Question wording lacks ambiguity

Double • Deal with only one central thought

barreled
Use of • Lead to misunderstandings
negatives • Respondents puzzled and give wrong answers
ITEM POOL GENERATION – SELECTING ITEMS
Double • Avoid
negatives • Weaken robustness of question

Value laden • Avoid items that will prejudice respondents

words
Leading • Respondents tends to answer in a way they desire to be socially
questions desired by interviewer

Hypothetical • Forces subject to think of something they have never considered

questions • Not produce clear and consistent data representing real opinion
ITEM POOL GENERATION

Certain
• Inappropriate questions
degree of can be removed later
redundancy

• Retain weak items

Conservative • assess their performance
approach in subsequent steps
• Difficult to restore later
ITEM POOL GENERATION
REFINEMENT OF ITEM POOL

Focus group Cognitive

discussion interviewing Pre testing
FOCUS GROUP DISCUSSION
Directed group discussion or focus group

Organised discussions led by a moderator

Volunteer to participate - given an incentive

Does not comprise a sufficiently large sample – 8 to 10 people

Participants are not randomly selected -similar in some important way

FOCUS GROUP DISCUSSION
Purpose of FGD - Briefed

What problem are you trying to solve?

What new information do you need to solve it?

Use informal loosely structured interviews

Audio and video recorded

 FOCUS GROUP DISCUSSION
Head start – knowing which question to ask in a survey

Does not substitute for quantitative surveys

Generate ideas or hypothesis for later testing

Questions and answers are not formulated ahead of time

Broder HL, McGrath C, Cisneros GJ. Questionnaire development: face validity and item impact testing of the
Child Oral Health Impact Profile. Community Dent Oral Epidemiol 2007;35(Suppl 1):8-19.
 FOCUS GROUP DISCUSSION -QUESTION PROBES

‘What do you think this section is testing?’

‘Are you unfamiliar with any of the terms used in this

question?’

‘Do you find this question confusing or intentionally

misleading?’

Broder HL, McGrath C, Cisneros GJ. Questionnaire development: face validity and item impact testing of
the Child Oral Health Impact Profile. Community Dent Oral Epidemiol 2007;35(Suppl 1):8-19.
FOCUS GROUP DISCUSSION
COGNITIVE INTERVIEWING
Focuses on the questionnaire

Explicitly focuses on cognitive processes

Done soon after initial drafting of the questionnaire

Trained interviewers conduct interviews with volunteer participants

COGNITIVE INTERVIEWING
A useful means to test questionnaire – DOES
NOT VALIDATE IT

Does not in itself result in improved questionnaire

Volunteer participants representative of target

population recruited and interviewed

Helps identify potential problems in items

COGNITIVE INTERVIEWING – PURPOSE
Test the response scale – their adequacy, relevance and
understanding

Assess the comprehensiveness, relevance and clarity of expression of

the formulated items

Generate new items /modify existing items from the opinions of the
participants

Elimination of irrelevant items and rephrasing of items from the scale

Reducing ambiguity and maximising clarity of the questions.

 ‘THINK ALOUD’ METHOD
ADVANTAGES DISADVANTAGES

Freedom from interviewer imposed Need for participant training

bias

Participant resistance
Minimal interviewing training
requirements
Burden on participant
Open ended format
Tendency for participant to stray from
task
 COGNITIVE SURVEY
INTERVIEWING
Collects data
Analyses questions
COGNITIVE INTERVIEW - QUESTION PROBES
Was this question hard or easy to answer?

Determine comprehension Ability to re call

How do you remember this?

To study recall strategy

How sure are you of your answer?

Confidence probe

What according to you is ‘ hard work(for eg.)?’

Interpretation of a specific term

COGNITIVE INTERVIEW - QUESTION PROBES
Did this question make sense to you? If not, why?

Will this make sense to people like you?

Will this question be understood differently?

Does this question make you feel uncomfortable?

Is there a better way to ask this?

COGNITIVE INTERVIEW – SUMMARY
Conducted with 5 to 10 participants

Emphasis on comprehension, question wording, format

Interviewers write up comments on each interview

Meet as a group to discuss results

Propose potential modifications, then revise questionnaire

TRANSLATION

• Evaluate the quality of instrument and

1 compare its use in the target culture

• Translate and back translate the

2 instrument

• Pilot test the instrument

3
TRANSLATION - STEPS
• Instrument translated from
Translation source to target language

Back • Instrument translated from

target to source language
translation
• Both versions of source
Finally language compared
TRANSLATION - QUALITY EQUIVALENCE
Similarity in meaning of concept when translated
into another language

Many challenges to adequately developing

questionnaires for multilingual use

Researchers inexperienced in cross cultural

studies, underestimate importance of translation

Many unfamiliar with its implications for

reliability and validity.
ISSUES TO CONSIDER

1 • Use simple sentences

2 • Use colloquial words, with caution

3 • Minimise redundancy in sentences

4 • Avoid too much detail

5 • Allocate budget for translation

PRE-TESTING

1 • Members of target audience interviewed –

accurate capture of their vernacular

2 • Mandatory

3 • Primary rationale – eliminate items that the

participants perceived as unimportant

4 • Jargon and technical terms used with caution

Dawis RV. Scale Construction. J Couns Psychol 1987;34: 481-489

PRE TEST INSTRUMENT

Conduct an initial study to assess usefulness of instrument

Make revisions based on results of initial study

Re evaluate the instrument, give revised instrument to

another group of subjects
PRE TESTING – TO CHECK FOR
Understanding of the questionnaire

Wording of terms

Adequacy of responses

Any regional differences

If filter questions and skip patterns are adequate

PRE TESTING

• Frequency of If too many respondents

endorsement do not want to answer
the same question

Consider
eliminating it
FEAST SCALE
VALIDITY TESTING
‘Subject Matter Experts’

Checked whether items reflected what they wanted to measure

Assess response options

Feasibility of using the instrument

Remarks were consolidated and duly incorporated

VALIDITY
 Face validity

Evaluated by a More the Calculates the

group of Subjective experts, higher amount of
experts process the face agreement
validity among experts
 Content validity
Extent to
which
questionnaire Cover Relies on
probes various representative Use sensible
aspects of collection of methods of past
areas it is items construction experience
suppose to
measure
 Criterion validity

Looks at how new Use of a gold

measures relates to standard/ Potential problems –
other measures or benchmark - Good finding a ‘Gold
diagnosis indicator standard’
 Construct validity

- Also called - Good constructs:

clear operational More a construct is
factorial validity used by researchers
defns. involving
measurable in more settings,
- Most difficult constructs more its construct
validity to measure! validity
RELIABILITY

Consistency of measurement
Degree to which an instrument measures the same
way each time it is used under the same condition
with the same participant
Ranges from 0 to 1
 Test re-test reliability

At two No change Average two

Assesses in weeks
consistency Most separate underlying Problems of
across time conservative times for interval
method each condition attrition
between test (10 - 14
participant 1 and test2 days)
 Internal consistency

Single Judging by
To assess measurement estimating how
consistency of instrument well the items Based on inter
results across administered to that reflect the correlation
items within a a group of same construct between items
test people on one yield similar
occasion results
RELIABILITY

• Kappa
Test – re test • 0.81 - 1

• Crohnbach’s
Internal alpha
consistency • 0.7 - 1
 VALIDITY/RELIABILITY
Repeated administration of a scale yields more
or less the same scores in stable participants -
fundamental property - defines the scale's
reproducibility and validity.

VALIDITY RELIABILITY
Theoretical aspect Statistical aspect

Guyatt GH, Deyo RA, Charlson M, Levine MN, Mitchell A. Responsiveness and validity in health status
measurement: a clarification. J Clin Epidemiol 1989;42:403-408.
FRONT COVER -DESIGN

Well
designed
cover

Informative Graphic Name of

title design sponsor
BACK COVER - DESIGN

Back cover

Invitation to Plenty of Thank you – Address to be Toll free

make blank space bottom of the returned number if they
comments page need help
COMMON ERRORS

Age • Consider getting date of birth

Income • Provide ranges to choose from

Relationship • Ask only if needed for the research

status, Religion
Occupational • Never open ended
status • Additional time - recoding
WRITING GOOD QUESTIONS

Be specific Don’t be vague

Use simple words Keep it short

Don’t talk down to respondents Don’t be too specific

KEY POINTS

Initial item • Great care

pool • Include 3 times of the finalised version

• Cognitive interviewing
Vital steps • Pre testing
• Expert review

Content • FGD – Qualitiative component

• Imperative to include a gamut of items
validity from various possible views
CONCLUSION
Critical to success
QUESTIONS
Independently built

Placed relative to each

other & then connected –
NEVER LEFT TO CHANCE
PILLARS
 REFERENCES
Salant P, Dillman D. How to conduct your own survey. John Wiley and sons
Handbook – Workshop on Instrument development. Organised by Samarth and
Scarf Collaboration
Vaughn AE, Tabak RG, Brynt MJ, Ward DS. Measuring parent food practices: a
systematic review of existing measures and examination of instruments. Int J
Behav Nutr Phys Act 2013;10:61.
Trakman GL, Forsyth A, Hoye R, Belski R. Developing and validating a nutrition
knowledge questionnaire: key methods and considerations. Public Health Nutr
2017;20:2670-2679.
Broder HL, McGrath C, Cisneros GJ. Questionnaire development: face validity
and item impact testing of the Child Oral Health Impact Profile. Community
Dent Oral Epidemiol 2007;35(Suppl 1):8-19.
REFERENCES
DeVellis RF. Scale Development: Theory and Applications. 2016. New
York. Sage Publications. Inc.
McGrath C, Yeung RWK, Comfort MB, McMillan AS. Development and
evaluation of a questionnaire to evaluate clinical dental teachers
(ECDT). Br Dent J 2005;198:45-48.
Parmenter K, Wardle J. Development of a general nutrition
knowledge questionnaire for adults. Eur J Clin Nutr 1999;53:298-308.
Guyatt GH, Deyo RA, Charlson M, Levine MN, Mitchell A.
Responsiveness and validity in health status measurement: a
clarification. J Clin Epidemiol 1989;42:403-408.
If you don’t ask the right question,
you don’t get the right answer
-Edward Hodnett
Thank you!!!