VALIDITY
AND
RELIABILITY OF
QUESTIONNAIRES
CONTENTS
Introduction
Steps in questionnaire designing
Validity
Concept of validity
Types of validity
Steps in questionnaire validation
Reliability
Types and measurement of reliability
Conclusion
References
INTRODUCTION
Questionnaire: Important method of data collection used
extensively
Advantages of questionnaire
Less expensive
Offers greater anonymity
Disadvantages
Application is limited
Response rate is low
Opportunities to clarify issues is lacking
Ideal requisites of a questionnaire:
Should be clear and easy to understand
Layout is easy to read and pleasant to eye
Sequence of questions easy to follow
Should be developed in an interactive style
Sensitive questions must be worded exactly
The terminologies research instrument, measuring instrument, scale
and test in various parts of the seminar represent questionnaire in
this context . . . And item represents each question in a questionnaire
Validity
The concept of validity
Validity is the ability of an instrument to measure what it is intended to measure.
Degree to which the researcher has measured what he has set out to measure
(Smith, 1991)
Are we measuring what we think we are measuring? (Kerlinger, 1973)
Extent to which an empirical measure adequately reflects the real meaning of
the concept under consideration (Babbie, 1989)
Why validity ?
Validity is done mainly to answer the following questions:
Is the research investigation providing answers to the research questions for
which it was undertaken?
If so, is it providing these answers using appropriate methods and
procedures?
Questions to ponder
Investigator
Readers of
report
Experts in the
field
Logic
Statistical tests
Types of validity
Validity
Content Criterion Construct
validity related validity
Face Concurre
Predictive
validity nt
CONTENT VALIDITY
Uses logical reasoning and hence easy to apply
Extent to which a measuring instrument covers a representative
sample of the domain of the aspects measured
Whether items and questions cover the full range of the issues or
problem being measured
FACE VALIDITY
The extent to which a measuring instrument appears valid on its
surface
Each question or item on the research instrument must have a logical
link with the objective
Face validity is not content validity. Why?
Face validity
Simply addresses whether a measuring instrument looks valid
Not a validity in technical sense because it does not refer to what
is actually being measured rather what it appears to measure
It has more to do with rapport and public relations than with
actual validity
Other aspects of content validity
Coverage of issue should be balanced
Each aspect should have similar and adequate representation in
questions
Problems associated with content validity
Based on subjective logic; no definitive conclusion can be drawn or
consensus reached
Extent to which questions reflect the objectives of the study may
differ. If wordings changed or question substituted, magnitude of
link changes
CRITERION VALIDITY
The extent to which a measuring instrument accurately predicts
behaviour or ability in a given area.
The measuring instrument is called ‘criteria’
It is of two types:
Predictive validity
Concurrent validity
Predictive validity
If the test is used to predict future performance
Eg: Entrance exam . . . . Performance of these tests correlates with
later performance in professional college
Eg: Written driving test
Eg: measurement of sugar exposure for caries development
Concurrent validity
If the test is used to estimate present performance or person’s
ability at the present time not attempting to predict future
outcomes
Professional college exam
Eg: driving test, pilot test
Eg: measurement of DMFT for caries experience
Problems in criterion validity
Cannot be used in all circumstances
Esp in social sciences where some conditions do not have a
relevant criteria
Eg: for measuring self-esteem, no criteria can be applied
CONSTRUCT VALIDITY
Most important type of validity
Assesses the extent to which a measuring instrument accurately
measures a theoretical construct it is designed to measure
Measured by correlating performance on the test with performance
on a test for which construct validity has been determined
Eg: a new index for measuring caries can be validated by
comparing its values with a standard index (like DMFT)
Another method is to show that scores of the new test differs
across people with different levels of outcomes being measured
Eg: Establishing the validity of a new caries index by applying it to
different stages of dental caries and calculating its accuracy
Summary of Validity
CONTENT CRITERION CONSTRUCT
CONCURREN PREDICTIVE
T
What it Whether the The ability of The ability of The extent to
measures test covers a the test to the test to which the
representative estimate predict future instrument
sample of the present performance measures a
domains to be performance theoretical
measured construct
How it is Ask experts to Correlate Correlate Correlate
accomplishe assess the performance performance performance
d test to on the test on the test on the
establish that with a with a instrument
the items are concurrent behaviour in with a
representative behaviour future performance
of the on an
outcome established
Steps in
questionnaire
validation
FACE VALIDITY
Evaluate in terms of:
Readability
Layout
Feasibi
and
lity
style
Clarity of
wording
CONTENT VALIDITY
Two phases
Specify the full
domain of content
that is relevant
Experts: to of
Enhancement
content
the issue of questionnaire
(Seven or more experts)
Sample specific
areas form this
domain
Put items/questions
Researcher: Conceptualization
and domain analysis in a form
that is
testable
Reliability
RELIABILITY
Definition: It is the ability of an instrument to create reproducible
results
Each time it is used, similar scores should be obtained
A questionnaire is said to be reliable if we get same/similar
answers repeatedly
Though it cannot be calculated exactly, it can be measured by
estimating correlation coefficients
Reliability measured in aspects of:
• Done to ensure that same results
are obtained when used
STABILITY consecutively for two or more
times
• Test-retest method is used
• To ensure all subparts of a
INTERNAL instrument measure the same
CONSISTENCY characteristic (Homogeneity)
• Split-half method
• Used when two observers study a
single phenomenon
EQUIVALENCE simulataneously
• Inter-rater reliability
Test-Retest reliability (for stability)
Test administered twice to the same participant at
different times
Used for things that are stable over time
Easy and straight-forward approach
Useful for questionnaires, checklist, rating scales etc
Disadvantages
Practice effect (mainly for tests)
Too short intervals in between (effect of memory)
Some traits may change with time
Statistical calculation
Administration of instrument to a sample on two
different occasions
Scores compared and calculated by using correlation
coefficient formula (pearson)
Correlation coefficient
Measures the degree of relationship between two sets of scores
Can range from -1 to +1
0 indicates absence of any relationships
Correlation coefficient Strength of
relationship
+/- 0.7 to 1.0 Strong
+/- 0.3 to 0.69 Moderate
+/- 0.0 to 0.29 None to weak
Summary of Reliability
TEST SPLIT HALF INTERRATER
RETEST
What it Stability over Equivalency of Agreement
measures time items between raters
How it is Administer Correlate Have multiple
accomplishe the same test performance for researchers
d to the same a group of measure same
people at two people on two instrument and
different equivalent determine
times halves of same percentage of
test agreement
between them
Conclusion
Validated questionnaire
It is one which has undergone a validation procedure to show
that it accurately measures what it aims to, regardless of who
responds, when they respond, and to whom they respond or
when self-administered and whose reliability has also been
examined thereby:
Reducing bias and ambiguities
Better quality of data and credible information
In a nutshell . . . .
A questionnaire can be reliable but invalid . . .
But a valid questionnaire is always reliable . . .
References
Linda Del Greco, Walop W, Richard H McCarthy. Questionnaire development: 2.
Validity and Reliability. CMAJ. 1987;136:699–700.
Sushil S, Verma N. Questionnaire validation made easy. Eur J Sci Res.
2010;46(2):172–8.
Polit DF, Cheryl Tatano Beck. The Content Validity Index: Are You Sure You Know
What’s Being Reported? Critique and Recommendations. Res Nurs Health.
2006;29:489–97.
Reliability and Validity Module 6. Cengage Learning; 2010.
Rama B Radhakrishna. Tips for Developing and Testing
Questionnaires/Instruments. J Ext. 2007;35(1):710–4.
[Link] [Internet]. [cited 2015 Apr 7]. Available from:
[Link]
pta_6871_6791004_64131.pdf [Internet]. [cited 2015 Apr 7]. Available from:
[Link]
Questionnaire designing and validation [Internet]. [cited 2015 Apr 7]. Available
from: [Link]
Suresh K Sharma. Nursing Research and Statistics. 1st ed. New Delhi: Elsevier
Saunders;
Edward G, Richard Zeller. Reliability and Validity Assessment. New Delhi: SAGE
publication; 1979.
Ranjit Kumar. Research Methodology - A step by step guide for beginners. 3rd
ed. New Delhi: SAGE publication; 2012.
Articles from Dr. Joe
THANK
YOU