Surveys

• Surveys gather information from many people in a short

period of time
• Surveys are useful for understanding correlations between
variables (cross-sectional), or cause and effect relationships
(longitudinal)
• Surveys can be used for exploratory, descriptive,
and explanatory studies
 Surveys
• Advantages:
• Economy of time and effort
• Large samples
• Standardization of responses
• High reliability
• Disadvantages:
• Somewhat artificial, superficial
• Relatively inflexible
• Low validity
 Good Survey Questions
• Questions should be administered and understood in a
consistent way
• A good question reliably communicates to all respondents the
type of answers that are acceptable
 Good Survey Questions
• Avoid making assumptions about the level of knowledge of
your respondents
• All respondents should have access to the information needed to
answer the question
• Poorly worded:
• “Do you agree with the new drinking and driving
legislation?”
• Better:
• “Lawmakers propose to lower the blood alcohol limit for
driving to .05. Do you agree with the new legislation?”
 Item Clarity
• Avoid vague wording
• To the extent words must be used with meanings which may not
be shared, definitions should always be provided
• Poorly worded:
• “How many times did you exercise last week?”
• Better:
• “How many times did you exercise last week?
Exercise is defined as walking, running, swimming, or any
other sport in which you took part in for at least 30
minutes.”
 Item Clarity
• Poorly worded:
• “When did you move to Winnipeg?”
• Possible answers include:
• 1980
• When I was 10
• After I graduated from high school
• Better:
• “In what year did you move to Winnipeg?”
 Time Frames
• In helping people to report their past behavior accurately,
a suitable time frame for recall should be established
• Poorly worded:
• “How often do you drink and drive?”
• Better:
• “Over the last 30 days, how many times did
you drink and drive?”
 Negative Items
• Negative items introduce ambiguity
• Avoid using the word “not” which often
contributes to double-negatives
• Poorly worded:
• “Marijuana should not be
decriminalized”
• Better:
• “Marijuana should remain illegal”
 Double-Barreled Questions
• Double-barreled questions ask for a single answer to a
question with multiple parts (biased)
• Poorly worded:
• “Do you value your education and are your tuition fees
reasonable?”
• Better:
• “Do you value your education?”
• “Do you think your tuition fees are
reasonable?”
 Double-Barreled Questions
• As a rule, if the word “and” appears in a question it is
probably double-barreled
• Questions may be double-barreled even if they do not contain
the word “and”
• Poorly worded:
• “Do you exercise regularly to avoid heart trouble?”
• Possible answer:
• “Yes, I exercise regularly, but not to avoid heart trouble.”
 Biased Items
• Be careful not to word questions so a particular response
is favored over another
• Avoid using strong words which may skew responses
• Poorly worded:
• “Are you in favor of welfare programs?”
• Better:
• “Are you in favor of social assistance programs?”
 Threatening Questions
• A major problem with surveys is social desirability bias
• Respondents have little incentive to acknowledge undesirable
behaviors and often over report desirable behaviors
• To avoid social desirability bias, the researcher can use
“lead-ins” to make questions appear less threatening
• Poorly worded:
• “Did you vote in the last election?”
• Better:
• “Were you able to vote in the last election?”
 Threatening Questions
• Questions should be worded in the least threatening
manner possible
• Poorly worded:
• “Have you ever driven home drunk?”
• Better:
• “Sometimes people will drink alcohol – for example, at a
social event – after which they drive a motor vehicle. Have
you done this within the last twelve months?”
 Threatening Questions
• Minimize the detail in the answers
• Poorly worded:
• “What is your annual income?”
• Better:
• “Circle the answer that best represents your income”
• $0 and $9,999
• $10,000 and $19,999
• $20,000 and $29,999
• $30,000 and $39,999
•…
 Short Items Are Best
• When in doubt make questions shorter rather than longer
to avoid misinterpretation
• A general guideline is to write questions as if they were for
people with a grade 10 education
• Avoid asking more than you need to know
• Poorly worded:
• “Please indicate how many children you have and the
years each of them was born”
• Better:
• “What are the ages of your children?”
Types of Questions
 Open-ended Questions
• Open-ended questions are useful for gathering detailed, in-
depth information
• Open-ended questions leave a space for respondents to provide
their own answer to a question
• For example, “Describe what you like
and about your current job.”
 Open-ended Questions
• Advantages:
•  Provides full expression of thoughts and ideas
•  Distinctions can be drawn between answers
•  Unanticipated answers
•  High validity
• Disadvantages:
•  Coding/Summarizing
•  Vagueness
•  Greater respondent involvement
•  Low reliability
 Close-ended Questions
• Closed-ended questions ask the respondent to select an
answer from among a list provided by the researcher
• “How satisfied are you with your job”
 Very satisfied  Somewhat satisfied  Neutral  Somewhat
dissatisfied  Very dissatisfied
• A full range of possible responses must be provided to
respondents
• There must be equal numbers of positive and negative
responses, as well as a “neutral” category
 Close-ended Questions
• Advantages:
•  Responses are uniform
•  Less demand placed on respondents
•  Recording is simplified
•  High reliability
• Disadvantages:
•  Inadequate response categories and loss
of spontaneity
•  Perceived superficiality of responses
•  Low validity
 Exhaustive
• There are two criteria for evaluating response categories:
 Exhaustive  Mutually exclusive
• Exhaustive – all possible responses expected are included
• Poorly worded:
• “What is the best part of living in Winnipeg”
 Nightlife  High Standard of Living
 Cheap to Live  Multicultural City
• Better:
•  Other: (Please specify) __________
 Mutually Exclusive
• Mutually exclusive – respondents should not be able to
select more than one response to a given question
• There should be no overlap in response categories
• Poorly worded:
0 - 10,000 10,000 - 20,000 20,000 - 30,000
30,000 - 40,000 40,000 - 50,000 etc.
• Better: 0-
9,999 10,000 - 19,999 20,000 - 29,999 30,000 -
39,999 40,000 - 49,999 etc.
 Instructions
• Instructions should always be provided at the beginning of
the questionnaire or after introducing a new topic in the
survey
• Never provide instructions prior to or after each question
• Please check the appropriate box
• Please circle the appropriate answer
• Indicate the response in the space provided
• Check all answers that apply to you
 Question Order
• The order in which questions appear can affect the
answers respondents provide
• Initial questions in a survey should not appear threatening
• Surveys should always include all relevant demographic data
(age, education, income, marital status, gender, place of residence,
etc.)
 Response Formatting
• The three most common types of response formats are
checking boxes, circling answers, and leaving blank spaces
(open-ended questions)
• Response formats can influence
the quality of the data collected
• Responses can be organized
vertically or horizontally
• The horizontal method saves
space and cost; however, the
overall readability suffers
 Matrix Questions
• Matrix questions are used when there is a set of questions
with the same answer categories
• Matrix responses use space effectively and make questionnaires
easier to complete
• A minimum of five
questions should
appear in a matrix
 Questionnaire Subsections
• It is important to separate different topics into different
sections
• Each section should have a heading to provide transitions
between topics
• Introduce each topic with a short statement regarding its
content and purpose
• For example, “We would like to examine attitudes towards
gambling in Manitoba to better inform policymakers about the
need for gambling support services like gamblers anonymous.”
 Contingency Questions
• A contingency question “filters” out certain respondents
depending on the relevancy of the questions asked in a
survey
• For example, the question “How satisfied are you with your
job?” is irrelevant to people who are unemployed
• Contingency questions require a clear format to ensure
respondents answer all the questions intended for them
 Contingency Questions
• Question 1: Are you currently employed?
• (1) Yes (2) No (If no, skip to question 3)

• Question 2: How satisfied are you with your job?

• (1) Very satisfied (2) Somewhat satisfied (3) Neutral
(4) Somewhat dissatisfied (5) Very dissatisfied

• Question 3: “What is your annual income?”

• (1) 0 - 9,999 (2) 10,000 - 19,999 (3) 20,000 - 29,999 (4) etc. …
 General Rules
• Survey rules:
•  The survey should be organized by topic sections
•  Questions should be organized within sections from least to
most threatening, easy to difficult, and general to specific
•  Never begin a question on one page and finish it on another
•  Surveys that are slightly longer but more spacious are
preferable to shorter surveys that look cluttered
•  No abbreviations should be used unless they are explicitly
defined
For more information on questionnaire design:
[Link]
 Pretesting
• Pretesting is used to determine whether:
• The questions and wording in a survey are understood and
convey consistent meaning
• The most likely answers to closed-ended questions are included
• Sufficient space is provided for answers to
open-ended questions
• The flow of the questions is in proper sequence
• The formatting is easy to follow
• Filter and skip instructions work as intended