Basic Questionnaires
Basic Questionnaires
o Question Types
o Phrasing the Question
o Programming Instructions
o Ordering the Questions in the Questionnaire
Writing a questionnaire involves the following steps:
Identify everything that you need to measure.
Questions that you need to answer. For example, whether people are interested in
buying a new product.
Diagnostics that will help you understand the answers to your key questions. For
example, reasons for liking or disliking a new product.
Questions that will be useful for profiling. For example, age and gender.
Questions that are useful for checking the representativeness of the study.
Writing one or more question for each thing that you need to measure
This involves:
Selecting the Question Type for each question
Phrasing the Questions.
Writing Programming Instructions.
Ordering the Questions in the Questionnaire.
Questions in a questionnaire are ordered so as to reduce the time taken to complete the
questionnaire and to reduce order effects.
Question Types
Open-Ended Questions
Please enter the name of the last soft drink you bought.
_____________
Most data collection programs will let you constrain the type of responses. For example,
when collecting numeric data, respondents have to provide a numeric response, and the range
of possible responses may be limited:
How many glasses of wine did you drink last night? ____
Open-ended questions are the easiest type of question to ask. However, they should generally
be used sparingly,[1] as they take longer for respondents to answer and many respondents will
put so little effort into answering the questions that their data becomes unhelpful.
Single Response Questions
Are you...
o Male
o Female
When writing a good single response question it is important to:
Ensure that each person will only fit into one and only one category.
Ensure that each person can find an appropriate category. Often this is achieved by
leaving an “Other” option. It is typically a good idea to require that the respondent
type what they mean into a box, as otherwise you can end up with a lot of responses
that are not intepretable.
Where the question forces people to compare the options (e.g., Which of these best
describes your personality?", provide a maximum of seven responses, as otherwise by
the time people have gotten to the end of the list they will have forgotten the earlier
responses).
Where the question is only requiring a simple act of recall (e.g., Where did you go for
your last holiday), order the responses alphabetically so they can quickly find the
appropriate response (this is particularly important when there are large numbers of
options).
Multiple Response Questions
This question type is also known as a pick any or a ‘multi’ or ‘multicode’ question.
Which of the following have you bought in the past week? Tick all that apply.
[] Coke
[] Pepsi
[] Fanta
[] None of these
Grid Questions
There are a variety of questions which are referred to as ‘grid’ questions. They differ in terms
of how they permit the respondent to choose options, with common alternatives being to limit
respondents to choosing a single option in each row, allowing them to tick multiple options in
each row, sliders and text boxes for entering numeric responses.
One response per row
Please rate your satisfaction with the following restaurants
Lo Hig
Medium
w h
McDonald’s o o o
Burger King o o o
Wendy’s o o o
Multiple responses per row
Which of these brands are Fun Sexy Masculine Coke [] [] [] Pepsi [] [] [] Fanta [] [] []
Typically, such questions are best instead asked as a series of multiple response questions, as
respondents can be daunted or confused by such a grid layout.
Numeric responses
In the past month, how many flights did you take on... Qantas United SAS Economy ______
______ ______ Premium Economy ______ ______ ______ Business Class ______ ______
______ First Class ______ ______ ______
Date Questions
Date questions can either be similar or open-ended questions, such as in the example below,
or can use a data selector (such as a calendar).
What is your date of birth?
____ / ____ / ____
Ranking Questions
The simplest ranking questions use a series of open-ended questions: Rank the following
brands according to how much you like them... Please place a 3 next to the brand you like
most, a 2 in your next preferred brand and a 1 next to your least preferred brand.
Coke ____
Pepsi ____
Fanta ____
More sophisticated versions use dragging and dropping.
Constant-Sum Questions
A constant-sum question asks the respondent to allocate a specific number of points, tokens,
dollars or some other quantity across different alternatives. For example:
If you had $10 to spend on soft drinks, how much would you spend on each of the following?
__ Coca-Cola __ Diet Coke __ Coke Zero __ Fanta
More advanced question types
There are many more advanced question types. Some of these are described in Advanced
Questions and Questionnaires.
Phrasing the Question
Writing a question involves selecting a question type and phrasing the question. For the
novice researcher, the secret to wording questions well is to copy others’ work. Plagiarism is
at the heart of asking good questions, as there are many, many ways of wording a question
poorly. The easiest way to get across the basics is to look at a few examples of bad questions
(all from real studies): What is your principal brand of soft-drink? What problems can you
see with this question? ‘Principal’ is an overly technical word. A better word than principal is
‘main’. However, this does not remove the ambiguity from the question: is it asking which
one is liked the most, or bought the most often? Such a question will likely end up measuring
brand salience (which brands come to mind) rather than anything else.[1]
And another example:
Do you prefer… [] Beer [] Wine [] Spirits [] Non-alcohol drinks
There are lots of problems with this question. What about cider drinkers? They are ignored,
as the categories are not exhaustive. Furthermore, people who do drink are likely to drink
different types of alcohol at different times, such as beer after sport, wine with dinner and
spirits after dinner; how should such a person answer the question?
Key principles when phrasing a question
Principle 1: The question must be answerable
The least-educated person who is likely to have to answer the question needs to be able to
figure out how to answer the questions accurately.
Principle 2: The question must not be ambiguous
The issue is not one of respondent confusion, as this is already addressed by the first
principle, and respondents are remarkably adept at answering incomprehensible questions
(presumably because this is the only way they can finish many questionnaires). The problem
with ambiguity is to avoid situations where, once we have collected the data, we cannot
discern what it means. For example, one study asked flyers about the importance of getting
‘discounts’ and found that it was important. However, the research did not clarify if the flyers
were happy with a 2% or 20% discount. As a result, the research was ambiguous and could
not be used to derive valid insights. To avoid being ambiguous, questions need to focus on
the current, the specific and the real.[2] A trick to achieving this is to focus on writing
questions around the 5Ws – who, what, where, when and why.[3]
A good way of checking for ambiguity is to use think aloud interviews, in which real
respondents are asked to answer the question, but are required to verbalize all their thoughts
while answering the question.
Principle 3: Incentive compatibility
The third principle is that questions must be incentive compatible, which is a term of art in
economics, and it means that questions need to be written in such a way that people have an
incentive to provide honest data. Consider the question:
Are you aged under 35? [] Yes [] No
If asked as a screener at the very beginning of a questionnaire, respondents conclude that an
answer of No will cause them to be screened out of the study (i.e., asked no more questions).
Consequently, when this question is asked at the beginning of questionnaires where people
are being paid more if they complete the whole questionnaire, some respondents lie and
pretend they are aged less than 35, when they are actually older. By contrast, if asked at the
beginning of a questionnaire where people are not being paid to do the study, people aged
under 35 pretend to be older, as this becomes a polite way of refusing to participate. An
incentive compatible way of screening people based on being aged under 35 instead asks:
How old are you? [] Under 18 [] 18 to 24 [] 25 to 34 [] 35 to 44 [] 45 to 54 [] 55 to 64 [] 65 or
more
and then screens people out if they select one of the older age groups. It is “incentive
compatible” because there is no obvious incentive to answer the question dishonestly.
Imagine yourself charged with the problem of pricing an iPad before they were launched.
How much should you charge? There were no competitors, and thus no straightforward way
of working out even a ballpark price point. A simple approach to this problem is to ask people
how much they might pay:
What is the most you would pay for the product?.
The amount that people nominate is referred to as either the reservation price or
their willingness-to-pay for the product. The analysis then proceeds by assuming that people
will buy a product if its price is less than or equal to the amount nominated by the
respondents.
Such questions are not incentive compatible. Imagine that you loved the concept and would
be prepared to pay $2,000 for an iPad. Would you tell this to an interviewer? If you did, it
could result in the product being sold at $2,000. Whereas if you lied and said $1,000, you
may end up with the product being sold for less and you not having to spend so much. Sure
some respondents may be honest, either because they are by nature honest or because they
believe that by being honest they will maximize their chance of the product they wish to buy
being sold, but others may lie. Observing that respondents have an incentive to lie does not
prove that they will lie (unless one is an economist). However, it is better to figure out ways
to conduct research that are incentive compatible, as then the risk is removed (this is
discussed in Advanced Questions and Questionnaires).
Frequency questions
Perhaps the most frequent mistakes made by novice researchers relate to frequency questions.
Consider this question:
How often do you go to the cinema? [] Never [] Rarely [] Sometimes [] Often
What do the scale points of Rarely, Sometimes and Often mean? One person’s often may be
another person’s sometimes. When measuring quantities it is always important to give people
precision about time and frequency, such as:
Thinking about the last twelve months, how many times did you go to the movies? [] None []
Once or twice [] 3 to 6 times [] 7 or more times
A problem with such a question is that respondents may not be able to answer it accurately
because they will not remember the precise number of trips to the cinema from the past
twelve months. Such a reservation is justified. A solution is to reduce the time interval:
Thinking about the last two weeks, that is, everything from today back to Wednesday two
weeks ago, how many times did you go to the movies? [] None [] Once [] Twice [] 3 to 6
times [] 7 or more times
Asking about such a short time period creates a different problem. Attendance at the cinema
is sporadic for most people. There is a big difference between somebody who goes six times a
year and somebody who does not go at all. However, asking about the last two weeks can
result in such people being grouped together, and thus it is better to ask about the longer time
interval.
So, what is the best way to ask about frequency of going to the cinema? A question like:
Thinking about the last twelve months, how many times did you go to the movies? [] None []
Once or twice [] 3 to 6 times [] 7 or more times
perhaps remains the best approach, as even though many people will not be able to answer
with complete accuracy, most of the people who have not gone will be able to select None
and most that have gone more than seven times will be able to select 7 or more times,
whereas with the vaguer Rarely, Sometimes and Often categories, we can not make any
deductions about what they actually mean.
Another example:
How often do you visit the web pages of this site? [] Several times a day [] Once a day []
Once a week [] Once a month [] Less often than once a month [] This is the first time I have
visited
These categories are not mutually exclusive (anybody who can tick the last category can also
tick the second last category). They are also not exhaustive. What should somebody answer if
they visit two or three times a week?
Programming Instructions
Once the basic question wording has been finalized it needs have instructions added so that
the person who has to program the questionnaire in the data collection software can ensure
that it works as intended. Traditionally, these instructions are entered in capitals, italics or
another color. In addition to the instructions, sometimes people enter specific values for each
of the options. In the example below a Student is represented by a 1; this information is used
to determine how the categories are represented in the resulting data file.
Example
Q2. ASK IF AGED 15 OR OVER: To make sure we have a cross section of people in this
study, we need to know your work status. Which of these best describes your ''work status''?
SINGLE RESPONSE Student 1 GO TO Q4 Homemaker 2 GO TO Q4 Retired 3 GO TO Q4
Not working 4 GO TO Q4 Part-time worker 5 GO TO Q4 Full-time worker 6 Don’t
know/refused 7 END SURVEY
Ordering the Questions in the Questionnaire
Questions in a questionnaire are ordered so as to reduce the time taken to complete the
questionnaire and to reduce order effects.
Minimizing the time to complete the questionnaire
Screeners
A screener is a question that can be asked at the beginning of a questionnaire with the
objective of determining eligibility. For example, if conducting a study on cola drinkers, then
a question about cola consumption is asked at the beginning of the questionnaire so that non-
cola consumers are not asked the rest of the questionnaire (are screened out).
Skips
A skip is an instruction in a questionnaire which makes respondents skip questions for which
their answers are not required. For example, if a person has no children then they should skip
a question asking about the ages of their children.
Filters
Options that are not relevant to some respondents should not be shown (filtered). For
example, if a consumers has never heard of a product they should not then be shown that
product in a following question asking which products they consume most often.
Multiple response questions and grids
Questions with similar structures should be grouped together. For example, true/false
questions can be grouped together as multiple response questions; questions which share a
response scale can be grouped into grids.
Deleting questions
The shorter the questionnaire the better. Consequently, questions should only be included that
are believed to be essential.
Minimizing order effects
An order effect occurs when people give answers in questionnaires that are influenced to an
extent by the order with which the questions, or options in a question, are presented to
respondents.
Knowledge and salience
The simplest type of order effect occurs when earlier questions unintentionally educate a
respondent, or, make them think about a particular thing. For example, if a question that asks
people if they have heard of Qantas and then, later in the questionnaire asks them to type the
names of all the airlines for which they can recall recently seeing TV advertisements, there is
a good chance that Qantas will get a higher result in the latter question than otherwise.
Leading questions
Leading questions cause respondents to change their opinions, which can effect either the
answer of the question that is leading, or, the answers to following questions. Consider the
following from the classic British sitcom Yes Prime Minister:[1]
Sir Humphrey: “You know what happens: nice young lady comes up to you. Obviously you
want to create a good impression, you don’t want to look a fool, do you? So she starts asking
you some questions: Mr. Woolley, are you worried about the number of young people without
jobs?”
Bernard Woolley: “Yes”
Sir Humphrey: “Are you worried about the rise in crime among teenagers?”
Bernard Woolley: “Yes”
Sir Humphrey: “Do you think there is a lack of discipline in our Comprehensive schools?”
Bernard Woolley: “Yes”
Sir Humphrey: “Do you think young people welcome some authority and leadership in their
lives?”
Bernard Woolley: “Yes”
Sir Humphrey: “Do you think they respond to a challenge?”
Bernard Woolley: “Yes”
Sir Humphrey: “Would you be in favour of reintroducing National Service?” [A requirement
that people join the army after leaving school.]
Bernard Woolley: “Oh…well, I suppose I might be.”
Sir Humphrey: “Yes or no?”
Bernard Woolley: “Yes”
Sir Humphrey: “Of course you would, Bernard. After all you told me can’t say no to that. So
they don’t mention the first five questions and they publish the last one.”
Bernard Woolley: “Is that really what they do?”
Sir Humphrey: “Well, not the reputable [market research firms] ones no, but there aren’t
many of those. So alternatively the young lady can get the opposite result.”
Bernard Woolley: “How?”
Sir Humphrey: “Mr. Woolley, are you worried about the danger of war?”
Bernard Woolley: “Yes”
Sir Humphrey: “Are you worried about the growth of armaments?”
Bernard Woolley: “Yes”
Sir Humphrey: “Do you think there is a danger in giving young people guns and teaching
them how to kill?”
Bernard Woolley: “Yes”
Sir Humphrey: “Do you think it is wrong to force people to take up arms against their will?”
Bernard Woolley: “Yes”
Sir Humphrey: “Would you oppose the reintroduction of National Service?”
Bernard Woolley: “Yes”
Sir Humphrey: “There you are, you see Bernard. The perfect balanced sample.”
Guiding and educating respondents
The opposite to leading a respondent is to guide them, as occurs in Probing and Laddering. In
studies for new products it is routine to progressively introduce more information about the
products throughout the questionnaire, so that the respondent is gently introduced to the
information and also so that the researcher can understand the incremental impact of the
additional information.
Fatigue
Harder questions should generally be asked towards the beginning of the questionnaire, so
that if respondents become fatigued while completing the questionnaire, the fatigue occurs
after they have already completed the more challenging questions. The order of questions can
also be used to maximize the energy of the respondent, mixing boring questions with more
interesting questions (e.g., projective questions).