Questionnaire Development

Process

Dr. Md. Zakir Hossain

Professor of Planning
Questionnaire

◼ Definition – A set of questions designed to generate

the information necessary to meet research
objectives
◼ Characteristics
◼ Elicits information from respondents
◼ Results can be tabulated
◼ Standardized across respondents
◼ Understandable to respondents
◼ A good questionnaire must:
◼ provide the necessary information
◼ be considerate of respondents
◼ meet coding and data processing requirements
 Open or closed questions?

◼ One of the most significant considerations for many researchers

is whether to ask a question in an open or closed format.
◼ The issue of whether to ask a question in an open or closed
format is relevant to the design of both structured interview and
self-completion questionnaire research.
◼ With an open question respondents are asked a question and
can reply however they wish. With a closed question they are
presented with a set of fixed alternatives from which they
have to choose an appropriate answer.
 Open questions?

Advantages:
◼ Respondents can answer in their own terms. They are not forced to answer in
the same terms as those foisted on them by the response choices.
◼ They allow unusual responses to be derived. Replies that the survey researcher
may not have contemplated (and that would therefore not form the basis for
fixed choice alternatives) are possible.
◼ The questions do not suggest certain kinds of answer to respondents.
Therefore, respondents’ levels of knowledge and understanding of issues can
be tapped. The salience of issues for respondents can also be explored.
◼ They are useful for exploring new areas or ones in which the researcher has
limited knowledge.
◼ They are useful for generating fixed-choice format answers.
 Open questions?

Disadvantages:
◼ They are time-consuming for interviewers to administer. Interviewees are likely to
talk for longer than is usually the case with a comparable closed question.

◼ Answers have to be ‘coded’, which is very time consuming. For each open question
it entails reading through answers, deriving themes that can be employed to form
the basis for codes, and then going through the answers again so that the answers
can be coded for entry into a computer spreadsheet. The process is essentially
identical to that involved in content analysis and is sometimes called post-coding to
distinguish it from pre-coding, whereby the researcher designs a coding frame in advance of
administering a survey instrument and often includes the pre-codes in the
questionnaire (as in Tips and skills ‘Processing a closed question’). However, in
addition to being time consuming, post-coding can be an unreliable process,
because it can introduce the possibility of variability in the coding of answers and
therefore of measurement error (and hence lack of validity).
 Open questions?

Disadvantages:
◼ They require greater effort from respondents. Respondents are likely to talk
for longer than would be the case for a comparable closed question, or, in
the case of a self-completion questionnaire, would need to write for much
longer. Therefore, it is often suggested that open questions have limited
utility in the context of self-completion questionnaires.

◼ There is the possibility in research based on structured interviews of

variability between interviewers in the recording of answers. This possibility
is likely to arise as a result of the difficulty of writing down verbatim what
respondents say to interviewers. The obvious solution is to employ a tape
recorder.
 Closed questions

◼ Advantages
Closed questions offer the following advantages to researchers.
◼ It is easy to process answers. For example, the respondent in a self-completion
questionnaire or the interviewer using a structured interview schedule will place
a tick or circle an answer for the appropriate response. The appropriate code can
then be almost mechanically derived from the selected answer, since the pre-
codes are placed to the side of the fixed-choice answers

◼ Closed questions enhance the comparability of answers.

◼ Closed questions may clarify the meaning of a question for respondents.

Sometimes, respondents may not be clear about what a question is getting at,
and the availability of answers may help to clarify the situation for them.
 Closed questions

◼ Closed questions are easy for interviewers and/or respondents to complete.

Precisely because interviewers and respondents are not expected to write
extensively and instead have to place ticks or circle answers, closed questions
are easier and quicker to complete.

◼ In interviews, closed questions reduce the possibility of variability in the

recording of answers in structured interviewing.
 Closed questions

Disadvantages
◼ There is a loss of spontaneity in respondents’ answers. There is always the possibility that they
might come up with interesting replies that are not covered by the fixed answers that are
provided.

◼ It can be difficult to make forced-choice answers mutually exclusive. The fixed answers that
respondents are provided should not overlap. If they do overlap, the respondents will not know
which one to choose and so will arbitrarily select one or the other or alternatively may tick both
answers. If a respondent were to tick two or more answers when one is required, it would mean
that you would have to treat the respondent’s answer as missing data, since you would not know
which of the ticked answers represented the true one. One of the most frequently encountered
forms of this problem can be seen in the following age bands:
◼ 18–30
◼ 30–40
◼ 40–50
◼ 50–60
◼ 60 and over
In which band would a 40-year-old position him- or herself?
 Closed questions

◼ It is difficult to make forced-choice answers exhaustive. All possible answers should

really be catered for, although in practice this may be difficult to achieve, since this
rule may result in excessively long lists of possible answers. Again, a category of
‘Other’ may be desirable to provide a wide range of answers.
◼ There may be variation among respondents in the interpretation of forced-choice
answers. There is always a problem when asking a question that certain terms may be
interpreted differently by respondents. If this is the case, then validity will be
jeopardized. The presence of forced-choice answers can exacerbate this possible
problem, because there may be variation in the understanding of key terms in the
answers.
◼ Closed questions may be irritating to respondents when they are not able to find a
category that they feel applies to them.
◼ In interviews, a large number of closed questions may make it difficult to establish
rapport, because the respondent and interviewer are less likely to engage with each
other in a conversation.
 Types of question

◼ There are various ways of classifying these, but here are some prominent types of
question:

◼ Personal factual questions. These are questions that ask the respondent to provide
personal information, such as age, education, occupation, marital status, income,
and so on. This kind of question also includes questions about behaviour. Such
factual questions may have to rely on the respondents’ memories, as when they
are asked about such things as frequency of church attendance, how often
they visit the cinema, or when they last ate out in a restaurant.

◼ Factual questions about others. Like the previous type, this one asks for personal
information about others, sometimes in combination with the respondent.
An example of such a question would be one about household income,
which would require respondents to consider their own incomes in
conjunction with those of their partners.
 Types of question

◼ Informant factual questions. Sometimes, we place people who are interviewed or who
complete a questionnaire in the position of informants rather than of
respondents answering questions about themselves.
◼ This kind of question can also be found in certain contexts, as when people
are asked about such things as the size of the firm for which they work, who
owns it, whether it employs certain technologies, and whether it has certain
specialist functions.
◼ Such questions are essentially about characteristics of an entity of which they
have knowledge, in this case, a firm. However, informant factual questions
may also be concerned with behaviour: the Charles and Kerr (1988)
questions are examples of this kind, in that the person being asked the
question is being asked to reply about behaviour in terms of the household
or family.
 Types of question

◼ Questions about attitudes. Questions about attitudes are very common

in both structured interview and self-completion questionnaire
research. The Likert scale is one of the most frequently
encountered formats for measuring attitudes.

◼ Questions about beliefs. Respondents are frequently asked about their

beliefs, possibly religious and political beliefs. Another form of
asking questions about beliefs is when respondents are asked
whether they believe that certain matters are true or false—for
example, a question asking whether the respondent believes the
UK is better off as a result of being a member of the European
Union.
 Types of question

◼ Questions about normative standards and values. Respondents may be

asked to indicate what principles of behaviour influence them or
they hold dear. The elicitation of such norms of behaviour is
likely to have considerable overlap with questions about
attitudes and beliefs, since norms and values can be construed as
having elements of both.

◼ Questions about knowledge. Questions can sometimes be employed to

‘test’ respondents’ knowledge in an area. For example, as part of
their study of the role of the mass media in the public
understanding of science, Hargreaves et al. (n.d.) asked survey
respondents to answer a large number of knowledge questions
relating to scientific issues.
 Rules for designing questions:

◼ General rules of thumb

Always bear in mind your research questions
The questions that you will ask in your self-completion questionnaire or
structured interview should always be geared to answering your research
questions. This first rule of thumb has at least two implications.

First, it means that you should make sure that you ask questions that relate to
your research questions. Ensure, in other words, that the questionnaire
questions you ask will allow your research questions to be addressed.

Second, it means that there is little point in asking questions that do not relate to
your research questions. It is also not fair to waste your respondents’ time
answering questions that are of little value.
 Rules for designing questions:

◼ General rules of thumb

◼ Rule of thumb number two is to decide exactly what it is you want to know.
Consider the seemingly harmless question:
Do you have a car?

What is it that the question is seeking to tap? Is it car ownership? If it is car

ownership, the question is inadequate, largely because of the ambiguity of the
word ‘have’. The question can be interpreted as: personally owning a car;
having access to a car in a household; and ‘having’ a company car or a car for
business use. Thus, an answer of ‘yes’ may or may not be indicative of car
ownership. If you want to know whether your respondent owns a car, ask him
or her directly about this matter.
 Rules for designing questions:

◼ General rules of thumb

Rule of thumb number three is to put yourself in the position of the respondent.
Ask yourself the question and try to work out how you would reply. If you do
this, there is at least the possibility that the ambiguity that is inherent in the
‘Do you have a car?’ question will manifest itself, and its inability to tap car
ownership would become apparent. Let us say as well that there is a follow-up
question to the previous one:
Have you driven the car this week?

Again, this looks harmless, but if you put yourself in the role of a respondent, it will be apparent that
the phrase ‘this week’ is vague. Does it mean the last seven days or does it mean the week in which
the questioning takes place, which will, of course, be affected by such things as whether the
question is being asked on a Monday or a Friday? In part, this issue arises because the question
designer has not decided what the question is about. Equally, however, a moment’s reflection in
which you put yourself in the position of the respondent might reveal the difficulty of answering
this question.
 Specific rules when designing questions

◼ Avoid ambiguous terms in questions

Avoid terms such as ‘often’ and ‘regularly’ as measures of frequency. They are
very ambiguous, because respondents will operate with different frames of
reference when employing them. Sometimes, their use is unavoidable, but
when there is an alternative that allows actual frequency to be measured, this
will nearly always be preferable. So, a question like:
How often do you usually visit the cinema?
◼ Very often ____
◼ Quite often ____
◼ Not very often ____

◼ Not at all ____

suffers from the problem that, with the exception of ‘not at all’, the terms in the
response categories are ambiguous.
 Specific rules when designing questions

Instead, try to ask about actual frequency, such as:

How frequently do you usually visit the cinema? (Please tick whichever category comes
closest to the number of times you visit the cinema)
◼ More than once a week ____
◼ Once a week ____
◼ 2 or 3 times a month ____

◼ Once a month ____

◼ A few times a year ____
◼ Once a year ____
◼ Less than once a year ____
 Specific rules when designing questions

◼ Avoid long questions

It is commonly believed that long questions are undesirable. In a structured

interview the interviewee can lose the thread of the question, and in a self-
completion questionnaire the respondent may be tempted to omit such
questions or to skim them and therefore not give them sufficient attention.
However, Sudman and Bradburn (1982) have suggested that this advice
applies better to attitude questions than to ones that ask about behaviour.
They argue that, when the focus is on behaviour, longer questions have certain
positive features in interviews— for example, they are more likely to provide
memory cues and they facilitate recall because of the time taken to complete
the question. However, by and large, the general advice is to keep questions
short.
 Specific rules when designing questions

◼ Avoid double-barrelled questions

Double-barrelled questions are ones that in fact ask about two things. The
problem with this kind of question is that it leaves respondents unsure about
how best to respond. Take the question:
How satisfied are you with pay and conditions in your job?

The problem here is obvious: the respondent may be satisfied with one but not
the other. Not only will the respondent be unclear about how to reply, but any
answer that is given is unlikely to be a good reflection of the level of
satisfaction with pay and conditions.
 Specific rules when designing questions

◼ Avoid very general questions

It is easy to ask a very general question when in fact what is wanted is a response
to a specific issue. The problem with questions that are very general is that
they lack a frame of reference. Thus,

How satisfied are you with your job?

seems harmless but it lacks specificity. Does it refer to pay, conditions, the nature
of the work, or all of these? If there is the possibility of such diverse
interpretations, respondents are likely to vary in their interpretations too, and
this will be a source of error.
 Specific rules when designing questions

◼ Avoid leading questions

Leading or loaded questions are ones that appear to lead the respondent in a
particular direction. Questions of the kind ‘Do you agree with the view that . .
. ?’ fall into this class of question. The obvious problem with such a question
is that it is suggesting a particular reply to respondents, although invariably
they do have the ability to rebut any implied answer. However, it is the fact
that they might feel pushed in a certain direction that is undesirable. Such a
question as:

Would you agree to cutting taxes further even though welfare provision for the
most needy sections of the population might be reduced?
 Specific rules when designing questions

◼ Avoid questions that include negatives

The problem with questions with ‘not’ or similar formulations in them is that it is easy for
the respondent to miss the word out when completing a self-completion questionnaire
or to miss it when being interviewed. If this occurs, a respondent is likely to answer in
the opposite way from the one intended. There are occasions when it is impossible to
avoid negatives, but a question like the following should be avoided as far as possible:
Do you agree with the view that students should not have to take out loans to finance
higher education?
Instead, the question should be asked in a positive format. Questions with double
negatives should be totally avoided, because it is difficult to know how to respond to
them. Oppenheim (1966) gives the following as an example of this kind of question:
Would you rather not use a non-medicated shampoo?
It is quite difficult to establish what an answer of ‘yes’ or ‘no’ would actually mean in
response to this question.
 Specific rules when designing questions

◼ Avoid technical terms

Use simple, plain language and avoid jargon. Do not ask a question like:
Do you sometimes feel alienated from work?

The problem here is that many respondents will not know what is meant by
‘alienated’, and furthermore are likely to have different views of what it means,
even if it is a remotely meaningful term to them.

◼ Does the respondent have the requisite knowledge?

There is little point in asking respondents lots of questions about matters of
which they have no knowledge. It is very doubtful whether meaningful data
about computer use could be extracted from respondents who have never
used or come into direct contact with one.
 Specific rules when designing questions

◼ Make sure that there is a symmetry between a closed question and its answers
A common mistake is for a question and its answers to be out of phase with each other.
Make sure that the answers provided for a closed question are balanced
A fairly common error when asking closed questions is for the answers that are provided to
be unbalanced. For example, imagine that a respondent is given a series of options such
as:
◼ Excellent ____
◼ Good ____
◼ Acceptable ____
◼ Poor ____
In this case, the response choices are balanced towards a favourable response. Excellent
and Good are both positive; Acceptable is a neutral or middle position; and Poor is a
negative response. In other words, the answers are loaded in favour of a positive rather
than a negative reply, so that a further negative response choice
 Specific rules when designing questions

◼ Memory problems

Do not rely too much on stretching people’s memories to the extent

that the answers for many of them are likely to be inaccurate. It
would be nice to have accurate replies to a question about the
number of times respondents have visited the cinema in the
previous twelve months, but it is highly unlikely that most will in
fact recall events accurately over such a long space of time (other
perhaps than those who have not gone at all or who have gone
only once or twice in the preceding twelve months). It was for
this reason that, in the similar question referred to above, the
time frame was predominantly just one month.
 Specific rules when designing questions

◼ Forced-choice rather than tick all that apply

Sometimes, when asking a question that allows the respondent to select more
than one answer, there is an instruction that says something like ‘Please tick all
that apply’. An example might be a question that asks which of a list of source
of regular exercise the respondent has engaged in during the previous six
months. The question might look something like this:
Which of the following sources of exercise have you engaged in during the
last six months? (Please tick all that apply)
◼ Going to a gym

◼ Sport
◼ Cycling on the road
◼ Jogging
◼ Long walks
◼ Other (please specify)
 Piloting and pre-testing questions

◼ If the main study is going to employ mainly closed questions,

open questions can be asked in the pilot to generate the fixed-
choice answers
◼ Piloting an interview schedule can provide interviewers with
some experience of using it and can infuse them with a greater
sense of confidence.
◼ If everyone (or virtually everyone) who answers a question replies
in the same way, the resulting data are unlikely to be of interest
because they do not form a variable. A pilot study allows such a
question to be identified.
◼ In interview surveys, it may be possible to identify questions that
make respondents feel uncomfortable and to detect any tendency
for respondents’ interest to be lost at certain junctures.
 Piloting and pre-testing questions

◼ Questions that seem not to be understood (more likely to be

realized in an interview than in a self-completion questionnaire
context) or questions that are often not answered should become
apparent.

◼ Pilot studies allow the researcher to determine the adequacy of

instructions to interviewers, or to respondents completing a self-
completion questionnaire.

◼ It may be possible to consider how well the questions flow and

whether it is necessary to move some of them around to improve
this feature