15 Questionnaires

Introduction O administering questionnaires

O self-administered questionnaires
The field of questionnaire design is vast. This
O postal questionnaires
chapter provides a straightforward introduction
O processing questionnaire data.
to its key elements, indicating the main issues
to be addressed, some important problematical It is suggested that researchers may find it useful
considerations and how they can be resolved. to work through these issues in sequence, though,
The chapter follows a sequence in designing a clearly, a degree of recursion is desirable.
questionnaire that, it is hoped, will be useful for The questionnaire is a widely used and useful
researchers. The serial order is: instrument for collecting survey information,
providing structured, often numerical data,
O ethical issues being able to be administered without the
O approaching the planning of a questionnaire presence of the researcher, and often being
O operationalizing the questionnaire comparatively straightforward to analyse (Wilson
O structured, semi-structured and unstruc- and McLean 1994). These attractions have to be
tured questionnaires counterbalanced by the time taken to develop,
O types of questionnaire items pilot and refine the questionnaire, by the possible
O closed and open questions compared unsophistication and limited scope of the data
O scales of data that are collected, and from the likely limited
O the dangers of assuming knowledge or flexibility of response (though, as Wilson and
viewpoints McLean (1994: 3) observe, this can frequently
O dichotomous questions be an attraction). The researcher will have to
O multiple choice questions judge the appropriateness of using a questionnaire
O rank ordering for data collection, and, if so, what kind of
O rating scales questionnaire it should be.
O constant sum questions
O ratio data questions
Ethical issues
O open-ended questions
O matrix questions The questionnaire will always be an intrusion into
O contingency questions, filters and branches the life of the respondent, be it in terms of time
O asking sensitive questions taken to complete the instrument, the level of
O avoiding pitfalls in question writing threat or sensitivity of the questions, or the possible
O sequencing the questions invasion of privacy. Questionnaire respondents are
O questionnaires containing few verbal items not passive data providers for researchers; they are
O the layout of the questionnaire subjects not objects of research. There are several
O covering letters or sheets and follow-up letters sequiturs that flow from this.
O piloting the questionnaire Respondents cannot be coerced into completing
O practical considerations in questionnaire a questionnaire. They might be strongly encour-
design aged, but the decision whether to become involved
318 QUESTIONNAIRES

and when to withdraw from the research is entirely of responses that their questions are likely to elicit.
theirs. Their involvement in the research is likely In Box 15.1 we illustrate a flow chart employed in a
to be a function of the following factors: commercial survey based upon an interview sched-
ule, although the application of the method to a
O Their informed consent (see Chapter 2 on the
self-completion questionnaire is self-evident.
ethics of educational research).
On a more positive note, Sellitz and her as-
O Their rights to withdraw at any stage or not to
sociates (1976) have provided a fairly exhaustive
complete particular items in the questionnaire.
guide to researchers in constructing their question-
O The potential of the research to improve their
naires which we summarize in Box 15.2 (see http://
situation (the issue of beneficence).
[Link]/textbooks/9780415368780 –
O The guarantees that the research will not harm
Chapter 15, file 15.1. ppt).
them (the issue of non-maleficence).
These are introductory issues, and the remainder
O The guarantees of confidentiality, anonymity and
of this chapter takes each of these and unpacks
non-traceability in the research.
them in greater detail. Additionally, one can set
O The degree of threat or sensitivity of the
out a staged sequence for planning a questionnaire,
questions, which may lead to respondents’
thus:
over-reporting or under-reporting (Sudman
and Bradburn 1982: 32 and Chapter 3). 1 Decide the purposes/objectives of the
O Factors in the questionnaire itself (e.g. its cov- questionnaire.
erage of issues, its ability to catch what respon- 2 Decide the population and the sample (as
dents want to say rather than to promote the questions about their characteristics will need
researcher’s agenda), i.e. the avoidance of bias to be included on the questionnaire under
and the assurance of validity and reliability in ‘personal details’).
the questionnaire – the issues of methodologi- 3 Generate the topics/constructs/concepts/
cal rigour and fairness. Methodological rigour issues to be addressed and data required
is an ethical not simply a technical matter in order to meet the objectives of the re-
(Morrison 1996c), and respondents have a search (this can be done from literature, or
right to expect reliability and validity. a pre-pilot, for example, focus groups and
O The reactions of the respondent, for example, semi-structured interviews).
respondents will react if they consider an 4 Decide the kinds of measures/scales/questions
item to be offensive, intrusive, misleading, /responses required.
biased, misguided, irritating, inconsiderate, 5 Write the questionnaire items.
impertinent or abstruse. 6 Check that each issue from (3) has been
addressed, using several items for each issue.
These factors impact on every stage of the use of a
7 Pilot the questionnaire and refine items as a
questionnaire, to suggest that attention has to be
consequence.
given to the questionnaire itself, the approaches
8 Administer the final questionnaire.
that are made to the respondents, the explanations
that are given to the respondents, the data analysis Within these stages there are several sub-
and the data reporting. components, and this chapter addresses these.

Approaching the planning of a Operationalizing the questionnaire

questionnaire
The process of operationalizing a questionnaire
At this preliminary stage of design, it can some- is to take a general purpose or set of purposes
times be helpful to use a flow chart technique to and turn these into concrete, researchable fields
plan the sequencing of questions. In this way, re- about which actual data can be gathered. First, a
searchers are able to anticipate the type and range questionnaire’s general purposes must be clarified
 APPROACHING THE PLANNING OF A QUESTIONNAIRE 319

Box 15.1

Chapter 15
A flow chart technique for question planning

Do you have double-glazing on any window in your house?

Yes No

Did you have it Do you have any

fitted or was it plans to get it
here beforehand? installed or not?

Fitted by Fitted Yes No

present occupant beforehand
What were the
What were the reasons for you
Do you think If you were given a
reasons for you getting it installed?
you would have grant to complete the
getting it installed?
moved in here if it work, would that make
was not installed? any difference or not?

What are its advantages? What do you think are its

What are its disadvantages? advantages? And its
etc. disadvantages? etc.

Source: Social and Community Planning Research 1972

and then translated into a specific, concrete aim or respect to the type of courses required, detailed
set of aims. Thus, ‘to explore teachers’ views about information would be needed about the duration of
in-service work’ is somewhat nebulous, whereas courses (one meeting, several meetings, a week, a
‘to obtain a detailed description of primary and month, a term or a year), the status of courses (non-
secondary teachers’ priorities in the provision award bearing, award bearing, with certificate,
of in-service education courses’ is reasonably diploma, degree granted by college or university),
specific. the orientation of courses (theoretically oriented
Having decided upon and specified the primary involving lectures, readings, etc., or practically
objective of the questionnaire, the second phase oriented involving workshops and the production
of the planning involves the identification and of curriculum materials).
itemizing of subsidiary topics that relate to its What we have in the example, then, is a move
central purpose. In our example, subsidiary issues from a generalized area of interest or purpose to a
might well include the types of courses required, very specific set of features about which direct data
the content of courses, the location of courses, the can be gathered. Wilson and McLean (1994: 8–9)
timing of courses, the design of courses, and the suggest an alternative approach which is to identify
financing of courses. the research problem, then to clarify the relevant
The third phase follows the identification and concepts or constructs, then to identify what kinds
itemization of subsidiary topics and involves of measures (if appropriate) or empirical indicators
formulating specific information requirements there are of these, i.e. the kinds of data required
relating to each of these issues. For example, with to give the researcher relevant evidence about the
320 QUESTIONNAIRES

Box 15.2
A guide for questionnaire construction

A Decisions about question content

1 Is the question necessary? Just how will it be useful?
2 Are several questions needed on the subject matter of this question?
3 Do respondents have the information necessary to answer the question?
4 Does the question need to be more concrete, specific and closely related to the respondent’s personal experience?
5 Is the question content sufficiently general and free from spurious concreteness and specificity?
6 Do the replies express general attitudes and only seem to be as specific as they sound?
7 Is the question content biased or loaded in one direction, without accompanying questions to balance the emphasis?
8 Will the respondents give the information that is asked for?
B Decisions about question wording
1 Can the question be misunderstood? Does it contain difficult or unclear phraseology?
2 Does the question adequately express the alternative with respect to the point?
3 Is the question misleading because of unstated assumptions or unseen implications?
4 Is the wording biased? Is it emotionally loaded or slanted towards a particular kind of answer?
5 Is the question wording likely to be objectionable to the respondent in any way?
6 Would a more personalized wording of the question produce better results?
7 Can the question be better asked in a more direct or a more indirect form?
C Decisions about form of response to the question
1 Can the question best be asked in a form calling for check answer (or short answer of a word or two, or a number),
free answer or check answer with follow-up answer?
2 If a check answer is used, which is the best type for this question – dichotomous, multiple-choice (‘cafeteria’
question), or scale?
3 If a checklist is used, does it cover adequately all the significant alternatives without overlapping and in a defensible
order? Is it of reasonable length? Is the wording of items impartial and balanced?
4 Is the form of response easy, definite, uniform and adequate for the purpose?
D Decisions about the place of the question in the sequence
1 Is the answer to the question likely to be influenced by the content of preceding questions?
2 Is the question led up to in a natural way? Is it in correct psychological order?
3 Does the question come too early or too late from the point of view of arousing interest and receiving sufficient
attention, avoiding resistance, and so on?

Source: Sellitz et al. 1976

concepts or constructs, e.g. their presence, their in- O elicits the most appropriate kinds of data to
tensity, their main features and dimensions, their answer the research purposes and sub-questions
key elements etc. O asks for empirical data.
What unites these two approaches is their
recognition of the need to ensure that the Structured, semi-structured and
questionnaire: unstructured questionnaires
Although there is a large range of types of
O is clear on its purposes questionnaire, there is a simple rule of thumb: the
O is clear on what needs to be included or cov- larger the size of the sample, the more structured,
ered in the questionnaire in order to meet the closed and numerical the questionnaire may have
purposes to be, and the smaller the size of the sample, the
O is exhaustive in its coverage of the elements of less structured, more open and word-based the
inclusion questionnaire may be.
O asks the most appropriate kinds of question The researcher can select several types of ques-
(discussed below) tionnaire, from highly structured to unstructured.
 TYPES OF QUESTIONNAIRE ITEMS 321

If a closed and structured questionnaire is used, the respondent may choose. Highly structured,

Chapter 15
enabling patterns to be observed and comparisons closed questions are useful in that they can
to be made, then the questionnaire will need to generate frequencies of response amenable to
be piloted and refined so that the final version statistical treatment and analysis. They also enable
contains as full a range of possible responses as comparisons to be made across groups in the sample
can be reasonably foreseen. Such a questionnaire (Oppenheim 1992: 115). They are quicker to code
is heavy on time early in the research; however, up and analyse than word-based data (Bailey 1994:
once the questionnaire has been set up, then the 118) and, often, they are directly to the point
mode of analysis might be comparatively rapid. and deliberately more focused than open-ended
For example, it may take two or three months to questions. Indeed it would be almost impossible, as
devise a survey questionnaire, pilot it, refine it and well as unnecessary, to try to process vast quantities
set it out in a format that will enable the data to be of word-based data in a short time frame.
processed and statistics to be calculated. However, If a site-specific case study is required, then
the trade-off from this is that the data analysis can qualitative, less structured, word-based and open-
be undertaken fairly rapidly. We already know the ended questionnaires may be more appropriate as
response categories, the nature of the data and the they can capture the specificity of a particular
statistics to be used; it is simply a matter of pro- situation. Where measurement is sought then
cessing the data – often using computer analysis. a quantitative approach is required; where rich
It is perhaps misleading to describe a and personal data are sought, then a word-based
questionnaire as being ‘unstructured’, as the whole qualitative approach might be more suitable.
devising of a questionnaire requires respondents Open-ended questions are useful if the possible
to adhere to some form of given structure. That answers are unknown or the questionnaire is
said, between a completely open questionnaire exploratory (Bailey 1994: 120), or if there are so
that is akin to an open invitation to ‘write what many possible categories of response that a closed
one wants’ and a completely closed, completely question would contain an extremely long list of
structured questionnaire, there is the powerful tool options. They also enable respondents to answer
of the semi-structured questionnaire. Here a series as much as they wish, and are particularly suitable
of questions, statements or items are presented for investigating complex issues, to which simple
and the respondents are asked to answer, respond answers cannot be provided. Open questions
to or comment on them in a way that they may be useful for generating items that will
think best. There is a clear structure, sequence subsequently become the stuff of closed questions
and focus, but the format is open-ended, enabling in a subsequent questionnaire (i.e. a pre-pilot).
respondents to reply in their own terms. The semi- In general closed questions (dichotomous,
structured questionnaire sets the agenda but does multiple choice, constant sum and rating scales)
not presuppose the nature of the response. are quick to complete and straightforward to
code (e.g. for computer analysis), and do not
discriminate unduly on the basis of how articulate
Types of questionnaire items respondents are (Wilson and McLean 1994: 21).
On the other hand, they do not enable respondents
Closed and open questions compared
to add any remarks, qualifications and explanations
There are several kinds of question and response to the categories, and there is a risk that the
modes in questionnaires, including, for example, categories might not be exhaustive and that there
dichotomous questions, multiple choice questions, might be bias in them (Oppenheim 1992: 115).
rating scales, constant sum questions, ratio data Open questions enable participants to write a
and open-ended questions. These are considered free account in their own terms, to explain and
below (see also Wilson 1996). Closed questions qualify their responses and avoid the limitations of
prescribe the range of responses from which pre-set categories of response. On the other hand,
322 QUESTIONNAIRES

open questions can lead to irrelevant and redun- matters in which researchers are interested. This is
dant information; they may be too open-ended a dangerous assumption. It is particularly a problem
for the respondent to know what kind of infor- when administering questionnaires to children,
mation is being sought; they may require much who may write anything rather than nothing. This
more time from the respondent to enter a response means that the opportunity should be provided
(thereby leading to refusal to complete the item), for respondents to indicate that they have no
and they may make the questionnaire appear opinion, or that they don’t know the answer to a
long and discouraging. With regard to analysis, particular question, or to state that they feel the
the data are not easily compared across partic- question does not apply to them. This is frequently
ipants, and the responses are difficult to code a matter in surveys of customer satisfaction in
and to classify (see [Link] social science, where respondents are asked, for
textbooks/9780415368780 – Chapter 15, file example, to answer a host of questions about the
15.2. ppt). services provided by utility companies (electricity,
We consider in more detail below the different gas, water, telephone) about which they have
kinds of closed and open questions. no strong feelings, and, in fact, they are only
interested in whether the service is uninterrupted,
Scales of data reliable, cheap, easy to pay for, and that their
complaints are solved.
The questionnaire designer will need to choose There is also the issue of choice of vocabulary
the metric – the scale of data – to be adopted. and the concepts and information behind them.
This concerns numerical data, and we advise It is essential that, regardless of the type of
readers to turn to Part Five for an analysis of question asked, the language and the concepts
the different scales of data that can be gathered behind the language should be within the grasp of
(nominal, ordinal, interval and ratio) and the the respondents. Simply because the researcher is
different statistics that can be used for analysis. interested in, and has a background in, a particular
Nominal data indicate categories; ordinal data topic is no guarantee that the respondents will be
indicate order (‘high’ to ‘low’, ‘first’ to ‘last’, like minded. The effect of the questionnaire on
‘smallest’ to ‘largest’, ‘strongly disagree’ to ‘strongly the respondent has to be considered carefully.
agree’, ‘not at all’ to ‘a very great deal’); ratio
data indicate continuous values and a true zero
(e.g. marks in a test, number of attendances per Dichotomous questions
year) (see [Link] A highly structured questionnaire will ask closed
9780415368780 – Chapter 15, file 15.3. ppt). questions. These can take several forms. Dichoto-
These are presented thus: mous questions require a ‘yes’/‘no’ response, e.g.
Question type Level of data ‘Have you ever had to appear in court?’, ‘Do you
Dichotomous questions Nominal prefer didactic methods to child-centred meth-
Multiple choice questions Nominal ods?’ (see [Link]
Rank ordering Ordinal 9780415368780 – Chapter 15, file 15.4. ppt). The
Rating scales Ordinal layout of a dichotomous question can be thus:
Constant sum questions Ordinal
Sex(please tick) : Male
Female

Ratio data questions Ratio
Open-ended questions Word-based data The dichotomous question is useful, for it compels
respondents to come off the fence on an issue. It
The dangers of assuming knowledge or
provides a clear, unequivocal response. Further, it
viewpoints
is possible to code responses quickly, there being
There is often an assumption that respondents will only two categories of response. A dichotomous
have the information or have an opinion about the question is also useful as a funnelling or sorting
 TYPES OF QUESTIONNAIRE ITEMS 323

device for subsequent questions, for example: ‘If designed to capture the likely range of responses

Chapter 15
you answered ‘‘yes’’ to question X, please go to to given statements (see [Link]
question Y; if you answered ‘‘no’’ to question X, com/textbooks/9780415368780 – Chapter 15, file
please go to question Z’ (see the section below 15.5. ppt). For example, the researcher might
on contingency questions). Sudman and Bradburn ask a series of questions about a new chemistry
(1982: 89) suggest that if dichotomous questions scheme in the school; a statement precedes a set
are being used, then it is desirable to use several of responses thus:
to gain data on the same topic, in order to reduce
The New Intermediate Chemistry Education (NICE)
the problems of respondents’ ‘guessing’ answers.
is:
On the other hand, the researcher must ask, for
instance, whether a ‘yes’/‘no’ response actually (a) a waste of time
provides any useful information. Requiring (b) an extra burden on teachers
respondents to make a ‘yes’/‘no’ decision may be (c) not appropriate to our school
inappropriate; it might be more appropriate to (d) a useful complementary scheme
have a range of responses, for example in a rating (e) a useful core scheme throughout the school
scale. There may be comparatively few complex (f) well-presented and practicable.
or subtle questions which can be answered with
The categories would have to be discrete (i.e.
a simple ‘yes’ or ‘no’. A ‘yes’ or a ‘no’ may be
having no overlap and being mutually exclusive)
inappropriate for a situation whose complexity is
and would have to exhaust the possible range of
better served by a series of questions which catch
responses. Guidance would have to be given on
that complexity. Further, Youngman (1984: 163)
the completion of the multiple-choice, clarifying,
suggests that it is a natural human tendency to
for example, whether respondents are able to tick
agree with a statement rather than to disagree
only one response (a single answer mode) or several
with it; this suggests that a simple dichotomous
responses (multiple answer mode) from the list. Like
question might build in respondent bias. Indeed
dichotomous questions, multiple choice questions
people may be more reluctant to agree with a
can be quickly coded and quickly aggregated to
negative statement than to disagree with a positive
give frequencies of response. If that is appropriate
question (Weems et al. 2003).
for the research, then this might be a useful
In addition to dichotomous questions (‘yes’/‘no’
instrument.
questions) a piece of research might ask
The layout of a multiple choice question can be
for information about dichotomous variables,
thus:
for example gender (male/female), type of
school (elementary/secondary), type of course Number of years in teaching 1–5
6–14

(vocational/non-vocational). In these cases only 15–24
25+

one of two responses can be selected. This enables Which age group do you teach at present (you
nominal data to be gathered, which can then may tick more than one)?
be processed using the chi-square statistic, the
Infant

binomial test, the G-test and cross-tabulations
Primary

(see Cohen and Holliday (1996) for examples).
Secondary (excluding sixth form)

Dichotomous questions are treated as nominal
Sixth form only

data (see Part Five).
Just as dichotomous questions have their parallel
in dichotomous variables, so multiple choice
Multiple choice questions
questions have their parallel in multiple elements
To try to gain some purchase on complexity, of a variable. For example, the researcher may be
the researcher can move towards multiple choice asking to which form a student belongs – there
questions, where the range of choices is being up to, say, forty forms in a large school, or
324 QUESTIONNAIRES

the researcher may be asking which post-16 course might be very useful statistically in making a deci-
a student is following (e.g. academic, vocational, sive judgement about a programme). In this case
manual, non-manual). In these cases only one this rough and ready statistic might be perfectly
response may be selected. As with the dichotomous acceptable.
variable, the listing of several categories or One can see in the example of chemistry
elements of a variable (e.g. form membership above not only ambiguity in the wording but
and course followed) enables nominal data to also a very incomplete set of response categories
be collected and processed using the chi-square which is hardly capable of representing all aspects
statistic, the G-test and cross-tabulations (Cohen of the chemistry scheme. That this might be
and Holliday 1996). Multiple choice questions are politically expedient cannot be overlooked, for
treated as nominal data (see Part Five). if the choice of responses is limited, then those
It may be important to include in the multiple responses might enable bias to be built into the
choices those that will enable respondents to research. For example, if the responses were limited
select the response that most closely represents to statements about the utility of the chemistry
their view, hence a pilot is needed to ensure that scheme, then the evaluator would have little
the categories are comprehensive, exhaustive and difficulty in establishing that the scheme was
representative. On the other hand, the researcher useful. By avoiding the inclusion of negative
may be interested in certain features only, and it is statements or the opportunity to record a negative
these that would figure in the response categories response the research will surely be biased. The
only. issue of the wording of questions has been discussed
The multiple choice questionnaire seldom gives earlier.
more than a crude statistic, for words are inherently Multiple choice items are also prone to
ambiguous. In the example above of chemistry, the problems of word order and statement order. For
notion of ‘useful’ is unclear, as are ‘appropriate’, example, Dillman et al. (2003: 6) report a study of
‘practicable’ and ‘burden’. Respondents could German students who were asked to compare their
interpret these words differently in their own high school teachers in terms of whether male
contexts, thereby rendering the data ambiguous. or female teachers were more empathetic. They
One respondent might see the utility of the found that respondents rated their female teachers
chemistry scheme in one area and thereby say more highly than their male teachers when asked
that it is useful – ticking (d). Another respondent to compare female teachers to male teachers than
might see the same utility in that same one area when they were asked to compare their male
but, because it is only useful in that single area, teachers to their female teachers. Similarly they
may see this as a flaw and therefore not tick report a study in which tennis was found to be less
category (d). With an anonymous questionnaire exciting than football when the tennis option was
this difference would be impossible to detect. presented before the football option, and more
This is the heart of the problem of question- exciting when the football option was placed
naires – that different respondents interpret the before the tennis option. These studies suggest
same words differently. ‘Anchor statements’ can that respondents tend to judge later items in terms
be provided to allow a degree of discrimination of the earlier items, rather than vice versa and
in response (e.g. ‘strongly agree’, ‘agree’ etc.) but that they overlook features specific to later items
there is no guarantee that respondents will always if these are not contained in the earlier items.
interpret them in the way that is intended. In This is an instance of the ‘primacy effect’ or ‘order
the example above this might not be a problem effect’, wherein items earlier in a list are given
as the researcher might only be seeking an index greater weight than items lower in the list. Order
of utility – without wishing to know the areas of effects are resilient to efforts to minimize them, and
utility or the reasons for that utility. The evaluator primacy effects are particularly strong in Internet
might be wishing only for a crude statistic (which questionnaires (Dillman et al. 2003: 22).
 TYPES OF QUESTIONNAIRE ITEMS 325

ask respondents to arrange priorities where there

Chapter 15
Rank ordering
are more than five ranks that have been requested.
The rank order question is akin to the multiple
In the case of the list of ten points above, the
choice question in that it identifies options from
researcher might approach this problem in one of
which respondents can choose, yet it moves
two ways. The list in the questionnaire item can be
beyond multiple choice items in that it asks
reduced to five items only, in which case the range
respondents to identify priorities. This enables
and comprehensiveness of responses that fairly
a relative degree of preference, priority, intensity
catches what the respondent feels is significantly
etc. to be charted (see [Link]
reduced. Alternatively, the list of ten items can
com/textbooks/9780415368780 – Chapter 15, file
be retained, but the request can be made to the
15.6. ppt). In the rank ordering exercise a list of
respondents only to rank their first five priorities,
factors is set out and the respondent is required to
in which case the range is retained and the task is
place them in a rank order, for example:
not overwhelming (though the problem of sorting
Please indicate your priorities by placing numbers the data for analysis is increased).
in the boxes to indicate the ordering of your views, An example of a shorter list might be:
1 = the highest priority, 2 = the second highest, and
Please place these in rank order of the most to
so on.
the least important, by putting the position (1–5)
The proposed amendments to the mathematics
against each of the following statements, number 1
scheme might be successful if the following factors
being the most important and number 5 being the
are addressed:
least important:
O the appropriate material resources are in
Students should enjoy school []
school

Teachers should set less homework []
O the amendments are made clear to all teachers

Students should have more choice of subjects
O the amendments are supported by the
in school []
mathematics team

Teachers should use more collaborative methods [ ]
O the necessary staff development is assured

Students should be tested more, so that they
O there are subsequent improvements to
work harder []
student achievement

O the proposals have the agreement of all Rankings are useful in indicating degrees of
teachers
response. In this respect they are like rating scales,
O they improve student motivation
discussed below. Ranking questions are treated
O parents approve of the amendments
as ordinal data (see Part Five for a discussion of
O they will raise the achievements of the ordinal data).
brighter students

O the work becomes more geared to
problem-solving
Rating scales
In this example ten items are listed. While this One way in which degrees of response, intensity
might be enticing for the researcher, enabling fine of response, and the move away from dichoto-
distinctions possibly to be made in priorities, it mous questions have been managed can be seen in
might be asking too much of the respondents to the notion of rating scales – Likert scales, semantic
make such distinctions. They genuinely might not differential scales, Thurstone scales and Guttman
be able to differentiate their responses, or they scaling. These are very useful devices for the re-
simply might not feel strongly enough to make searcher, as they build in a degree of sensitivity
such distinctions. The inclusion of too long a and differentiation of response while still gener-
list might be overwhelming. Indeed Wilson and ating numbers. This chapter will focus on the
McLean (1994: 26) suggest that it is unrealistic to first two of these, though readers will find the
326 QUESTIONNAIRES

others discussed in Oppenheim (1992) (see http:// the scale should be measuring only one thing at a
[Link]/textbooks/9780415368780 – time (Oppenheim 1992: 187–8). Indeed this is a
Chapter 15, file 15.7. ppt). A Likert scale (named cornerstone of Likert’s (1932) own thinking.
after its deviser, Rensis Likert 1932) provides a It is a very straightforward matter to convert
range of responses to a given question or statement, a dichotomous question into a multiple choice
for example: question. For example, instead of asking the ‘do
you?’, ‘have you?’, ‘are you?’, ‘can you?’ type
How important do you consider work placements to
questions in a dichotomous format, a simple
be for secondary school students?
addition to wording will convert it into a much
1 = not at all more subtle rating scale, by substituting the words
2 = very little ‘to what extent?’, ‘how far?’, ‘how much?’, ‘how
3 = a little often?’ etc.
4 = quite a lot A semantic differential is a variation of a rating
5 = a very great deal scale which operates by putting an adjective at
one end of a scale and its opposite at the other, for
All students should have access to free higher
example:
education.
How informative do you consider the new set of
1 = strongly disagree
2 = disagree history textbooks to be?
3 = neither agree nor disagree
4 = agree 1 2 3 4 5 6 7
5 = strongly agree Useful − − − − − − − Useless
Such a scale could be set out thus: Respondents indicate their opinion by circling
Please complete the following by placing a tick in or putting a mark on that position on the scale
one space only, as follows: which most represents what they feel. Researchers
devise their own terms and their polar opposites,
1 = strongly disagree; 2 = disagree;
for example:
3 = neither agree nor disagree; Approachable .... Unapproachable
4 = agree; 5 = strongly agree Generous .... Mean
Friendly .... Hostile
1 2 3 4 5
Caring .... Uncaring
Senior school staff [ ] [ ] [ ] [ ] [ ]
Attentive .... Inattentive
should teach more
Hard-working .... Lazy
In these examples the categories need to be
Osgood et al. (1957), the pioneers of this
discrete and to exhaust the range of possible
technique, suggest that semantic differential
responses which respondents may wish to give.
scales are useful in three contexts: evaluative
Notwithstanding the problems of interpretation
(e.g. valuable-valueless, useful-useless, good-bad);
which arise as in the previous example – one
potency (e.g. large-small, weak-strong, light-
respondent’s ‘agree’ may be another’s ‘strongly
heavy); and activity (e.g. quick-slow; active-
agree’, one respondent’s ‘very little’ might be
passive, dynamic-lethargic).
another’s ‘a little’ – the greater subtlety of response
There are several commonly used categories in
which is built into a rating scale renders this a very
rating scales, for example:
attractive and widely used instrument in research.
These two examples both indicate an important Strongly disagree/disagree/neither agree nor disagree/
feature of an attitude scaling instrument, namely agree/strongly agree
the assumption of unidimensionality in the scale; Very seldom/occasionally/quite often/very often
 TYPES OF QUESTIONNAIRE ITEMS 327

Very little/a little/somewhat/a lot/a very great deal a pilot and also to include a category entitled

Chapter 15
Never/almost never/sometimes/often/very often ‘other (please state)’.
Not at all important/unimportant/neither important O Most of us would not wish to be called extrem-
nor unimportant/important/very important ists; we often prefer to appear like each other
Very true of me/a little bit true of me/don’t know/not in many respects. For rating scales this means
really true of me/very untrue of me that we might wish to avoid the two extreme
Strongly agree/agree/uncertain/disagree/strongly poles at each end of the continuum of the
agree rating scales, reducing the number of positions
in the scales to a choice of three (in a 5-point
To these could be added the category ‘don’t know’
scale). That means that in fact there could be
or ‘have no opinion’. Rating scales are widely used
very little choice for us. The way round this
in research, and rightly so, for they combine the
is to create a larger scale than a 5-point scale,
opportunity for a flexible response with the ability
for example a 7-point scale. To go beyond a
to determine frequencies, correlations and other
7-point scale is to invite a degree of detail and
forms of quantitative analysis. They afford the
precision which might be inappropriate for the
researcher the freedom to fuse measurement with
item in question, particularly if the argument
opinion, quantity and quality.
set out above is accepted, that one respondent’s
Though rating scales are powerful and useful
scale point 3 might be another’s scale point 4.
in research, the investigator, nevertheless, needs
O There is a tendency for participants to opt for
to be aware of their limitations. For example,
the mid-point of a 5-point or 7-point scale (the
the researcher may infer a degree of sensitivity and
central tendency). This is notably an issue in
subtlety from the data that they cannot bear. There
East Asian respondents, where the ‘doctrine of
are other cautionary factors about rating scales, be
the mean’ is advocated in Confucian culture.
they Likert scales or semantic differential scales:
One option to overcome this is to use an even
O There is no assumption of equal intervals number scaling system, as there is no mid-
between the categories, hence a rating of 4 point. On the other hand, it could be argued
indicates neither that it is twice as powerful that if respondents wish to sit on the fence and
as 2 nor that it is twice as strongly felt; one choose a mid-point, then they should be given
cannot infer that the intensity of feeling in the option to do so.
the Likert scale between ‘strongly agree’ and O On the scales so far there have been mid-
‘disagree’ somehow matches the intensity of points; on the 5-point scale it is category 3,
feeling between ‘strongly disagree’ and ‘agree’. and on the 7-point scale it is category 4. The
These are illegitimate inferences. The problem use of an odd number of points on a scale
of equal intervals has been addressed in enables this to occur. However, choosing an
Thurstone scales (Thurstone and Chave 1929; even number of scale points, for example a
Oppenheim 1992: 190–5). 6-point scale, might require a decision on rating
O We have no check on whether respondents to be indicated.
are telling the truth. Some may be deliberately
falsifying their replies. For example, suppose a new staffing structure has
O We have no way of knowing if the respondent been introduced into a school and the headteacher
might have wished to add any other comments is seeking some guidance on its effectiveness. A
about the issue under investigation. It might 6-point rating scale might ask respondents to
have been the case that there was something indicate their response to the statement:
far more pressing about the issue than the rat-
ing scale included but which was condemned The new staffing structure in the school has enabled
to silence for want of a category. A straight- teamwork to be managed within a clear model of line
forward way to circumvent this issue is to run management.
328 QUESTIONNAIRES

(Circle one number) are treated as ordinal data (see Part Five), using
modal scores and non-parametric data analysis,
1 2 3 4 5 6
though one can find very many examples where
Strongly − − − − − − Strongly
this rule has been violated, and non-parametric
agree disagree
data have been treated as parametric data. This is
Let us say that one member of staff circled 1, unacceptable.
eight staff circled 2, twelve staff circled 3, nine It has been suggested that the attraction of
staff circled 4, two staff circled 5, and seven rating scales is that they provide more opportunity
staff circled 6. There being no mid-point on this than dichotomous questions for rendering data
continuum, the researcher could infer that those more sensitive and responsive to respondents. This
respondents who circled 1, 2 or 3 were in some makes rating scales particularly useful for tapping
measure of agreement, while those respondents attitudes, perceptions and opinions. The need for a
who circled 4, 5 or 6 were in some measure of pilot study to devise and refine categories, making
disagreement. That would be very useful for, say, them exhaustive and discrete, has been suggested
a headteacher, in publicly displaying agreement, as a necessary part of this type of data collection.
there being twenty-one staff (1 + 8 + 12) agreeing Questionnaires that are going to yield numerical
with the statement and eighteen (9 + 2 + 7) or word-based data can be analysed using computer
displaying a measure of disagreement. However, programmes (for example SPSS or Ethnograph
one could point out that the measure of ‘strongly respectively). If the researcher intends to process
disagree’ attracted seven staff – a very strong the data using a computer package it is essential
feeling – which was not true for the ‘strongly agree’ that the layout and coding system of the
category, which attracted only one member of staff. questionnaire are appropriate for that particular
The extremity of the voting has been lost in a crude computer package. Instructions for layout in order
aggregation. to facilitate data entry are contained in manuals
Further, if the researcher were to aggregate the that accompany such packages.
scoring around the two mid-point categories (3 Rating scales are more sensitive instruments
and 4) there would be twenty-one members of staff than dichotomous scales. Nevertheless, they are
represented, leaving nine (1 + 8) from categories limited in their usefulness to researchers by their
1 and 2 and nine (2 + 7) from categories 5 and fixity of response caused by the need to select from
6; adding together categories 1, 2, 5 and 6, a a given choice. A questionnaire might be tailored
total of eighteen is reached, which is less than the even more to respondents by including open-ended
twenty-one total of the two categories 3 and 4. It questions to which they can reply in their own
seems on this scenario that it is far from clear that terms and own opinions. We consider these later.
there was agreement with the statement from the
staff; indeed taking the high incidence of ‘strongly
disagree’, it could be argued that those staff who Constant sum questions
were perhaps ambivalent (categories 3 and 4),
In this type of question respondents are asked to
coupled with those who registered a ‘strongly
distribute a given number of marks (points) be-
disagree’, indicate not agreement but disagreement
tween a range of items (see [Link]
with the statement.
com/textbooks/9780415368780 – Chapter 15, file
The interpretation of data has to be handled
15.8. ppt). For example:
very carefully; ordering data to suit a researcher’s
own purposes might be very alluring but quite Please distribute a total of 10 points among the
illegitimate. The golden rule here is that crude data sentences that you think most closely describe your
can yield only crude interpretation; subtle statistics behaviour. You may distribute these freely: they may
require subtle data. The interpretation of data must be spread out, or awarded to only a few statements,
not distort the data unfairly. Rating scale questions or all allocated to a single sentence if you wish.
 TYPES OF QUESTIONNAIRE ITEMS 329

I can take advantage of new opportunities [ ] Constant sum data are ordinal, and this means

Chapter 15
I can work effectively with all kinds of people [ ] that non-parametric analysis can be performed on
Generating new ideas is one of my strengths [ ] the data (see Part Five).
I can usually tell what is likely to work in
practice [ ]
I am able to see tasks through to the very end [ ]
Ratio data questions
I am prepared to be unpopular for the good of the
school [ ] We discuss ratio data in Part Five and we
This enables priorities to be identified, comparing refer the reader to the discussion and definition
highs and lows, and for equality of choices to there (see [Link]
be indicated, and, importantly, for this to be 9780415368780 – Chapter 15, file 15.9. ppt). For
done in the respondents’ own terms. It requires our purposes here we suggest that ratio data
respondents to make comparative judgements and questions deal with continuous variables where
choices across a range of items. For example, we there is a true zero, for example:
may wish to distribute 10 points for aspects of an How much money do you have in the bank? ––
individual’s personality: How many times have you been late for school? ––
Talkative [ ] How many marks did you score in the mathematics
Cooperative [ ] test? ––
Hard-working [ ] How old are you (in years)? ––
Lazy [ ] Here no fixed answer or category is provided,
Motivated [ ] and the respondent puts in the numerical answer
Attentive [ ] that fits his/her exact figure, i.e. the accuracy is
This means that the respondent has to consider higher, much higher than in categories of data. This
the relative weight of each of the given aspects enables averages (means), standard deviations,
before coming to a decision about how to award range, and high-level statistics to be calculated,
the marks. To accomplish this means that the all- e.g. regression, factor analysis, structural equation
round nature of the person, in the terms provided, modelling (see Part Five).
has to be considered, to see, on balance, which An alternative form of ratio scaling is where the
aspect is stronger when compared to another.1 respondent has to award marks out of, say, ten, for a
The difficulty with this approach is to particular item. This is a device that has been used
decide how many marks can be distributed (a in business and commerce for measuring service
round number, for example 10, makes subsequent quality and customer satisfaction, and is being used
calculation easily comprehensible) and how many in education by Kgaile and Morrison (2006); see
statements/items to include, e.g. whether to have for example Box 15.3.
the same number of statements as there are marks, This kind of scaling is often used in telephone
or more or fewer statements than the total of marks. interviews, as it is easy for respondents to
Having too few statements/items does not do understand. The argument could be advanced that
justice to the complexity of the issue, and having this is a sophisticated form of rating scale, but the
too many statements/items may mean that it is terminology used in the instruction clearly suggests
difficult for respondents to decide how to distribute that it asks for ratio scale data.
their marks. Having too few marks available may
be unhelpful, but, by contrast, having too many
marks and too many statements/items can lead to
Open-ended questions
simple computational errors by respondents. Our
advice is to keep the number of marks to ten and The open-ended question is a very attractive
the number of statements to around six to eight. device for smaller scale research or for those
330 QUESTIONNAIRES

Box 15.3
A 10-point marking scale in a questionnaire

Please give a mark from 1 to 10 for the following statements, with 10 being excellent and 1 being very poor. Please circle
the appropriate number for each statement.
Teaching and learning
Very poor Excellent
1 The attention given to teaching and learning at 1 2 3 4 5 6 7 8 9 10
the school
2 The quality of the lesson preparation 1 2 3 4 5 6 7 8 9 10
3 How well learners are cared for, guided and 1 2 3 4 5 6 7 8 9 10
supported
4 How effectively teachers challenge and engage 1 2 3 4 5 6 7 8 9 10
learners
5 The educators’ use of assessment for maximiz- 1 2 3 4 5 6 7 8 9 10
ing learners’ learning
6 How well students apply themselves to learning 1 2 3 4 5 6 7 8 9 10
7 Discussion and review by educators of the qual- 1 2 3 4 5 6 7 8 9 10
ity of teaching and learning

sections of a questionnaire that invite an hon- honesty and candour which, as is argued elsewhere
est, personal comment from respondents in ad- in this book, are the hallmarks of qualitative data.
dition to ticking numbers and boxes (see http:// Oppenheim (1992: 56–7) suggests that a
[Link]/textbooks/9780415368780 – sentence-completion item is a useful adjunct to
Chapter 15, file 15.10. ppt). The questionnaire an open-ended question, for example:
simply puts the open-ended questions and leaves
a space (or draws lines) for a free response. It is Please complete the following sentence in your own
the open-ended responses that might contain the words:
‘gems’ of information that otherwise might not be An effective teacher. . .
caught in the questionnaire. Further, it puts the
responsibility for and ownership of the data much or
more firmly into respondents’ hands.
The main things that I find annoying with disruptive
It is useful for the researcher to provide some
students are . . .
support for respondents, so that they know the
kind of reply being sought. For example, an open Open-endedness also carries problems of data han-
question that includes a prompt could be: dling. For example, if one tries to convert opinions
into numbers (e.g. so many people indicated some
Please indicate the most important factors that reduce
degree of satisfaction with the new principal’s
staff participation in decision-making.
management plan), then it could be argued that
Please comment on the strengths and weaknesses of
the questionnaire should have used rating scales
the mathematics course.
in the first place. Further, it might well be that the
Please indicate areas for improvement in the teaching
researcher is in danger of violating one principle of
of foreign languages in the school.
word-based data, which is that they are not validly
This is not to say that the open-ended question susceptible to aggregation, i.e. that it is trying to
might well not frame the answer, just as the stem bring to word-based data the principles of numer-
of a rating scale question might frame the response ical data, borrowing from one paradigm (quanti-
given. However, an open-ended question can tative, positivist methodology) to inform another
catch the authenticity, richness, depth of response, paradigm (qualitative, interpretive methodology).
 TYPES OF QUESTIONNAIRE ITEMS 331

Further, if a genuinely open-ended question is Here five questions have been asked in only five

Chapter 15
being asked, it is perhaps unlikely that responses lines, excluding, of course, the instructions and
will bear such a degree of similarity to each explanations of the anchor statements. Such a
other so as to enable them to be aggregated too layout is economical of space.
tightly. Open-ended questions make it difficult A second example indicates how a matrix
for the researcher to make comparisons between design can save a considerable amount of space
respondents, as there may be little in common to in a questionnaire. Here the size of potential
compare. Moreover, to complete an open-ended problems in conducting a piece of research is
questionnaire takes much longer than placing a asked for, and data on how much these problems
tick in a rating scale response box; not only were soluble are requested. For the first issue
will time be a constraint here, but there is an (the size of the problem), 1 = no problem, 2 = a
assumption that respondents will be sufficiently or small problem, 3 = a moderate problem, 4 = a
equally capable of articulating their thoughts and large problem, 5 = a very large problem. For the
committing them to paper. second issue (how much the problem was solved),
In practical terms, Redline et al. (2002) report 1 = not solved at all, 2 = solved only a very little,
that using open-ended questions can lead to 3 = solved a moderate amount, 4 = solved a lot,
respondents overlooking instructions, as they are 5 = completely solved (see Box 15.4).
occupied with the more demanding task of writing Here thirty questions (15 × 2) have been able
in their own words than reading instructions. to be covered in just a short amount of space.
Despite these cautions, the space provided for an Laying out the questionnaire like this enables
open-ended response is a window of opportunity the respondent to fill in the questionnaire rapidly.
for the respondent to shed light on an issue or On the other hand, it risks creating a mind set in
course. Thus, an open-ended questionnaire has the respondent (a ‘response set’: Baker 1994: 181)
much to recommend it. in that the respondent may simply go down the
questionnaire columns and write the same number
each time (e.g. all number 3) or, in a rating
Matrix questions scale, tick all number 3. Such response sets can
Matrix questions are not types of questions but be detected by looking at patterns of replies and
concern the layout of questions. Matrix questions eliminating response sets from subsequent analysis.
enable the same kind of response to be given to The conventional way of minimizing response
several questions, for example ‘strongly disagree’ sets has been by reversing the meaning of some
to ‘strongly agree’. The matrix layout helps to save of the questions so that the respondents will need
space, for example: to read them carefully. However, Weems et al.
(2003) argue that using positively and negatively
Please complete the following by placing a tick in worded items within a scale is not measuring
one space only, as follows: the same underlying traits. They report that
1 = not at all; 2 = very little; 3 = a moderate some respondents will tend to disagree with
amount; 4 = quite a lot; 5 = a very great deal a negatively worded item, that the reliability
How much do you use the following for assessment levels of negatively worded items are lower
purposes? than for positively worded items, and that
negatively worded items receive greater non-
1 2 3 4 5 response than positively worded items. Indeed
(a) commercially published tests [ ] [ ] [ ] [ ] [ ] Weems et al. (2003) argue against mixed-item
(b) your own made-up tests [ ] [ ] [ ] [ ] [ ] formats, and supplement this by reporting that
(c) students’ projects [ ] [ ] [ ] [ ] [ ] inappropriately worded items can induce an
(d) essays [ ] [ ] [ ] [ ] [ ] artificially extreme response which, in turn,
(e) samples of students’ work [ ] [ ] [ ] [ ] [ ] compromises the reliability of the data. Mixing
332 QUESTIONNAIRES

Box 15.4
Potential problems in conducting research

Size of the How much

problem the problem
(1–5) was solved
(1–5)
1 Gaining access to schools and teachers
2 Gaining permission to conduct the research (e.g. from principals)
3 Resentment by principals
4 People vetting what could be used
5 Finding enough willing participants for your sample
6 Schools suffering from ‘too much research’ by outsiders and insiders
7 Schools or people not wishing to divulge information about themselves
8 Schools not wishing to be identifiable, even with protections guaranteed
9 Local political factors that impinge on the school
10 Teachers’ fear of being identified/traceable, even with protections guaranteed
11 Fear of participation by teachers (e.g. if they say critical matters about the
school or others they could lose their contracts)
12 Unwillingness of teachers to be involved because of their workload
13 The principal deciding on whether to involve the staff, without consultation
with the staff
14 Schools’ or institutions’ fear of criticism or loss of face
15 The sensitivity of the research: the issues being investigated

negatively and positively worded items in the same the respondent as it is not always clear how
scale, they argue, compromises both validity and to proceed through the sequence of questions
reliability. Indeed they suggest that respondents and where to go once a particular branch has
may not read negatively worded items as carefully been completed. Redline et al. (2002) found
as positively worded items. that respondents tend to ignore, misread and
incorrectly follow branching instructions, such
Contingency questions, filters and that item non-response occurs for follow-up
branches questions that are applicable only to certain
subsamples, and respondents skip over, and
Contingency questions depend on responses to therefore fail to follow-up on those questions
earlier questions, for example: ‘if your answer to that they should have completed. Redline et al.
question (1) was ‘‘yes’’ please go to question (4)’. (2002) found that the increased complexity of
The earlier question acts as a filter for the later the questionnaire brought about by branching
question, and the later question is contingent instructions negatively influenced its correct
on the earlier, and is a branch of the earlier completion.
question. Some questionnaires will write in words Redline et al. (2002: 7) report that the number
the number of the question to which to go (e.g. of words in the question affected the respondents’
‘please go to question 6’); others will place an arrow ability to follow branching instructions – the
to indicate the next question to be answered if your greater the number of words in the question,
answer to the first question was such-and-such. the greater was the likelihood of the respondents
Contingency and filter questions may be useful overlooking the branching instructions. Redline
for the researcher, but they can be confusing for et al. (2002: 19) report that up to seven items,
 ASKING SENSITIVE QUESTIONS 333

and no more, could be retained in the short-term Asking sensitive questions

Chapter 15
memory. This has implications for the number of
Sudman and Bradburn (1982: ch. 3) draw
items in a list of telephone interviews, where there
attention to the important issue of including
is no visual recall or checking possible. Similarly,
sensitive items in a questionnaire. While the
the greater was the number of answer categories,
anonymity of a questionnaire and, frequently,
the greater was the likelihood of making errors,
the lack of face-to-face contact between the
e.g. overlooking branching instructions. They
researcher and the respondents in a questionnaire
report that respondents tend to see branching
might facilitate responses to sensitive material, the
instructions when they are placed by the last
issues of sensitivity and threat cannot be avoided,
category, particularly if they have chosen that last
as they might lead to under-reporting (non-
category.
disclosure and withholding data) or over-reporting
Further, Redline et al. (2002: 8) note that
(exaggeration) by participants. Some respondents
sandwiching branching instructions between items
may be unwilling to disclose sensitive information,
that do not branch is likely to lead to errors of
particularly if it could harm themselves or others.
omission and commission being made: omitting to
Why should they share private matters (e.g.
answer all the questions and answering the wrong
about family life and opinions of school managers
questions. Further, locating the instructions for
and colleagues) with a complete stranger (Cooper
branching some distance away from the preceding
and Schindler 2001: 341)? Even details of age,
answer box can also lead to errors in following
income, educational background, qualifications,
the instructions. Redline et al. (2002: 17) report
and opinions can be regarded as private and/or
that ‘altering the visual and verbal design of
sensitive matters.
branching instructions had a substantial impact
Sudman and Bradburn (1982: 55–6) identify
on how well respondents read, comprehend,
several important considerations in addressing
and act upon the branching instructions’. It
potentially threatening or sensitive issues,
follows from this that the clear location and
for example socially undesirable behaviour
visual impact of instructions are important for
(e.g. drug abuse, sexual offences, violent
successful completion of branching instructions.
behaviour, criminality, illnesses, employment and
Most respondents, they acknowledge, did not
unemployment, physical features, sexual activity,
deliberately ignore branching instructions; they
behaviour and sexuality, gambling, drinking,
simply were unaware of them.
family details, political beliefs, social taboos). They
The implications of the findings from Redline
suggest the following strategies:
et al. (2002) is that instructions should be placed
where they are to be used and where they can be
seen. O Open rather than closed questions might
We would advise judicious and limited use of be more suitable to elicit information about
filtering and branching devices. It is particularly socially undesirable behaviour, particularly
important to avoid having participants turning frequencies.
pages forwards and backwards in a questionnaire O Long rather than short questions might be
in order to follow the sequence of questions that more suitable for eliciting information about
have had filters and branches following from them. socially undesirable behaviour, particularly
It is a particular problem in Internet surveys frequencies.
where the screen size is much smaller than the O Using familiar words might increase the
length of a printed page. One way of overcoming number of reported frequencies of socially
the problem of branches is to sectionalize the undesirable behaviour.
questionnaire, keeping together conceptually close O Using data gathered from informants, where
items and keeping the branches within that possible, can enhance the likelihood of
section. obtaining reports of threatening behaviour.
334 QUESTIONNAIRES

O Deliberately loading the question so that refer readers to Chapter 5 on sensitive educational
overstatements of socially desirable behaviour research.
and understatements of socially undesirable
behaviour are reduced might be a useful means
Avoiding pitfalls in question writing
of eliciting information.
O With regard to socially undesirable behaviour, Although there are several kinds of questions that
it might be advisable first to ask whether can be used, there are some caveats about the
the respondent has engaged in that behaviour framing of questions in a questionnaire (see http://
previously, and then move to asking about his [Link]/textbooks/9780415368780 –
or her current behaviour. By contrast, when Chapter 15, file 15.11. ppt):
asking about socially acceptable behaviour
O Avoid leading questions, that is, questions
the reverse might be true, i.e. asking about
that are worded (or their response categories
current behaviour before asking about everyday
presented) in such a way as to suggest to
behaviour.
respondents that there is only one acceptable
O In order to defuse threat, it might be useful to
answer, and that other responses might
locate the sensitive topic within a discussion
or might not gain approval or disapproval
of other more or less sensitive matters, in order
respectively. For example:
to suggest to respondents that this issue might
not be too important. Do you prefer abstract, academic-type courses, or
O Use alternative ways of asking standard down-to-earth, practical courses that have some
questions, for example sorting cards, or putting pay-off in your day-to-day teaching?
questions in sealed envelopes, or repeating
The guidance here is to check the ‘loadedness’
questions over time (this has to be handled
or possible pejorative overtones of terms or
sensitively, so that respondents do not feel
verbs.
that they are being ‘checked’), and in order to
O Avoid highbrow questions even with sophisti-
increase reliability.
cated respondents. For example:
O Ask respondents to keep diaries in order to
increase validity and reliability. What particular aspects of the current positivis-
O At the end of an interview ask respondents tic/interpretive debate would you like to see
their views on the sensitivity of the topics that reflected in a course of developmental psychology
have been discussed. aimed at a teacher audience?
O If possible, find ways of validating the data.
Where the sample being surveyed is rep-
resentative of the whole adult population,
misunderstandings of what researchers take to
Indeed, Sudman and Bradburn (1982: 86) suggest
be clear, unambiguous language are common-
that, as the questions become more threatening
place. Therefore it is important to use clear and
and sensitive, it is wise to expect greater bias and
simple language.
unreliability. They draw attention to the fact that
O Avoid complex questions. For example:
several nominal, demographic details might be
considered threatening by respondents (Sudman Would you prefer a short, non-award-bearing
and Bradburn 1982: 208). This has implications for course (3, 4 or 5 sessions) with part-day release
their location within the questionnaire (discussed (e.g. Wednesday afternoons) and one evening per
below). The issue here is that sensitivity and threat week attendance with financial reimbursement
are to be viewed through the eyes of respondents for travel, or a longer, non-award-bearing course
rather than the questionnaire designer; what might (6, 7 or 8 sessions) with full-day release, or
appear innocuous to the researcher might be the whole course designed on part-day release
highly sensitive or offensive to participants. We without evening attendance?
 AVOIDING PITFALLS IN QUESTION WRITING 335

Avoid irritating questions or instructions. For Avoid extremes in rating scales, e.g. ‘never’,

Chapter 15
O O

example: ‘always’, ‘totally’, ‘not at all’ unless there

is a good reason to include them. Most
Have you ever attended an in-service course of
respondents are reluctant to use such
any kind during your entire teaching career?
extreme categories (Anderson and Arsenault
If you are over forty, and have never attended an
2001: 174).
in-service course, put one tick in the box marked
O Avoid pressuring/biasing by association, for
NEVER and another in the box marked OLD.
example: ‘Do you agree with your headteacher
O Avoid questions that use negatives and that boys are more troublesome than girls?’.
double negatives (Oppenheim 1992: 128). For In this case the reference to the headteacher
example: should simply be excised.
How strongly do you feel that no teacher should O Avoid statements with which people tend
enrol on the in-service, award-bearing course who either to disagree or agree (i.e. that have
has not completed at least two years’ full-time built-in skewedness (the ‘base-rate’ problem,
teaching? in which natural biases in the population affect
the sample results).
Or:
Do you feel that without a parent/teacher
association teachers are unable to express their Finally, avoid ambiguous questions or questions
views to parents clearly? that could be interpreted differently from
the way that is intended. The problem of
In this case, if you feel that a parent/teacher ambiguity in words is intractable; at best
association is essential for teachers to express it can be minimized rather than eliminated
their views, do you vote ‘yes’ or ‘no’? The altogether. The most innocent of questions
hesitancy involved in reaching such a decision, is replete with ambiguity (Youngman 1984:
and the possible required re-reading of the 158–9; Morrison 1993: 71–2). Take the following
question could cause the respondent simply examples:
to leave it blank and move on to the
next question. The problem is the double Does your child regularly do homework?
negative, ‘without’ and ‘unable’, which creates What does ‘regularly’ mean – once a day; once a
confusion. year; once a term; once a week?
O Avoid too many open-ended questions
on self-completion questionnaires. Because How many students are there in the school?
self-completion questionnaires cannot probe What does this mean: on roll; on roll but absent;
respondents to find out just what they marked as present but out of school on a field trip;
mean by particular responses, open-ended at this precise moment or this week (there being a
questions are a less satisfactory way of eliciting difference in attendance between a Monday and a
information. (This caution does not hold in the Friday), or between the first term of an academic
interview situation, however.) Open-ended year and the last term of the academic year for
questions, moreover, are too demanding of secondary school students as some of them will
most respondents’ time. Nothing can be more have left school to go into employment and others
off-putting than the following format: will be at home revising for examinations or have
Use pages 5, 6 and 7 respectively to respond to completed them?
each of the questions about your attitudes to in-
How many computers do you have in school?
service courses in general and your beliefs about
their value in the professional life of the serving What does this mean: present but broken;
teacher. including those out of school being repaired; the
336 QUESTIONNAIRES

property of the school or staffs’ and students’ own The golden rule is to keep questions as short
computers; on average or exactly in school today? and as simple as possible.
Have you had a French lesson this week?
Sequencing the questions
What constitutes a ‘week’: the start of the school
week (i.e. from Monday to a Friday), since last To some extent the order of questions in a schedule
Sunday (or Saturday, depending on one’s religion) is a function of the target sample (e.g. how they
or, if the question were put on a Wednesday, since will react to certain questions), the purposes of the
last Wednesday; how representative of all weeks questionnaire (e.g. to gather facts or opinions),
is this week – there being public examinations in the sensitivity of the research (e.g. how personal
the school for some of the week? and potentially disturbing the issues are that will
be addressed), and the overall balance of the
How old are you? questionnaire (e.g. where best to place sensitive
15–20 questions in relation to less threatening questions,
20–30 and how many of each to include).
30–40 The ordering of the questionnaire is important,
40–50 for early questions may set the tone or the mind-
50–60 set of the respondent to later questions. For
example, a questionnaire that makes a respondent
The categories are not discrete; will an old-looking irritated or angry early on is unlikely to have
40 year old flatter himself and put himself in the managed to enable that respondent’s irritation or
30–40 category, or will an immature 20-year old anger to subside by the end of the questionnaire.
seek the maturity of being put into the 20–30 As Oppenheim (1992: 121) remarks, one covert
category? The rule in questionnaire design is to purpose of each question is to ensure that the
avoid any overlap of categories. respondent will continue to cooperate.
Vocational education is available only to the lower
Further, a respondent might ‘read the signs’
in the questionnaire, seeking similarities and
ability students but it should be open to every student.
resonances between statements so that responses
This is, in fact, a double question. What does the to early statements will affect responses to later
respondent do who agrees with the first part of statements and vice versa. While multiple items
the sentence -‘vocational education is available may act as a cross-check, this very process might
only to the lower ability students’ – but disagrees be irritating for some respondents.
with the latter part of the sentence, or vice versa? Krosnick and Alwin (1987) found a ‘primacy
The rule in questionnaire design is to ask only one effect’ (discussed earlier), i.e. respondents tend to
question at a time. choose items that appear earlier in a list rather than
Although it is impossible to legislate for items that appear later in a list. This is particularly
the respondents’ interpretation of wording, the important for branching instructions, where the
researcher, of course, has to adopt a common- instruction, because it appears at the bottom of the
sense approach to this, recognizing the inherent list, could easily be overlooked. Krosnick (1999)
ambiguity but nevertheless still feeling that it is also found that the more difficult a question
possible to live with this indeterminacy. is, the greater is the likelihood of ‘satisficing’,
An ideal questionnaire possesses the same i.e. choosing the first reasonable response option
properties as a good law, being clear, unambiguous in a list, rather than working through a list
and practicable, reducing potential errors in methodically to find the most appropriate response
participants and data analysts, being motivating category.
for participants and ensuring as far as possible that The key principle, perhaps, is to avoid creating
respondents are telling the truth (Davidson 1970). a mood-set or a mind-set early on in the
 QUESTIONNAIRES CONTAINING FEW VERBAL ITEMS 337

questionnaire. For this reason it is important Indeed, many questionnaires keep questions about

Chapter 15
to commence the questionnaire with non- personal details until the very end.
threatening questions that respondents can readily The issue here is that the questionnaire designer
answer. After that it might be possible to move has to anticipate the sensitivity of the topics in
towards more personalized questions. terms of the respondents, and this has a large
Completing a questionnaire can be seen as a sociocultural dimension. What is being argued
learning process in which respondents become here is that the logical ordering of a questionnaire
more at home with the task as they proceed. has to be mediated by its psychological ordering.
Initial questions should therefore be simple, have The instrument has to be viewed through the eyes
high interest value, and encourage participation. of the respondent as well as the designer.
This will build up the confidence and motivation In addition to the overall sequencing of
of the respondent. The middle section of the questionnaire, Oppenheim (1992: ch. 7)
the questionnaire should contain the difficult suggests that the sequence within sections of the
questions; the last few questions should be of questionnaire is important. He indicates that the
high interest in order to encourage respondents to questionnaire designer can use funnels and filters
return the completed schedule. within the question. A funnelling process moves
A common sequence of a questionnaire is as from the general to the specific, asking questions
follows: about the general context or issues and then
moving toward specific points within that. A filter
O Commence with unthreatening factual ques- is used to include and exclude certain respondents,
tions (that, perhaps, will give the researcher i.e. to decide if certain questions are relevant or
some nominal data about the sample, e.g. age irrelevant to them, and to instruct respondents
group, sex, occupation, years in post, qualifica- about how to proceed (e.g. which items to jump
tions etc.). to or proceed to). For example, if respondents
O Move to closed questions (e.g. dichotomous, indicate a ‘yes; or a ‘no’ to a certain question,
multiple choice, rating scales, constant sum then this might exempt them from certain other
questions) about given statements or questions, questions in that section or subsequently.
eliciting responses that require opinions,
attitudes, perceptions, views.
Questionnaires containing few verbal
O Then move to more open-ended questions (or,
items
maybe, to intersperse these with more closed
questions) that seek responses on opinions, The discussion so far has assumed that
attitudes, perceptions and views, together questionnaires are entirely word-based. This
with reasons for the responses given. These might be off-putting for many respondents,
responses and reasons might include sensitive particularly children. In these circumstances a
or more personal data. questionnaire might include visual information
and ask participants to respond to this (e.g.
The move is from objective facts to subjective pictures, cartoons, diagrams) or might include
attitudes and opinions through justifications and some projective visual techniques (e.g. to draw
to sensitive, personalized data. Clearly the ordering a picture or diagram, to join two related pictures
is neither as discrete nor as straightforward as this. with a line, to write the words or what someone
For example, an apparently innocuous question is saying or thinking in a ‘bubble’ picture), to tell
about age might be offensive to some respondents, the story of a sequence of pictures together with
a question about income is unlikely to go down well personal reactions to it. The issue here is that in
with somebody who has just become unemployed, tailoring the format of the questionnaire to the
and a question about religious belief might be seen characteristics of the sample, a very wide embrace
as an unwarranted intrusion into private matters. might be necessary to take in non-word-based
338 QUESTIONNAIRES

techniques. This is not only a matter of appeal to suggesting what might be a desired answer to the
respondents, but, perhaps more significantly, is a subsequent questions). Verma and Mallick (1999:
matter of accessibility of the questionnaire to the 121) suggest the use of emboldening to draw the
respondents, i.e. a matter of reliability and validity. respondent’s attention to significant features.
Ensure that short, clear instructions accompany
each section of the questionnaire. Repeating
The layout of the questionnaire instructions as often as necessary is good practice
The appearance of the questionnaire is vitally in a postal questionnaire. Since everything hinges
important. It must look easy, attractive and on respondents knowing exactly what is required
interesting rather than complicated, unclear, of them, clear, unambiguous instructions, boldly
forbidding and boring. A compressed layout is and attractively displayed, are essential.
uninviting and it clutters everything together; Clarity and presentation also impact on the
a larger questionnaire with plenty of space for numbering of the questions. For example a four-
questions and answers is more encouraging to page questionnaire might contain sixty questions,
respondents. Verma and Mallick (1999: 120) broken down into four sections. It might be off-
suggest the use of high quality paper if funding putting to respondents to number each question
permits. (1–60) as the list will seem interminably long,
Dillman et al. (1999) found that respondents whereas to number each section 1–4 makes the
tend to expect less of a form-filling task than is questionnaire look manageable. Hence it is useful,
actually required. They expect to read a question, in the interests of clarity and logic, to break
read the response, make a mark, and move on to down the questionnaire into subsections with
the next question, but in many questionnaires it section headings. This will also indicate the overall
is more complicated than this. The rule is simple: logic and coherence of the questionnaire to the
keep it as uncomplicated as possible. respondents, enabling them to ‘find their way’
It is important, perhaps, for respondents to be through the questionnaire. It might be useful to
introduced to the purposes of each section of a preface each subsection with a brief introduction
questionnaire, so that they can become involved that tells them the purpose of that section.
in it and maybe identify with it. If space permits, The practice of sectionalizing and sublettering
it is useful to tell the respondent the purposes and questions (e.g. Q9 (a) (b) (c). . .) is a useful
focuses of the sections of the questionnaire, and technique for grouping together questions about
the reasons for the inclusion of the items. a specific issue. It is also a way of making the
Clarity of wording and simplicity of design questionnaire look smaller than it actually is!
are essential. Clear instructions should guide This previous point also requires the question-
respondents: ‘Put a tick’, for example, invites naire designer to make it clear if respondents are
participation, whereas complicated instructions exempted from completing certain questions or
and complex procedures intimidate respondents. sections of the questionnaire (discussed earlier in
Putting ticks in boxes by way of answering a the section on filters). If so, then it is vital that
questionnaire is familiar to most respondents, the sections or questions are numbered so that the
whereas requests to circle precoded numbers at respondent knows exactly where to move to next.
the right-hand side of the questionnaire can be Here the instruction might be, for example: ‘If
a source of confusion and error. In some cases you have answered ‘‘yes’’ to question 10 please go
it might also be useful to include an example to question 15, otherwise continue with question
of how to fill in the questionnaire (e.g. ticking 11’, or, for example: ‘If you are the school principal
a box, circling a statement), though, clearly, please answer this section, otherwise proceed to
care must be exercised to avoid leading the section 3’.
respondents to answering questions in a particular Arrange the contents of the questionnaire in
way by dint of the example provided (e.g. by such a way as to maximize cooperation. For
 COVERING LETTERS OR SHEETS AND FOLLOW-UP LETTERS 339

example, include questions that are likely to be of bottom of the page, perhaps the inclusion of less

Chapter 15
general interest. Make sure that questions that important items at that point. Redline et al. (2002)
appear early in the format do not suggest to suggest that questions with branching instructions
respondents that the enquiry is not intended for should not be placed at the bottom of a page.
them. Intersperse attitude questions throughout Finally, a brief note at the very end of
the schedule to allow respondents to air their the questionnaire can: ask respondents to check
views rather than merely describe their behaviour. that no answer has been inadvertently missed
Such questions relieve boredom and frustration out; solicit an early return of the completed
as well as providing valuable information in the schedule; thank respondents for their participation
process. and cooperation, and offer to send a short
Coloured pages can help to clarify the overall abstract of the major findings when the analysis is
structure of the questionnaire and the use completed.
of different colours for instructions can assist
respondents. Covering letters or sheets and follow-up
It is important to include in the questionnaire, letters
perhaps at the beginning, assurances of confi-
dentiality, anonymity and non-traceability, for The purpose of the covering letter or sheet is
example by indicating that respondents need not to indicate the aim of the research, to convey
give their name, that the data will be aggregated, to respondents its importance, to assure them of
that individuals will not be able to be identified confidentiality, and to encourage their replies. The
through the use of categories or details of their covering letter or sheet should:
location etc. (i.e. that it will not be possible to O provide a title to the research
put together a traceable picture of the respondents O introduce the researcher, her/his name,
through the compiling of nominal, descriptive data address, organization, contact telephone/fax/
about them). In some cases, however, the ques- email address, together with an invitation to
tionnaire might ask respondents to put their names feel free to contact the researcher for further
so that they can be traced for follow-up interviews clarification or details
in the research (Verma and Mallick 1999: 121); O indicate the purposes of the research
here the guarantee of eventual anonymity and O indicate the importance and benefits of the
non-traceability will still need to be given. research
Redline et al. (2002) indicate that the placing O indicate why the respondent has been selected
of the response categories to the immediate right for receipt of the questionnaire
of the text increases the chance of it being O indicate any professional backing, endorse-
answered (the visual location), and making the ment or sponsorship of, or permission for, the
material more salient (e.g. through emboldening research (e.g. university, professional associa-
and capitalization) can increase the chances of tions, government departments: the use of a
it being addressed (the visibility issue). This is logo can be helpful here)
particularly important for branching questions and O set out how to return the questionnaire
instructions. (e.g. in the accompanying stamped, addressed
Redline et al. (2002) also note that questions envelope, in a collection box in a particular
placed at the bottom of a page tend to receive institution, to a named person; whether the
more non-response than questions placed further questionnaire will be collected – and when,
up on the page. Indeed they found that putting where and by whom)
instructions at the bottom of the page, particularly O indicate the address to which to return the
if they apply to items on the next page, can easily questionnaire
lead to those instructions being overlooked. It is O indicate what to do if questions or uncertainties
important, then, to consider what should go at the arise
340 QUESTIONNAIRES

O indicate a return-by date your school development, support management and

O indicate any incentives for completing the monitoring of school effectiveness, and the project
questionnaire will facilitate a change management programme that
O provide assurances of confidentiality, will be tailor-made for the school. This questionnaire
anonymity and non-traceability is seeking to identify the nature, strengths and
O indication of how the results will and will not weaknesses of different aspects of your school,
be disseminated, and to whom particularly in respect of those aspects of the school
O thank respondents in advance for their co- over which the school itself has some control. It
operation. would be greatly appreciated if you would be involved
in this process by completing the sheets attached,
Verma and Mallick (1999: 122) suggest that,
and returning them to me. Please be as truthful as
where possible, it is useful to personalize the letter,
possible in completing the questionnaire.
avoiding ‘Dear colleague’, ‘Dear Madam/Ms/Sir’
etc., and replacing these with exact names. You do not need to write your name, and no
With these intentions in mind, the following individuals will be identified or traced from this, i.e.
practices are to be recommended: confidentiality and anonymity are assured. If you wish
O The appeal in the covering letter must be to discuss any aspects of the review or this document
tailored to suit the particular audience. Thus, a please do not hesitate to contact me. I hope that you
survey of teachers might stress the importance will feel able to take part in this project.
of the study to the profession as a whole. Thank you.
O Neither the use of prestigious signatories,
nor appeals to altruism, nor the addition of Signed
handwritten postscripts affect response levels
to postal questionnaires. Contact details (address, fax, telephone, email)
O The name of the sponsor or the organization Another example might be:
conducting the survey should appear on the
letterhead as well as in the body of the covering Dear Colleague,
letter.
O A direct reference should be made to the PROJECT ON CONDUCTING EDUCATIONAL
confidentiality of respondents’ answers and the RESEARCH
purposes of any serial numbers and codings
should be explained. I am conducting a small-scale piece of research into
O A pre-survey letter advising respondents of the issues facing researchers undertaking investigations in
forthcoming questionnaire has been shown to education. The topic is very much under-researched
have substantial effect on response rates. in education, and that is why I intend to explore the
A short covering letter is most effective; aim at area.
no more than one page. An example of a covering
I am asking you to be involved as you yourself have
letter for teachers and senior staff might be as
conducted empirical work as part of a Master’s or doc-
follows:
torate degree. No one knows the practical problems
Dear Colleague, facing the educational researcher better than you.

The enclosed questionnaire forms part of my investi-

IMPROVING SCHOOL EFFECTIVENESS
gation. May I invite you to spend a short time in its
completion?
We are asking you to take part in a project to
improve school effectiveness, by completing this If you are willing to be involved, please complete the
short research questionnaire. The project is part of questionnaire and return it to XXX by the end of
 PILOTING THE QUESTIONNAIRE 341

November. You may either place it in the collection to eliminate ambiguities or difficulties in

Chapter 15
O

box at the General Office at my institution or send wording

it by post (stamped addressed envelope enclosed), or O to check readability levels for the target
by fax or email attachment. audience
O to gain feedback on the type of question and
The questionnaire will take around fifteen minutes its format (e.g. rating scale, multiple choice,
to complete. It employs rating scales and asks for open, closed etc.)
your comments and a few personal details. You do O to gain feedback on response categories for
not need to write your name, and you will not be closed questions and multiple choice items, and
able to be identified or traced. ANONYMITY AND for the appropriateness of specific questions or
NON-TRACEABILITY ARE ASSURED. When stems of questions
completed, I intend to publish my results in an O to identify omissions, redundant and irrelevant
education journal. items
If you wish to discuss any aspects of the study then O to gain feedback on leading questions
please do not hesitate to contact me.
O to gain feedback on the attractiveness and
appearance of the questionnaire
I very much hope that you will feel able to partici- O to gain feedback on the layout, sectionalizing,
pate. May I thank you, in advance, for your valuable numbering and itemization of the question-
cooperation. naire
O to check the time taken to complete the
Yours sincerely, questionnaire
Signed
O to check whether the questionnaire is too long
or too short, too easy or too difficult
Contact details (address, fax, telephone, email) O to generate categories from open-ended
responses to use as categories for closed
For a further example of a questionnaire see response-modes (e.g. rating scale items)
the accompanying web site ([Link] O to identify how motivating/non-motivating
.com/textbooks/9780415368780 – Chapter 15, file /sensitive/threatening/intrusive/offensive items
[Link]). might be
O to identify redundant questions (e.g. those
questions which consistently gain a total ‘yes’
Piloting the questionnaire or ‘no’ response: Youngman 1984: 172), i.e.
It bears repeating that the wording of ques- those questions with little discriminability
tionnaires is of paramount importance and that O to identify which items are too easy, too
pretesting is crucial to their success (see http:// difficult, too complex or too remote from the
[Link]/textbooks/9780415368780 – respondents’ experience
Chapter 15, file 15.12. ppt). A pilot has several O to identify commonly misunderstood or non-
functions, principally to increase the reliabil- completed items (e.g. by studying common
ity, validity and practicability of the question- patterns of unexpected response and non-
naire (Oppenheim 1992; Morrison 1993: Wilson response (Verma and Mallick 1999: 120))
and McLean 1994: 47): O to try out the coding/classification system for
data analysis.
O to check the clarity of the questionnaire items,
instructions and layout In short, as Oppenheim (1992: 48) remarks,
O to gain feedback on the validity of the everything about the questionnaire should be
questionnaire items, the operationalization of piloted; nothing should be excluded, not even
the constructs and the purposes of the research the type face or the quality of the paper.
342 QUESTIONNAIRES

The above outline describes a particular kind Practical considerations in questionnaire

of pilot: one that does not focus on data, but on design
matters of coverage and format, gaining feedback
Taking the issues discussed so far in questionnaire
from a limited number of respondents and experts
design, a range of practical implications for
on the items set out above.
designing a questionnaire can be highlighted:
There is a second type of pilot. This is one
which starts with a long list of items and, through
statistical analysis and feedback, reduces those O Operationalize the purposes of the question-
items (Kgaile and Morrison 2006). For example, naire carefully.
a researcher may generate an initial list of, O Be prepared to have a pre-pilot to generate
for example, 120 items to be included in a items for a pilot questionnaire, and then be
questionnaire, and wish to know which items ready to modify the pilot questionnaire for the
to excise. A pilot is conducted on a sizeable final version.
and representative number of respondents (e.g. O If the pilot includes many items, and the
50–100) and this generates real data – numerical intention is to reduce the number of items
responses. These data can be analysed for the through statistical analysis or feedback, then
following factors: be prepared to have a second round of piloting,
after the first pilot has been modified.
O Decide on the most appropriate type of
O Reliability: those items with low reliability
question – dichotomous, multiple choice, rank
(Cronbach’s alpha for internal consistency: see
orderings, rating scales, constant sum, ratio,
Part Five) can be removed.
closed, open.
O Collinearity: if items correlate very strongly
O Ensure that every issue has been explored
with others then a decision can be taken
exhaustively and comprehensively; decide on
to remove one or more of them, provided,
the content and explore it in depth and
of course, that this does not result in the
breadth.
loss of important areas of the research (i.e.
O Use several items to measure a specific
human judgement would have to prevail over
attribute, concept or issue.
statistical analysis).
O Ensure that the data acquired will answer the
O Multiple regression: those items with low betas
research questions.
(see Part Five) can be removed, provided,
O Ask more closed than open questions for ease
of course, that this does not result in the
of analysis (particularly in a large sample).
loss of important areas of the research (i.e.
O Balance comprehensiveness and exhaustive
human judgement would have to prevail over
coverage of issues with the demotivating factor
statistical analysis).
of having respondents complete several pages
O Factor analysis: to identify clusters of key
of a questionnaire.
variables and to identify redundant items (see
O Ask only one thing at a time in a question. Use
Part Five).
single sentences per item wherever possible.
O Keep response categories simple.
As a result of such analysis, the items for O Avoid jargon.
removal can be identified, and this can result O Keep statements in the present tense wherever
in a questionnaire of manageable proportions. It is possible.
important to have a good-sized and representative O Strive to be unambiguous and clear in the
sample here in order to generate reliable data wording.
for statistical analysis; too few respondents to this O Be simple, clear and brief wherever possible.
type of pilot and this may result in important items O Clarify the kinds of responses required in open
being excluded from the final questionnaire. questions.
 PRACTICAL CONSIDERATIONS IN QUESTIONNAIRE DESIGN 343

Balance brevity with politeness (Oppenheim provide the instructions for introducing,

Chapter 15
O

1992: 122). It might be advantageous to replace completing and returning (or collection
a blunt phrase like ‘marital status’ with a gentler of) the questionnaire (provide a stamped
‘Please indicate whether you are married, living addressed envelope if it is to be a postal
with a partner, or single . . ..’ or ‘I would be questionnaire).
grateful if would tell me if you are married, O Pilot the questionnaire, using a group of re-
living with a partner, or single’. spondents who are drawn from the possible
O Ensure a balance of questions that ask for sample but who will not receive the final,
facts and opinions (this is especially true if refined version.
statistical correlations and cross-tabulations are O With the data analysis in mind, plan so
required). that the appropriate scales and kinds of data
O Avoid leading questions. (e.g. nominal, ordinal, interval and ratio) are
O Try to avoid threatening questions. used.
O Do not assume that respondents know the O Decide how to avoid falsification of re-
answers, or have information to answer sponses (e.g. introduce a checking mech-
the questions, or will always tell the truth anism into the questionnaire responses to
(wittingly or not). Therefore include ‘don’t another question on the same topic or
know’, ‘not applicable’, ‘unsure’, ‘neither agree issue).
nor disagree’ and ‘not relevant’ categories. O Be satisfied if you receive a 50 per cent
O Avoid making the questions too hard. response to the questionnaire; decide what
O Balance the number of negative questions you will do with missing data and what is the
with the number of positive questions (Black significance of the missing data (that might
1999: 229). have implications for the strata of a stratified
O Consider the readability levels of the sample targeted in the questionnaire), and why
questionnaire and the reading and writing the questionnaires have not been completed
abilities of the respondents (which may lead and returned. For example, were the questions
the researcher to conduct the questionnaire as too threatening or was the questionnaire too
a structured interview). long? (This might have been signalled in the
O Put sensitive questions later in the question- pilot).
naire in order to avoid creating a mental set O Include a covering explanation, thanking the
in the mind of respondents, but not so late potential respondent for anticipated coopera-
in the questionnaire that boredom and lack of tion, indicating the purposes of the research,
concentration have set it. how anonymity and confidentiality will be ad-
O Intersperse sensitive questions with non- dressed, who you are and what position you
sensitive questions. hold, and who will be party to the final
O Be very clear on the layout of the questionnaire report.
so that it is unambiguous and attractive O If the questionnaire is going to be ad-
(this is particularly the case if a computer ministered by someone other than the
program is going to be used for data researcher, ensure that instructions for ad-
analysis). ministration are provided and that they are
O Avoid, where possible, splitting an item over clear.
more than one page, as the respondent may
think that the item from the previous page is A key issue that permeates this lengthy list is
finished. for the researcher to pay considerable attention
O Ensure that the respondent knows how to respondents; to see the questionnaire through
to enter a reply to each question, e.g. their eyes, and envisage how they will regard it (e.g.
by underlining, circling, ticking, writing; from hostility to suspicion to apathy to grudging
344 QUESTIONNAIRES

compliance to welcome; from easy to difficult, On the other hand, having the researcher
from motivating to boring, from straightforward to present may be threatening and exert a sense of
complex etc.). compulsion, where respondents may feel uncom-
fortable about completing the questionnaire, and
may not want to complete it or even start it. Re-
Administering questionnaires spondents may also want extra time to think about
Questionnaires can be administered in several and complete the questionnaire, maybe at home,
ways, including: and they are denied the opportunity to do this.
Having the researcher present also places
O self-administration pressure on the researcher to attend at an agreed
O post time and in an agreed place, and this may be time-
O face-to-face interview consuming and require the researcher to travel
O telephone extensively, thereby extending the time frame
O Internet. for data collection. Travel costs for conducting
the research with dispersed samples could also be
Here we discuss only self-administered and postal
expensive.
questionnaires. Chapter 16 covers administration
by face-to-face interview and by telephone, and
Chapter 10 covers administration by the Internet. Self-administered questionnaires without the presence
We also refer readers to Chapter 9 on surveys, to of the researcher
the section on conducting surveys by interview.
The absence of the researcher is helpful in that it
enables respondents to complete the questionnaire
Self-administered questionnaires in private, to devote as much time as they wish
to its completion, to be in familiar surroundings,
There are two types of self-administered and to avoid the potential threat or pressure to
questionnaire: those that are completed in the participate caused by the researcher’s presence.
presence of the researcher and those that are filled It can be inexpensive to operate, and is more
in when the researcher is absent (e.g. at home, in anonymous than having the researcher present.
the workplace). This latter point, in turn, can render the data
more or less honest: it is perhaps harder to tell lies
or not to tell the whole truth in the presence of the
Self-administered questionnaires in the presence of
researcher, and it is also easier to be very honest
the researcher
and revealing about sensitive matters without the
The presence of the researcher is helpful in presence of the researcher.
that it enables any queries or uncertainties to The down side, however, is that the researcher
be addressed immediately with the questionnaire is not there to address any queries or problems
designer. Further, it typically ensures a good that respondents may have, and they may omit
response rate (e.g. undertaken with teachers at items or give up rather than try to contact
a staff meeting or with students in one or more the researcher. Respondents may also wrongly
classes). It also ensures that all the questions are interpret and, consequently, answer questions
completed (the researcher can check these before inaccurately. They may present an untrue picture
finally receiving the questionnaire) and filled in to the researcher, for example answering what
correctly (e.g. no rating scale items that have more they would like a situation to be rather than
than one entry per item, and no missed items). what the actual situation is, or painting a falsely
It means that the questionnaires are completed negative or positive picture of the situation or
rapidly and on one occasion, i.e. it can gather data themselves. Indeed, the researcher has no control
from many respondents simultaneously. over the environment in which the questionnaire
 ADMINISTERING QUESTIONNAIRES 345

is completed, e.g. time of day, noise distractions, Use first-class – rapid – postage services, with

Chapter 15
O

presence of others with whom to discuss the stamped rather than franked envelopes
questions and responses, seriousness given to the wherever possible.
completion of the questionnaire, or even whether O Enclose a first-class stamped envelope for the
it is completed by the intended person. respondent’s reply.
O In surveys of the general population, Thursday
is the best day for mailing out; in surveys
Postal questionnaires
of organizations, Monday or Tuesday are
Frequently, the postal questionnaire is the best recommended.
form of survey in an educational inquiry. Take, for O Avoid at all costs a December survey
example, the researcher intent on investigating the (questionnaires will be lost in the welter of
adoption and use made of a new curriculum series Christmas postings in the western world).
in secondary schools. An interview survey based
upon some sampling of the population of schools Follow-up letter
would be both expensive and time-consuming. A
postal questionnaire, on the other hand, would Of the four factors that Hoinville and Jowell
have several distinct advantages. Moreover, given (1978) discuss in connection with maximizing
the usual constraints over finance and resources, it response levels, the follow-up letter has been
might well prove the only viable way of carrying shown to be the most productive. The following
through such an inquiry. points should be borne in mind in preparing
What evidence we have about the advantages reminder letters:
and disadvantages of postal surveys derives O All of the rules that apply to the covering
from settings other than educational. Many of letter apply even more strongly to the follow-up
the findings, however, have relevance to the letter.
educational researcher. Here, we focus upon some O The follow-up should re-emphasize the
of the ways in which educational researchers can importance of the study and the value of the
maximize the response level that they obtain when respondent’s participation.
using postal surveys. O The use of the second person singular, the
A number of myths about postal questionnaires conveying of an air of disappointment at non-
are not borne out by the evidence (see Hoinville response and some surprise at non-cooperation
and Jowell 1978). Response levels to postal have been shown to be effective ploys.
surveys are not invariably less than those obtained O Nowhere should the follow-up give the
by interview procedures; frequently they equal, impression that non-response is normal or that
and in some cases surpass, those achieved in numerous non-responses have occurred in the
interviews. Nor does the questionnaire necessarily particular study.
have to be short in order to obtain a satisfactory O The follow-up letter must be accompanied by a
response level. With sophisticated respondents, further copy of the questionnaire together with
for example, a short questionnaire might appear a first-class stamped addressed envelope for its
to trivialize complex issues with which they are return.
familiar. Hoinville and Jowell (1978) identify a O Second and third reminder letters suffer from
number of factors in securing a good response rate the law of diminishing returns, so how many
to a postal questionnaire. follow-ups are recommended and what success
rates do they achieve? It is difficult to
generalize, but the following points are worth
Initial mailing
bearing in mind. A well-planned postal survey
O Use good-quality envelopes, typed and should obtain at least a 40 per cent response
addressed to a named person wherever possible. rate and with the judicious use of reminders,
346 QUESTIONNAIRES

a 70 per cent to 80 per cent response level respondent’s efforts and, according to Hoinville
should be possible. A preliminary pilot survey and Jowell (1978), should be as neutral as possible.
is invaluable in that it can indicate the general In this respect, they suggest that books of postage
level of response to be expected. The main stamps or ballpoint pens are cheap, easily packaged
survey should generally achieve at least as high in the questionnaire envelopes, and appropriate to
as and normally a higher level of return than the task required of the respondent.
the pilot inquiry. The Office of Population The preparation of a flow chart can help the
Censuses and Surveys recommends the use of researcher to plan the timing and the sequencing of
three reminders which, they say, can increase the various parts of a postal survey. One such flow
the original return by as much as 30 per cent chart suggested by Hoinville and Jowell (1978) is
in surveys of the general public. A typical shown in Box 15.5. The researcher might wish to
pattern of responses to the three follow-ups is add a chronological chart alongside it to help plan
as follows: the exact timing of the events shown here.
O Original dispatch 40 per cent
O First follow-up +20 per cent Validity
O Second follow-up +10 per cent
O Third follow-up +5 per cent Our discussion, so far, has concentrated on
O Total 75 per cent ways of increasing the response rate of postal
questionnaires; we have said nothing yet about
Bailey (1994: 163–9) shows that follow-ups can the validity of this particular technique.
be both by mail and by telephone. If a follow-up Validity of postal questionnaires can be
letter is sent, then this should be around three seen from two viewpoints according to Belson
weeks after the initial mailing. A second follow- (l986). First, whether respondents who complete
up is also advisable, and this should take place questionnaires do so accurately, and second,
one week after the first follow up. Bailey (1994: whether those who fail to return their
165) reports research that indicates that a second questionnaires would have given the same
follow-up can elicit up to a 95.6 per cent response distribution of answers as did the returnees.
rate compared to a 74.8 per cent response with The question of accuracy can be checked by
no follow-up. A telephone call in advance of the means of the intensive interview method, a tech-
questionnaire can also help in boosting response nique consisting of twelve principal tactics that
rates (by up to 8 per cent). include familiarization, temporal reconstruction,
probing and challenging. The interested reader
should consult Belson (1986: 35–8).
Incentives
The problem of non-response (the issue of
An important factor in maximizing response ‘volunteer bias’ as Belson calls it) can, in part,
rates is the use of incentives. Although be checked on and controlled for, particularly
such usage is comparatively rare in British when the postal questionnaire is sent out on a
surveys, it can substantially reduce non-response continuous basis. It involves follow-up contact
rates particularly when the chosen incentives with non-respondents by means of interviewers
accompany the initial mailing rather than being trained to secure interviews with such people. A
mailed subsequently as rewards for the return comparison is then made between the replies of
of completed schedules. The explanation of the respondents and non-respondents.
effectiveness of this particular ploy appears to lie
in the sense of obligation that is created in the
Processing questionnaire data
recipient. Care is needed in selecting the most
appropriate type of incentive. It should clearly be Let us assume that researchers have followed
seen as a token rather than a payment for the the advice we have given about the planning
 PROCESSING QUESTIONNAIRE DATA 347

Box 15.5

Chapter 15
A flow chart for the planning of a postal survey

Address and serial number Prepare stamped

Prepare questionnaires Prepare incentives
labels and attach to addressed return
and covering letters (if relevant)
outward envelopes envelopes

Enter serial number

from labels on
questionnaires and
on covering letter

Insert questionnaires, covering letters,

(incentives) and return envelopes
into outward envelopes

Seal and stamp

outward envelopes

MAILING

Mail first reminder

Book in completed
to non-respondents
questionnaires against
sample serial numbers Mail second reminder
to non-respondents

Prepare final Send letter of thanks

response summaries to all respondents

Transfer questionnaires
to data preparation staff

Source: Hoinville and Jowell 1978

of postal questionnaires and have secured a high case of small surveys; by computers when numbers
response rate to their surveys. Their task is now are larger. First, however, prior to coding, the
to reduce the mass of data they have obtained to questionnaires have to be checked. This task is
a form suitable for analysis. ‘Data reduction’, as referred to as editing.
the process is called, generally consists of coding Editing questionnaires is intended to identify
data in preparation for analysis – by hand in the and eliminate errors made by respondents. (In
348 QUESTIONNAIRES

addition to the clerical editing that we discuss in questions – male 1, female 2, for example; or single
this section, editing checks are also performed by 1, married 2, separated 3, divorced 4. For questions
the computer. For an account of computer-run such as those whose answer categories are known
structure checks and valid coding range checks, in advance, a coding frame is generally devel-
see Hoinville and Jowell (1978: 150–5). Moser oped before the interviewing commences so that
and Kalton (1977) point to three central tasks in it can be printed into the questionnaire itself. For
editing: open-ended questions (Why did you choose this
particular in-service course rather than XYZ?), a
O Completeness: a check is made that there is
coding frame has to be devised after the comple-
an answer to every question. In most surveys,
tion of the questionnaire. This is best done by
interviewers are required to record an answer
taking a random sample of the questionnaires (10
to every question (a ‘not applicable’ category
per cent or more, time permitting) and generating
always being available). Missing answers
a frequency tally of the range of responses as a pre-
can sometimes be cross-checked from other
liminary to coding classification. Having devised
sections of the survey. At worst, respondents
the coding frame, the researcher can make a further
can be contacted again to supply the missing
check on its validity by using it to code up a fur-
information.
ther sample of the questionnaires. It is vital to get
O Accuracy: as far as is possible, a check is made
coding frames right from the outset – extending
that all questions are answered accurately. In-
them or making alterations at a later point in the
accuracies arise out of carelessness on the part
study is both expensive and wearisome.
of either interviewers or respondents. Some-
There are several computer packages that
times a deliberate attempt is made to mislead.
will process questionnaire survey data. At the
A tick in the wrong box, a ring round the
time of writing one such is SphinxSurvey. This
wrong code, an error in simple arithmetic – all
package, like others of its type, assists researchers
can reduce the validity of the data unless they
in the design, administration and processing of
are picked up in the editing process.
questionnaires, either for paper-based or for on-
O Uniformity: a check is made that interview-
screen administration. Responses can be entered
ers have interpreted instructions and ques-
rapidly, and data can be examined automatically,
tions uniformly. Sometimes the failure to give
producing graphs and tables, as well as a wide
explicit instructions over the interpretation
range of statistics (the Plus edition offers lexical
of respondents’ replies leads to interviewers
analysis of open-ended text, and the Lexica
recording the same answer in a variety of
Edition has additional functions for qualitative
answer codes instead of one. A check on uni-
data analysis). A web site for previewing a
formity can help eradicate this source of error.
demonstration of this programme can be found at
The primary task of data reduction is coding, that [Link] and is typical of several
is, assigning a code number to each answer to of its kind.
a survey question. Of course, not all answers to While coding is usually undertaken by the
survey questions can be reduced to code num- researcher, Sudman and Bradburn (1982: 149)
bers. Many open-ended questions, for example, also make the case for coding by the respondents
are not reducible in this way for computer anal- themselves, to increase validity. This is particularly
ysis. Coding can be built into the construction valuable in open-ended questionnaire items,
of the questionnaire itself. In this case, we talk though, of course, it does assume not only the
of precoded answers. Where coding is developed willingness of respondents to become involved
after the questionnaire has been administered and post hoc but also that the researcher can identify
answered by respondents, we refer to post-coded and trace the respondents, which, as was indicated
answers. Precoding is appropriate for closed-ended earlier, is an ethical matter.