Effective Questionnaire Design for Surveys
Effective Questionnaire Design for Surveys
STRAND: STATISTICS
Unit 4 Sampling
TEXT
Contents
Section
4.2 Sampling
CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
4 Sampling
4.1 Questionnaires and Surveys
The UK Government's census, first undertaken in 1801 and with the exception of 1941,
continuing to be held every ten years, is a vast and costly exercise but is important for
policy and planning purposes. The most recent UK census took place on 27th March
2011; data is available from the Office for National Statistic (ONS) at
[Link]
There have been suggestions that a sample (that is, a subgroup of people) rather than a
full census (everyone) might be a more efficient method of collecting the data.
In either case though, as for any worthwhile survey, the questions asked must be clear,
unambiguous and unbiased. This is the theme of this section.
Note
When designing a questionnaire to use as part of a survey, bear in mind the following
guidelines.
• The questionnaire should try to find out the information you need.
• You should know how you are going to collect your responses.
• Questions should be clear and concise with no ambiguity.
• Do not ask for information you do not need.
• Allow for all possible responses.
• Questionnaires should be fair and not biased in any way.
• The people asked to complete the questionnaire should be from a variety of
backgrounds.
It should also be noted that if you need to design a questionnaire for your project work, it
is strongly recommended that you first try it out with a small number of people. This is
called a pilot survey.
Questions can be designed in two distinct ways, namely those that require a specified
response to be chosen from a number of options or by giving a number, and those that
allow more detailed responses. The first group are often referred to as closed questions;
the second as open questions. Here are some examples.
Closed
1
4.1 CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
The second example of a closed question shows an example with multiple responses – by
which we mean more than 2! So 'YES / NO' is not regarded as a multiple response.
There is though a problem with these responses as there is potential overlap. i.e. in which
category do you place 1 hour or 2 hours, etc.? It should be made clear, i.e. 0 – 1 includes
everything up to and including 1, etc.
Open
The second example of open questions is good for a face-to-face discussion, as it allows a
wide range of answers. The questioner will then be able to react to these answers, and
continue the discussion. This example is bad for a questionnaire for the same reason!
Any answer is possible but the writer of the questionnaire does not have the chance to
follow up answers. It could though be modified and become a closed question with
multiple responses: for example,
"Please tick a box to show where you went on holiday last year."
Worked Example 1
Consider this questionnaire:
Are you concerned about the environment?
Are you concerned about the level of pollution caused by cars?
Do you think the health of young children is at risk due to exhaust fumes from
cars?
Is there too much congestion in the city centre?
Is public transport under-used?
Do you think cars should be banned from the city centre?
Comment on the questions given here.
Solution
This questionnaire is very biased and has been designed to lead people towards answering
'yes' to the last question. There are no questions about the advantages of cars, the
problems of using public transport or other ways of reducing pollution.
2
CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
4.1
Worked Example 2
What is wrong with the following survey, used to find the favourite sports of the students
in one school?
Solution
The question used in this survey does not cover all the possible responses well. Many
students might have, for example, football as their favourite sport and so would tick the
box marked 'Other'. This would give unhelpful results. Also, students are asked to make
only one choice although they may well like more than one sport.
The following wording would produce more useful responses.
I've no great
I dislike this
preference
I hate this
this sport
I like this
sport
sport
sport
Tennis
Rugby
Netball
Basketball
Swimming
Snooker
Please put any
other sport that
you like here
3
4.1 CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
Exercises
1. Consider the following two sets of questions.
(a) Comment on the way that the last question might be answered in each case.
(b) Write a better set of questions to find out if people like eating meat.
She then says that her class's favourite breakfast cereal is Cornflakes.
(a) Is her conclusion valid?
(b) Criticise her question.
(c) Write a better question for her to use.
5. You have been asked to estimate what percentage of all cars on the road have each
registration letter (M, N, etc.).
(a) Describe how you would collect the data for a survey to answer this
problem.
(b) Describe any problems that might arise as you collect data.
4
CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
4.1
6. Akiko did a survey to find out which colours of cars are popular.
She found that red was the most popular colour. She did not find any cars that were
purple or pink.
Think about a survey which you could do. It must not be about the colour of cars.
(a) What is your survey about?
(b) Write down two things which you might expect to find out.
% Boys % Girls
(a) Which two sports are the most popular with both boys and girls?
(b) The question asked in the survey was:
Which sports do you take part in?
You want to find out more about pupils' involvement in sport.
Write down another question that you could ask.
5
CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
4.1
9. The school governors are worried about road safety outside the school gates.
They think there ought to be a Pelican Crossing.
To decide on this, they need to find out how much traffic comes along the road at
different times of day and on different days of the school week.
Design an observation sheet to gather this information.
10. Mee Ling thinks that pupils who come to school by bus are more likely to be late
than those who do not travel by bus.
In order to test whether or not this is true, she carries out a survey on 100 pupils,
from years 7 and 8, for 5 consecutive Tuesdays.
The results are shown in the following table.
Bus 150 40
Cycle 50 10
Car 100 22
Walk 200 25
TOTAL 500 97
You are asked to conduct a survey to find out what students eat for
breakfast.
Design an observation sheet to collect the data you need.
Invent the first 20 entries on your data sheet.
6
4.1 CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
(b) The newspaper made the following statement about the eating habits of
teenagers.
Only one in a hundred teenagers eats fruit
and vegetables each day. Over half eat no
vegetables other than chips.
You are asked to find out whether this statement is true in your area.
Give three questions you could ask teenagers to see if what the article says is
true in your area.
12. A bus company attempted to estimate the number of people who travel on local
buses in a certain town. They telephoned 100 people in the town one evening and
asked 'Have you travelled by bus in the last week?'
Nineteen people said 'Yes'. The bus company concluded that 19% of the town's
population travel on local buses.
Give 3 criticisms of this method of estimation.
4.2 Sampling
In 1975 the Government wanted census information before the 1981 full census, so a ten
per cent census was carried out using 1 in 10 of the population. This is known as a survey
or sample. Data are obtained by asking people to fill in forms which are then given to
collectors trained to sort out any queries.
When deciding how to carry out a data collection there are several decisions to be made:
(a) What size of sample can you reasonably expect to take, given limited time, money
and resources?
(b) How are items to be used in the sample to be chosen to avoid introducing bias?
(c) How are the data to be collected to avoid bias?
The answer to question (a) clearly depends on the individual circumstances. It should be
obvious, however, that the larger the sample the more sensitive the result.
In questions (b) and (c) the key element is to eliminate possible bias. For example, if you
are a political pollster wanting to know the state of the parties, then a sample taken only
in an affluent, expensive out-of-town suburb will be biased just as a sample taken in a
run-down inner city area will be. So the sample must not be biased – but the same also
applies to the questions. Questions such as
"Should we increase spending on education?"
are biased as they do not represent the complete picture. If we spend more on education,
this means that we must spend less on something else. A less biased question would be,
"Should we spend more on education and less on other services?"
Here, we will be looking for sampling methods that lead to a representative sample, by
which we mean that the sample has similar properties to the total population. As we said
before, here we use the word 'population' not just for human or animal populations but to
represent the complete group under study, e.g. all schools in the UK, all cars in England,
etc.
7
CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
4.2
The main methods used for sampling in practice are described below. First, we create a
list of all members of the population; this is called the sampling frame.
(a) Random – to be truly random each individual must have an equal chance of being
chosen. This method is often used for selecting people from Electoral Registers. If
the researcher is calling at people's houses the system must be rigidly adhered to
(i.e. call back if people are out). It does not necessarily ensure a representative
sample.
(b) Systematic – taking items at regular intervals, e.g. every 4th tree when sampling in
a forest. Although this does not necessarily ensure a representative sample it
should be better than random sampling. Again the system must be rigidly adhered
to. This method is often used when sampling goods on a production line.
(c) Stratified – this is used to ensure that the sample is representative and that it has
the same proportions as the population. To do this you first need to divide the
whole of the population into appropriate categories and then, inside each category,
choose a random sample of appropriate size. This can be difficult in practice.
What is commonly used in street surveys is a quota sampling method where
interviewers are simply asked to interview a certain proportion of each category,
e.g. age groups, and these can be chosen at random.
(d) Cluster – here, because of the large size of a population, particular regions are
drawn at random, and then an appropriate number of individuals are drawn at
random from each chosen region.
In the following example you will see how these methods are used in practice.
Worked Example 1
There are 12 teachers in a small school. A sample of
1. Mrs Skirton
size 4 is to be selected from this population.
2. Mrs Pearson
(a) Create a systematic sample for the population. 3. Mrs Green
(b) Create a random sample for the population. 4. Mrs Crocker
5. Mrs Barnes
Solution 6. Mr Vinner
7. Mr Krishnan
(a) As there are 12 teachers, a systematic sample
8. Mr Gadd
could be made by selecting every third
teacher from the list. This would create a 9. Mr Hampson
sample containing: 10. Mr Thompson
11. Mr Mobey
Mrs Green, Mr Vinner, Mr Hampson and Mr Grainger.
12. Mr Grainger
80 02 86 03
23 90 18 56
84 34 73 51
(b) The list of random numbers shown here has been
31 98 73 75
generated using a spreadsheet. The numbers have
28 73 32 83 been arranged to give two digit numbers. Numbers
52 70 53 23 greater than 12 are not needed in this case so these
47 14 69 68 numbers can be deleted.
15 45 05 18
59 79 80 51
75 11 01 10
8
4.2 CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
Worked Example 3
The diagram below shows 57 small fish. To estimate the average fish length (that is, the
length in mm from the tip of its 'nose' to the middle of its tail, as shown here length
take random samples of size
(a) 5 (b) 10.
Which will give you the more accurate estimate?
1
4
3 5
2
6 7
10 11
12
13 14
15
16
19
17 18
23
21 22
20
27
24 25 26
29
28
30
32
31
35
33
36
34
37
39
38
40
41
44
43
42
45
46
48
47
51 49 50
53 54
52
57
55 56
10
4.2 CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
Solution
Here there are no obvious different categories so we will take a random sample. There is
a table of random numbers on page 15.
Starting arbitrarily on row 16, combining two digits together gives numbers from 00 to
99. We use only numbers in the range 01 to 57 (and ignore any repeats), so that each fish
has an equal chance of being picked.
8 4 1 3 8 4 9 1 5 4 1 1 9 9 8 2 8 4 7 5 4 3 3 4 3 4 0 9 0 8
84 13 84 91 54 11 99 82 84 75 43 34 34 9 8
↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓
13 54 11 43 34 (repeat so 9 8
do not use)
For a sample of size 10, we continue in the same way with the random number table,
choosing as our sample, the
{13th, 54th, 11th, 43rd, 34th, 9th, 8th, 20th, 32nd, 23rd } fish
and these have an average length of
(3.1 + 2.6 + 1.8 + 3.0 + 3.3 + 1.7 + 3.5 + 1.5 + 1.7 + 2.5) = 2.47 mm
10
We would expect this second estimate of 2.47 mm to be more accurate than the first one
since it is obtained from a larger sample size.
Worked Example 4
Irving is a community worker and is asked to assess the views of the community in which
he works on the introduction of a neighbourhood watch scheme. The community has
three different types of housing consisting of a new building development of 270 privately
owned homes, a large housing association estate of 450 houses and 180 flats for the
elderly.
Explain how Irving could obtain a stratified random sample of size 100.
There are three categories of housing in the community. Housing Assn. 450
Flats 180
Total 900
11
4.2 CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
Since Irving requires a sample of size 100 there will No. No. ÷ 9
be one sample member for every 9 in the
population. Private 270 30
This gives the representative numbers as shown
Housing Assn. 450 50
on the right:
Flats 180 20
Inside each category, they need to be chosen at random by numbering, for example, the
private houses 001 to 270, and using the first 30 numbers in this range from a table of
random digits, taken three at a time.
Worked Example 5
A survey is carried out to investigate the quality of new houses.
All the people in the country who have recently bought a new house are to be surveyed.
(a) Give one reason why a pilot survey might be carried out first.
(b) Give one reason why the survey should be carried out by post.
(c) Give one possible disadvantage of a postal survey.
Solution
(a) To see if there are any difficulties with any of the questions in the survey.
(b) It is far cheaper than sending people to each new house in the country.
(c) Not everyone will respond to a postal survey so the survey might not be truly
representative.
Exercises
1. A factory contains 24 identical machines which are labelled:
A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X
12
4.2 CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
4. In order to form a sample a number of pupils are selected from a number of different
teaching groups. The size of the samples and the teaching groups are given in the
table below.
13
CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
4.2
11 57 42 70 41 12 88 51 Create a second random sample, by
32 38 14 65 49 46 18 62 starting in a different place in the
48 20 47 44 56 65 46 36 list of random numbers.
42 59 61 41 78 42 72 11
50 42 58 78 71 78 12 37 How many squares do the two
66 08 21 84 94 61 31 30 samples have in common?
14 64 51 05 53 93 45 86
93 49 05 27 54 18 64 57
87 61 55 67 23 26 70 75
03 87 19 48 10 69 35 61
42 52 83 74 35 09 13 36
51 43 76 62 91 39 89 75
6. What problems might be encountered if samples are formed in the following ways?
(a) Selecting people at random from a telephone directory.
(b) Selecting every third person entering a shopping arcade.
(c) Selecting people at random at a football match.
(d) Selecting people leaving a golf course in red cars.
8. Describe how you might select a sample if you were asked to conduct a survey to
find out if:
(a) the parents of primary school children were happy with their schools,
(b) users of mobile phones were happy with the service provided,
(c) the local bowling green was properly maintained,
(d) the local public transport services are adequate.
9. (a) In a school there are 420 pupils in the lower school, 310 pupils in the middle
school, and 130 pupils in the upper school.
(i) How many pupils from each part of the school should be included in a
stratified random sample of size 100?
(ii) Explain why a stratified random sample should be taken rather than a
simple random sample.
(b) These 100 pupils were asked to keep a record of the number of hours of
television that they watched in one week.
The results are summarised below.
No. of viewers 7 3 15 35 22 10 6 2
14
CMM Subject Support Strand: STATISTICS Unit 4 Sampling: Text
50339 66056 60354 65281 29894 37775 34516 11016 69852 54617
67913 58978 35732 47891 64046 04856 97359 73264 19503 37402
80090 57366 79265 48560 14360 87736 91582 27931 38780 05247
13249 74207 05827 30621 83365 47467 32843 91140 18080 40854
76852 80223 46216 65444 64815 81402 50385 77524 73167 43195
16731 07241 55052 54549 92503 74292 68834 89765 41287 36098
98006 83605 33978 33014 52325 17769 53144 13260 34459 85508
56587 43498 48494 71779 58770 27636 32147 99218 01012 26591
83471 72666 64849 24946 63288 51123 85098 50281 64871 47266
52502 18474 08744 40914 79954 28137 63705 89423 49704 90259
03846 57786 80554 51830 19528 19318 70590 50811 50034 99810
40864 30009 89931 33920 20729 04074 14893 06818 83086 36458
37140 43492 22727 88547 56098 32953 53634 91516 82806 39158
35137 83718 60390 45365 31548 65730 64725 96106 70568 79585
26709 48705 88170 52680 26408 18142 71563 73812 87382 45203
15