GESIS Survey Guidelines

Open-Ended Questions

Cornelia Züll

December 2016, Version 2.0

Abstract

This contribution addresses two aspects of open-ended survey questions. First, I deal with the
inclusion of open-ended questions in surveys: When are they useful, and what purpose do they
serve? Who answers such questions? And what should be taken into account when designing
them? Second, I outline possible methods of analysing responses to open-ended questions.
They include content analysis, which has a long tradition in the analysis of responses to open-
ended questions. Computer-assisted, dictionary-based content analysis also plays a major role
in this regard. It is especially suitable for analysing responses to open-ended questions because
they are, as a rule, short and limited by the context of the question. Co-occurrence analysis,
which can provide an overall view of the responses, is a relatively new, additional method of
analysing answers to open-ended questions.

Citation
Züll, C. (2016). Open-Ended Questions. GESIS Survey Guidelines. Mannheim, Germany: GESIS – Leibniz
Institute for the Social Sciences. doi: 10.15465/gesis-sg_en_002
1. What are open-ended questions and where are they applied?
All survey questions that do not include a set of response options are known as open-ended questions.
Open-ended questions require respondents to formulate a response in their own words and to express
it verbally or in writing. Respondents are not steered in a particular direction by predefined response
categories.
Depending on the survey mode, either the question is read out and the answer is recorded by the
interviewer (face-to-face or telephone interviews) or the question appears on the computer screen/on
paper, and the respondent enters the answer into the text field provided (web or postal surveys). In the
first case, it is important that interviewers be given clear instructions as to what should be recorded –
that is, whether the respondent’s answer should be transcribed exactly and in full or whether only
keywords should be recorded. In the second case, the respondent enters the answer him- or herself, and
its length can be limited by the format of the text field.
Open-ended questions may request textual information (e.g., “Can you tell me what you associate with
the term ‘right’?”) or numerical information (e.g., “How many minutes per week do you do sport?”).
Because numerical information differs from textual information in terms of its function and the
cognitive response process involved, the present contribution focuses only on aspects of the collection
and analysis of textual responses to open-ended questions.
Figure 1 provides an example of an open-ended question taken from the German General Social Survey
(ALLBUS 2008; face-to-face interview, [Link]

Can you tell me what you associate with the term “left”?

Record the response given precisely!!

: ____________________________________________

____________________________________________

____________________________________________

Figure 1: Extract from the ALLBUS 2008 questionnaire

When developing a questionnaire, researchers are often faced with the problem of whether or not to
include open-ended questions. There is no generally applicable rule as far as this is concerned. Whether
an open-ended question is appropriate depends on a number of different framework conditions. A
decision for or against the inclusion of open-ended questions in a survey should not be influenced by
1
personal preferences for, or aversions to, this type of question, or by the effort involved in the
subsequent analysis. Rather, it should depend on the research question and the current state of
research.
Open-ended questions can be employed both to gather information and to motivate respondents
(Porst, 2011). They are used for information-gathering purposes in the following cases:
• Knowledge measurement
Open-ended questions are more suitable than closed-ended questions for measuring knowledge,
because they not only minimise the likelihood that respondents will try to guess the right answer
but also often yield more reliable and valid information. However, when respondents cannot
immediately recall the correct answer, open-ended questions also elicit more “don’t know”
responses or refusals than closed questions do (see Krosnick & Presser, 2010).
• Unknown range of possible answers
The use of open-ended questions is recommended if it is not yet possible to clearly delimit the
subject of inquiry, or if one expects new topics to emerge. An example of this is a question fielded
in ALLBUS 2008 about respondents’ associations with the terms left and right (Figure 1). Although
it is known from many surveys what respondents associated with the political dimensions left and
right at the time the respective surveys were conducted (see Fuchs & Klingemann, 1990), it must be
assumed that people’s image of left and right has changed over time and that these terms may
now be associated with other dimensions/topics than they were 20 years ago. To determine
whether this is the case, an open-ended question could be asked.
• Avoidance of excessively long lists of response options
A further field of application for open-ended questions is when the number of possible answers is
so long that the question cannot be asked with predefined response categories. An example of this
is the occupation question. Although researchers know what occupations there are, respondents
cannot, of course, be presented with a list of several hundred response options.
• Avoidance of directive questions
Open-ended questions can be used to avoid steering respondents in a particular direction. Porst
(2011, p. 67) cited as an example the question “Who, in your view, is the best rock band of all
time?” Response categories comprised a selection of the best-known bands and reflected only the
researcher’s own ideas. Such a selection would steer respondents in a certain direction, and there
would be a danger that the categories would be chosen not because they reflected the
respondents’ true preferences or opinions but because they were the only options provided.
• Cognitive pretesting
Open-ended questions are frequently employed in cognitive pretests to determine why a certain
scale value was chosen or to check whether the meaning of a question was understood as intended
(see also the GESIS Survey Guidelines contribution “Cognitive Pretesting”; Lenzner, Neuert, & Otto,
2016).
In addition to employing open-ended questions for information-gathering purposes, they can also be
used to motivate respondents by giving them an opportunity – in between the many closed-ended
questions with predefined response categories – to express their opinions freely and in their own words.
Moreover, in order to give respondents an opportunity to express criticism or make comments, the
following open-ended question is often asked on the last page of the questionnaire: “Is there anything
else that you would like to say about our survey?” (see Porst, 2011, p. 157).

2
2. What should be taken into account when designing open-ended
questions?

2.1 Who answers open-ended questions?

Answering open-ended questions places a greater burden on the respondent than does selecting a
response category in closed-ended questions. The answer must be formulated in the respondent’s own
words, which requires not only a willingness to answer but also the ability to freely articulate the
response. Various researchers have investigated which respondents are even capable of answering
open-ended questions. They found that, as a rule, all respondents are capable of answering this type of
question (Geer, 1988). A major role in answering open-ended questions is played by respondents’ level
of interest in the topic of the question and by the relevance of the topic for them (see Geer, 1991;
Groves, Presser, & Dipko, 2004; Holland & Christian, 2009).
Other studies (e.g., employee surveys) have shown that dissatisfied respondents are more likely than
satisfied respondents to answer, and to give longer answers, because dissatisfied respondents are often
more interested in expressing their dissatisfaction than are satisfied respondents (Andrews, 2005; Borg
& Zuell, 2012; Poncheri, Lindberg, Foster Thompson, & Surface, 2008).

2.2 How should open-ended questions be designed?

2.2.1 Clarity of the question and expected length of the answer

Schuman and Presser (1981) reported an experiment in which either an open-ended or a closed-ended
question on one and the same topic was administered to different subsamples. The content of the
responses elicited by the two question formats was very different, which led the authors to recommend
“… that open-ended questions, lacking the additional cues of fixed alternatives, may need to be more
clearly focused than closed questions” (Schuman & Presser, 1981, p. 105). In other words, in the case of
an open-ended question, respondents must understand very clearly what is expected of them.
When open-ended questions are included in a survey, care should always be taken to avoid addressing
more than one topic per question. If the aim is to ask about both positive and negative aspects of a
topic or about topics that are, or are not, of importance to the respondent, it is better to ask two
questions in each case in order to ensure that respondents address both positive and negative aspects
or important and unimportant topics.
Not only the substantive scope of possible responses but also the length of the answers should be clear
to the respondents. It is therefore recommended that respondents should always be told what type of
response they are expected to give (e.g., only keywords, one item of information, several items of
information/a list, a small essay, etc.).
Some studies have shown that the size of the answer box provided in self-administered questionnaires
correlates positively with the length and detail of the responses (see Dillman, Smyth, & Christian, 2009,
pp. 115-116). In the case of web surveys, for example, it is recommended that (a) the answer boxes
should be adapted to the expected length of the answer or (b) text fields should be used that can be
enlarged by scrolling (Emde & Fuchs, 2012).

2.2.2 Motivation
Dillman et al. (2009) offered a number of recommendations for increasing respondents’ motivation to
answer open-ended questions. According to the authors, open-ended questions should be used
sparingly to avoid overburdening respondents. Moreover, the importance of the information for the
3
researcher and his or her work should be emphasised because this appears to have a motivation-
enhancing effect.

2.2.3 Question design and survey mode

In the case of open-ended questions asked during face-to-face interviews, an important role is played
by the interviewer. By providing additional information and asking follow-up questions, interviewers
can help respondents and motivate them to provide better and more comprehensive answers (Dillman
& Christian, 2005). However, interviewers are also responsible for recording the responses – that is, they
must enter the respondents’ utterances into the questionnaire (as completely and faithfully as possible).
In postal and web surveys, on the other hand, the questionnaire designer plays an important role
because, in this case, the respondents independently enter the responses to the questions without
further support. By providing motivation texts and designing the answer boxes appropriately,
respondents can be motivated to give more detailed answers (see, e.g., Smyth et al., 2009; Emde &
Fuchs, 2012; Züll, Menold, & Körber 2014).

3. How can open-ended questions be coded/analysed?

3.1 Quantitative content analysis

Quantitative content analysis is the classical method of analysing responses to open-ended questions.
One or more coders code the open responses on the basis of a predefined categorisation scheme. The
procedure can be outlined only in brief here. However, a detailed description can be found in Früh
(2011). Quantitative content analysis consists of the following steps:
• Development of a categorisation scheme: Content analysis aimed at coding responses to open-
ended questions always begins with the development of a categorisation scheme that describes the
relevant coding categories. These categories can be derived from theory or directly from the textual
data. As a rule, both sources are used when developing a scheme. Take, for example, the open-
ended question “Can you tell me what you associate with the term ‘left’?” Here, one could
conceivably use, first, the ideology-related categories employed by Fuchs and Klingemann (1990) –
for example, communism, socialism, SPD, and CDU (theory-driven category development).
Then, further categories are defined on the basis of the data (empirically driven approach). This
involves examining a sample of the text and deriving categories on the basis of this examination
(e.g., social equality, social market economy, and other topical issues).
Each category in the categorisation scheme is assigned a label and a category number, followed by
a category definition and examples. If necessary, the category is delineated from other categories
by means of coding instructions. Anchor examples are a useful way of clarifying the categories.
Anchor examples are text fragments that illustrate the meaning of the category particularly well.
An example of such a category description can be found in Züll, Scholz, and Schmitt (2010). The
authors provide a description of the complete categorisation scheme for coding open-ended
questions about respondents’ associations with the terms left and right.
• Coder training: The second step in the content analysis process involves training coders to use the
categorisation scheme and pilot testing the categories. Building on this, the scheme can be revised
if necessary. This process is continued until satisfactory coding results have been achieved – that is,
until an adequate level of intercoder agreement has been reached.

4
• Coding: In Step 3, all the responses are coded. Coding can be carried out on paper or directly on the
computer – for example, in SPSS, in an Excel form, or in special software (e.g., MAXQda,
[Link]).
• Reliability: After coding has been completed, the reliability of the coding should be tested. In Step
4, therefore, a sample of the response texts (depending on the volume of material, e.g., 10% of all
open-ended responses) is independently coded by a second coder and a measure of reliability is
computed that can be used as an indicator of the quality of the coding. Several measures of
reliability are available, for example simple percent agreement, Cohen’s kappa, Scott’s pi, and
Krippendorff’s alpha (see, e.g., Freelon, 2010).
• Data management: Finally, the results of the coding (one or several codes per response) are directly
added to the other survey data and variables and are jointly analysed with them.

3.2 Computer-assisted content analysis

The coding of responses to open-ended questions can be automated with the help of computer-assisted
(dictionary-based) content analysis. This involves coding the responses on the basis of a content-
analysis dictionary, which has the same function as the categorisation scheme described above. In this
case, the coding rules are formulated as lists of words. Instead of a verbal definition of the categories
and the anchor examples, words and phrases are defined that are unequivocal indicators of a particular
category. Whenever these words or phrases appear in a response, the corresponding code is assigned.
One example of such a dictionary-based category definition is category 1540 (racism) in the
categorisation scheme on the meaning of left and right (Züll et al., 2010). Here, the list of words
includes, for example, anti-semitism, fight against ethnic minorities, racial hatred, dark-skinned
people, and marginalisation of other ethnic groups. A detailed description of the procedures to be
followed when conducting a computer-assisted content analysis of responses to open-ended questions
can be found in Züll and Mohler (2001).
In the case of computer-assisted content analyses, the texts must always be available in machine-
readable form. For most surveys, this is no longer a problem because they are conducted as CAPI, CATI
or web surveys. However, in the case of face-to-face interviews without computer assistance or postal
surveys, the effort involved in transcribing the texts must be taken into account.
The results of computer-assisted content analyses are in the same form as those of coder-based
content analyses – namely, one or more codes per response, which can be directly added to the other
survey data.
As in the case of coder-based content analysis, the quality of the coding in computer-assisted content
analysis should also be tested. Here, a sample of the texts is coded by a coder, and the level of
agreement between the coder-based and the machine-generated coding is calculated as described
above.
The implementation of computer-assisted content analysis calls for suitable software that supports the
development of the dictionary and carries out the automatic coding. These software programs include,
for example:
• MAXDictio, an add-on module of MAXQDA ([Link])
• TextQuest ([Link]/pages/[Link])
• WORDSTAT ([Link]
The advantage of computer-assisted content analysis is that large volumes of data – either in surveys
with very many respondents or in several surveys in which the same question was fielded – can be

5
coded quickly and reliably. However, the effort involved in the definition and validation of suitable lists
of words should not be underestimated.

3.3 Other approaches to coding and analysis

3.3.1 Semi-automatic coding

By now, a number of semi-automated procedures have been developed to improve the coding of texts.
They include, for example, “supervised machine learning,” an approach proposed by Giorgetti and
Sebastiani (2003): On the basis of a manually coded text sample (“a training set of answers”) further
texts can be automatically coded. The program “learns” from the manually coded texts how particular
responses should be coded.
An alternative semi-automated procedure is the structural topic model proposed by Roberts et al.
(2014), which is based on a machine learning approach to the analysis of textual data. Information
about the respondents, such as their sex, age, or political affiliation, is incorporated into the analysis.

3.3.2 Co-occurrence analysis

Besides the coding of the texts, there are a number of different procedures for directly analysing
responses to open-ended questions without assigning one or more codes to each respondent (each
individual response). These procedures include co-occurrence analysis, which focuses on words that
occur together within a response. This co-occurrence forms the basis of the analysis. The thus
generated similarity or distance matrix is further analysed, for example by subjecting it to cluster,
correspondence, or multidimensional scaling analysis. Kronberger and Wagner (2002) described at
length their approach to analysing open-ended questions. When doing so, they also addressed the
related specifications.
Programs such as TLab ([Link]/[Link]) and Alceste ([Link]
[Link]/[Link]) are suitable for this type of analysis.
Conceptual mapping, a procedure proposed by Jackson and Trochim (2002), is a similar approach.

References

Andrews, M. (2005). Who is being heard? Response bias in open-ended responses in a large
government employee survey. Paper presented at the 60th Annual Conference of the American
Association for Public Opinion Research, Miami Beach, FL.
Borg, I., & Zuell, C. (2012). Write-in comments in employee surveys. International Journal of Manpower,
33(2), 206-220. doi: 10.1108/01437721211225453
Dillman, D. A., & Christian, L. M. (2005). Survey mode as a source of instability in responses across
surveys. Field Methods, 17(30), 30-52.
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet, mail, and mixed-mode surveys. The
tailored design method. Hoboken, NJ: Wiley.
Emde, M., & Fuchs, M. (2012). Using adaptive questionnaire design in open-ended questions: A field
experiment. Paper presented at the American Association for Public Opinion Research (AAPOR)
67th Annual Conference, San Diego, USA.

6
Freelon, D. G. (2010). ReCal: Intercoder reliability calculation as a web service. International Journal of
Internet Science, 5(1) 20-33.
Früh, W. (2011). Inhaltsanalyse - Theorie und Praxis (Vol. 7., revised ed.). Konstanz: UVK Medien.
Fuchs, D., & Klingemann, H.-D. (1990). The left-right schema. In M. K. Jennings & J. W. Van Deth (Eds.),
Continuities in political action: A longitudinal study of political orientations in three Western
democracies (pp. 203-234). Berlin: de Gruyter.
Geer, J. G. (1988). What do open-ended questions measure? Public Opinion Quarterly, 52(3), 365-371.
doi: 10.1086/269113
Geer, J. G. (1991). Do open-ended questions measure "salient" issues? Public Opinion Quarterly, 55(3),
360-370. doi: 10.1086/269268
Giorgetti, D., & Sebastiani, F. (2003). Automatic survey coding by multiclass text categorization
techniques. Journal of the American Society for Information Science and Technology, 54(14),
1269-1277.
Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions.
Public Opinion Quarterly, 68(1), 2-31. doi: 10.1093/poq/nfh002
Holland, J. L., & Christian, L. M. (2009). The influence of topic interest and interactive probing on
responses to open-ended questions in web surveys. Social Science Computer Review, 27(2), 197-
212. doi: 10.1177/0894439308327481
Jackson, K. M., & Trochim, W. M. K. (2002). Concept mapping as an alternative approach for the analysis
of open-ended survey responses. Organizational Research Methods, 5(4), 307-336.
Kronberger, N., & Wagner, W. (2002). Keywords in context: Statistical analysis of text features. In M. W.
Bauer & G. Gaskell (Eds.), Qualitative researching with text, image and sound: A practical
handbook (pp. 299-317). Thousand Oaks, CA: Sage.
Krosnick, J. A., & Presser, S. (2010). Question and questionnaire design. In P. V. Marsden & J. D. Wright
(Eds.), Handbook of survey research (Vol. 2, pp. 263-314). Bingley, UK: Emerald Group Publishing
Limited.
Lenzner, T., Neuert, C., & Otto, W. (2016). Cognitive Pretesting. GESIS Survey Guidelines. Mannheim,
Germany: GESIS – Leibniz Institute for the Social Sciences. doi: 10.15465/gesis-sg_en_010
Poncheri, R. M., Lindberg, J. T., Foster Thompson, L., & Surface, E. A. (2008). A comment on employee
surveys: Negativity bias in open-ended responses. Organizational Research Methods, 11, 614-
630.
Porst, R. (2011). Fragebogen. Ein Arbeitsbuch. Wiesbaden: VS-Verlag.
Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., & Rand,
D. G. (2014). Structural topic models for open-ended survey responses. American Journal of
Political Science, 58(4), 1064-1082., doi: 10.1111/ajps.12103
Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys. Thousand Oaks, CA: Sage.
Smyth, J. D., Dillman, D. A., Christian, L. M., & McBride, M. (2009). Open-ended questions in web surveys.
Can increasing the size of answer boxes and providing extra verbal instructions improve response
quality? Public Opinion Quarterly, 73(2), 325-337.
Züll, C., Menold, N., & Körber, S. (2014). The influence of answer box size on item nonresponse to open-
ended questions in a web survey. Social Science Computer Review, doi:10.1177/
0894439314528091

7
Züll, C., & Mohler, P. P. (2001). Computerunterstützte Inhaltsanalyse: Codierung und Analyse von
Antworten auf offene Fragen. GESIS How-To-Reihe. Retrieved from
[Link]
Züll, C., Scholz, E., & Schmitt, H. (2010). Kategorienschema für die Vercodung von Assoziationen mit den
politischen Richtungsbegriffen "links" und "rechts". GESIS Technical Report 2010/19.