Reliability Matters

WHITE PAPER

Writing Test Questions

Writing Test Questions
Writing meaningful test questions is as challenging as writing Application uses a general concept to solve problems in a specific
effective learning objectives. How do you best assess student situation, using learned material in new and concrete situations.
performance and knowledge to ensure they grasped the informa- An example is content and assessments to prepare students for
tion from the training? the NERC certification exam. The exam determines the operator’s
ability to apply the applicable NERC standards to their role as a
Test questions should tie back to learning objectives in the same
Real-time system operator.
way learning objectives tie back to job tasks. Often, the best way
to write good test questions is to write them at the same time you
Building a Test Question
write your learning objectives. That way you know you are testing
students on what you want them to learn. Once you identify the skill or knowledge to teach, learning
objectives are developed. Test questions should be created at this
Test questions are part of the Evaluation phase of the ADDIE pro-
time too.
cess. In this phase, the student’s capability to perform the tasks
are evaluated. Again, the assessment is based on how the learning Let’s begin with the structure of a quality test question:
objectives are written. 1. Identify the criterion to assess, which should be one
concept within the overall assessment
Why do We Test? 2. Select an assessment method. If you’ve chosen to create
This white paper discusses best practices for writing test ques- a written test, this step includes selecting a question type
tions and grouping them in such a way that test takers who have 3. Compose the question stem, or the question part. The
mastered the subject matter consistently pass the exam. Testing stem is the introductory statement completed, answered,
or made true by the correct response.
occurs for several reasons:
4. Compose the correct response
• Student assessment
5. Compose distractors or answer options. Types of
• Student selection and placement distractors depend on the question type used and level
• Student motivation of difficulty Many standardized tests use the following
formula:
• Instructional improvement
a. A distractor very similar to the correct answer.
• Program evaluation If you want to increase the level of difficulty,
• Program feedback include a correct answer that is not the best
answer
Bloom’s Taxonomy b. A distractor related to the correct answer.
The distractor should be recognized as a
Before creating any test, we must identify what we are testing and
potential correct answer to a learner some-
the outcome we want to achieve based on our learning objectives. what familiar with the concept being tested
Bloom’s Taxonomy is a useful reference which helps define each c. A distractor that is not related to the subject
skill and related behavior. Bloom’s Taxonomy shows six levels of matter. This distractor is easily eliminated if
learning and behavior that results from training. In this white paper, the student is familiar with the general
we focus on writing test questions to measure the bottom three concept
levels – Knowledge, Comprehension, and Application. Here are a few tips for writing questions:
Knowledge is the ability to recall or remember something without • Clearly define a single problem using a direct question
necessarily understanding, applying, or changing it – the first • Contain directions clearly indicating the response required
assessment level. by the student
Comprehension, the next level, is understanding something com- • Avoid negatives such as not or except. Negatives can
municated without necessarily relating it to anything else. cause unnecessary confusion or difficulty

WRITING LEARNING OBJECTIVES | SOS 2

Writing response options is equally important. Distractors should • Include most of the information in the stem. The stem
challenge the student, but not be so confusing that a student should clearly state the problem and help orient the student
who has learned the material has trouble determining the correct by providing a frame of reference. The prepared student
should be able to anticipate the correct answer after read-
answer.
ing the complete stem
Tips on writing responses include: • Avoid using all of the above or none of the above. Some
• Be as similar as possible in type, concept, focus, students may read the first response, realize it is correct,
and length and choose the answer without reading the alternatives.
Even if only one response is incorrect, all of the above is
• Use the same grammatical format eliminated quickly. Similary, if a student finds two correct
• Arrange in a logical order responses, they quickly determine that all of the above
Distractors should: is the correct response
– Consider options such as B and C are correct.
• Be plausible and appeal to students who do not know the
For NERC CEH providers, multiple choice
correct answers single answer is NERC’s preferred method
• Not be mutually exclusive, which reduces the student’s – Note: none of the above can be used to
ability to eliminate responses discourage students from guessing, making
• Appeal to students who do not know the correct the question more difficult
answers • Use negatives sparingly. Items with not should be used
only when necessary due to the confusion they can cause.
When possible, randomize the correct response position across
• Avoid having the longest answer option as the correct
the questions in the assessment. Make sure the correct answer one. Many students quickly realize the longest answer is
is repositioned on each question so it doesn’t appear in the same correct and select it before reading the question
place each time. This practice reduces students’ ability to memo-
rize answer positions on retakes. True/False
Determining the difficulty level of questions is also important when The advantage of true/false questions is students can answer
writing test questions. The difficulty of each question should be many questions in a short period. This format is especially useful
appropriate in relation to the training goals and learning objectives. in assessing factual knowledge. It’s also easy to score many ques-
Regardless of the complexity of the content being evaluated, the tions. However, students also have a 50/50 chance of guessing the
level of difficulty can be adjusted at the question level. Going back correct answer. Since there are only two choices, the stem should
to the first three levels of Bloom’s Taxonomy, is the goal to remem- be challenging enough that the answer is not immediately obvious.
ber, understand, or apply knowledge to their role? Best practices for true/false questions include:

Question Types • Word the question so superficial knowledge suggests

an incorrect response
Written tests come in a variety of formats. We discuss the charac- • Be sure only one idea is tested with each question.
teristics of different tests in the following information. Focusing the scope of the question helps you avoid
overly confusing questions. Similarly, true/false
Multiple Choice questions should be written so a question is either
Multiple choice questions are some completely true or completely false
of the most common. Each multiple • Avoid using negatives
choice question should test one which tend to confuse
students as they expect
point, fact, or concept. Because
questions to be worded
these questions contain several an- in the positive. Students
swers that differ in relative correct- may skim over the
ness, you may ask the student to negative neglecting
make subtle distinctions among the it altogether
responses. Multiple choice items
are versatile, can be scored quickly Matching
and accurately, are easy to respond Matching requires stu-
to, and are familiar to students. Mul- dents to select answers
tiple choice questions are the basis from a list of given re-
of most standardized examinations, sponses. This question
like the NERC certification exam. type is versatile, ranging
The following are suggestions and from fill-in-the-blank with
best practices for writing good a word bank to associat-
multiple choice questions: ing concepts in two
distinct lists – a column
• Write direct questions. In of stimulus words or
most cases, a direct question is preferred
phrases and an adjoining list of responses.
to an incomplete stem

WRITING LEARNING OBJECTIVES | SOS 3

Stems range in length from complete sentences to a single word. This process is also important for NERC compliance. To pass a
Students can answer many questions in a short time period. NERC audit, you must demonstrate questions are mapped to the
learning objectives and students have mastered the objectives.
Some tips for writing matching questions include:
Reliability, in this case the ability of a test to accurately and con-
• Use homogeneous lists. A matching section should be
grouped by a general concept. If the questions within a sistently measure the student’s mastery of a cognitive ability, is
matching section vary greatly, students can easily match another concern when constructing a test.
items due to their distinctness Some ways to enhance a test’s reliability are:
• Use longer phrases as the stem (typically the left-hand
column) • Make the test longer
• Include a few more responses than stems. The more • Make the test the appropriate length if it is timed.
responses provided, the more difficult the questions Most test takers can answer four matching, three true/
become, which makes the process of elimination false, one and a half multiple choice, and one short
strategy more difficult for the test taker answer per minute
• Place the stems and responses in a logical order –
either alphabetical or chronological • Ensure the appropriate level of difficulty

• Keep each matching section to 10 items or less • Standardize the administration procedure for the
examination
Short Answer and Essay
Short answer and essay questions are much more challenging
Feedback
for the student. These questions require the student to produce Many assessments are online, making the scoring and administra-
the correct answer, rather than recognize it in a list of responses tion consistent, unbiased, and reportable. It is also easy to include
which may trigger memory. Providing a correct response to a short feedback.
answer and essay question requires a thorough understanding of As part of the training assessment, it is a best practice to:
the subject matter. These questions can also incorporate more
than one concept and prompt students to outline relationships • Test students for understanding of the material
between concepts. • Offer evaluative feedback to incorrect responses
However, these question types are time consuming to score. The • Provide reinforcement feedback to correct responses
scoring process for open-ended questions is also subjective. For
partially correct responses or those that do not state the correct
answer in its entirety, the scorer’s judgement must be relied on.
This challenge makes short answer and essay questions difficult
for many types of tests, particularly those that do not test writing
ability.

Constructing Tests
Once you’ve created individual test questions to accurately measure
singular concepts, the next step is to combine these questions into
a comprehensive test.
Content validity is extremely important when constructing a test.
As with individual questions, you should build your test based on
your curriculum and learning objectives. Learning objectives tell
students what they will be tested on. All questions must map back
to the learning objectives to maintain content validity.

To make sure you are evaluating students’ skills and

knowledge, follow the steps below:
1. Tie the task to be trained to the learning objective which states the behavior to achieve
2. Tie the content to the learning objective
3. Tie the test question to the learning objectives
Remember, the more complex the task, the more difficult the question.
Putting all of this together enables you to write test questions and group these questions into a test so students who have
mastered the subject matter consistently achieve a passing score.
Continued...

WRITING LEARNING OBJECTIVES | SOS 4

Question Example:
Using procedures and learning objectives from our first two white papers, we developed an example test question.
Task: Direct transmission switching.
Learning objectives: Given a set of data, the student will be able to direct transmission switching without error.
Question:
Display a scenario of switching options for the student.
Based on the data available, which transmission equipment could safely be tagged out while
maintaining adequate generation to load ratios?
Select the best response.
a. The Kilarney line
b. The Queen line
c. The Baker line
d. The Charlie line
Feedback:
Incorrect. Charlie line is the safest and best choice. Selecting any of the alternates, especially the Queen line, would
impact frequency on the BES, eventually causing an island.

If you are a NERC continuing education provider, NERC’s Continuing Education Administrative Manual Version 4.4 states in section
4.9 – “The learning assessment measures the extent to which the training achieved its objectives and improved the learner’s
knowledge or skills.”
The NERC manual includes guidelines for writing assessments. NERC requires methods and instruments to allow individual
students to demonstrate mastery of the learning objectives. This evaluation should determine the degree to which learning occurred
for each learning objective. Assessments may be in the form of written tests, the successful completion of exercises, or the
successful completion of problem-solving activities such as simulations. The assessment type must be specified at the beginning
of the learning activity.
Qualified personnel or subject matter experts must be involved in planning and conducting each learning activity. Requirements for
the assessments are in the NERC Continuing Education Administrative Manual Version 4.4. To meet these requirements, a provider
must:
• Include a specified passing score that represents the • Only allow open book assessments if the learner is
level of knowledge required to successfully accomplish required to locate, not recall, information
the objectives • Include the method used to ensure assessment integrity,
• Base assessments entirely on the subject matter or such as a proctor. The provider must be able to prove
content delivered the person awarded the CEHs actually did the work
• Directly relate questions to the stated learning objective • Elicit a participant response to the test for understanding
• Include the criteria for and evidence of individual • Provide reinforcement feedback for correct answers
achievement of learning objectives and evaluative feedback for incorrect answers
• Ensure knowledge and performance components are • Have a review process to ensure material is continuously
assessed separately updated and NERC program criteria are met

Make your training and

compliance programs all they
can be with SOS solutions
Contact:
[Link]@[Link]
704.815.7906

[Link]
[Link] ( 877.767.4685 )

WRITING LEARNING OBJECTIVES | SOS 5