EDUCATIONAL ASSESSMENT AND

EXAMINATIONS
SERVICE

Examination Development
Presented to: Regional Education Beauro Head Exam and Developers

September, 2017/2025
Adama, Ethiopia

Presented by: (Abraraw Teka From National Examination Executive)

 INTRODUCTION
• Examination is a formal test that is used to measure the knowledge
and ability of students in a particular subject.
• Item development is a careful, multi- step process that begins once
the test purpose has been defined.
• Effective exam development requires a systematic, detail-oriented
approach based on sound theoretical educational measurement principles.
• This discussion presents the steps of the art of developing exams in the
twenty-first century.
• The purpose of this presentation is to present
1. sound steps required to construct tests that have sufficient
validity evidence.
2. Important tips about item construction.
 Item development Working Model in EAES

Item Writers Reviewers Editors

Exam development Experts ( EAES)

 Exam assembly
 Item writing
 Booklet
 Reviewing
 Randomization
 Editing
 Formatting
 Fourteen Steps for Effective Exam Development
[Link] plan
2. Content definition
• These steps are listed as a
3. Exam/Test specifications
linear model or as a 4. Item development
sequential timeline, from 5. Test design and assembly
a discrete beginning to a 6. Test production
final end point. 7. Pilot exam/Tryout
• In practice, many of 8. Item banking
these activities may occur 9. Selecting items for large scale examination
simultaneously or the 10. Exam administration
order of some of these 11. Scoring test responses
steps may be modified. 12. Passing scores
13. Reporting test results
14. Exam technical report
 STEP 1: OVERALL PLAN
• Systematic guidance for all exam development activities.
• Every exam program needs some type of overall plan.
• An exam well begun is often an exam well ended.

What is to be measured?
Fundamental What is the clear purpose of the exam?
Who creates the items / writer, reviewer, editor?
decisions to be What exam format is most appropriate?
included in When/how the exam is administrated /who?
this critical What is security and quality control mechanism?
Who maintains the item bank/pool?
beginning What score interpretations/analysis are desired?
stage Who scores the exam?
How the scores reported?
 STEP 2: CONTENT DEFINITION
• One of the most important questions to be answered in the earliest
stages of test development is: What content and cognitive domain is
to be tested?
• Defining the content domain to be sampled by the examination is a
critical issue.
• Content defining methods vary depending
on the purpose of the exam
the consequences of decisions made from the resulting exam scores
the amount of defensibility required for any decisions resulting from exam
scores
• Content Defining must be systematic, comprehensive and defensible.
Because it is ultimately a matter of human judgment.
STEP 3: EXAM SPECIFICATIONS: BLUEPRINTING THE EXAM
• The process of creating exam specifications guides detailed exam development
activities.
• It completes the operational planning for tests in a systematic manner.
• An exam blueprint defines and precisely outlines the number (proportion) of
test questions to be
1. allocated to each major and minor content area.
2. designed to assess specific cognitive levels.
 Period allotment, MLC, Importance of content, expert judgment

The total number and format of exam items

The exam  The cognitive classification system to be used
 Whether or not the exam items will contain visual stimuli
specification  The expected item scoring rules
can describe  How exam scores will be interpreted?
 The time limit for each item
 STEP 4: ITEM DEVELOPMENT

• Creating effective exam items is both an art and science.

• The most essential quality of an effective item writer is content expertise.
• The creation and production of effective questions, designed to measure
important content at an appropriate cognitive level, is one of the greater
challenges for exam developers.
• The principles of writing effective items are well established and many of
these principles have a solid basis in the research literature.
• Yet, knowing the principles of effective item writing is no guarantee of an
item writer’s ability to actually produce effective test questions. Knowing is
not necessarily doing.
• Thus, one of the more important validity issues associated with exam
development concerns the selection and training of item writers.
• Writing ability is also a trait closely associated with the best and most
creative item writers.
 Item Writer Training
• Effective item writing is a unique skill and must be learned and practiced.
• Without specific training, most novice item writers tend to create poor-quality,
flawed(defective), low-cognitive-level test questions that test unimportant or trivial
content.
• Although item writers must be expert in their own disciplines, there is no reason to
believe that their subject matter expertise generalizes to effective item writing
expertise.
• For new item writers, it is often helpful and important to provide specific
instruction using an item writer’s guide, paired with a hands-on training
workshop.
• As with all skill learning, feedback from expert item writers and peers is required.
• The instruction practice – feedback – reinforcement loop is important for the
effective development and maintenance of solid item writing skills.
Important source of  Training of item writers
validity evidence  Competent content review
 Professional reviewing and editing of items
Item Writers Training Cont….
• Professional reviewing and editing of items
 increases the accuracy and clarity of the questions.
 corrects flaws in item content.
 reduce cultural, ethnic, religious, or gender sensitivities.
 find content errors or inconsistencies.
• Poor quality and flawed exam items potentially decrease student passing rates
by increasing the mean item difficulty.
• Ideally, all item writers with any responsibility for achievement assessment have
some special expertise in item writing and test development, gained through
effective training and practice.
• Unfortunately, this is too often not the case and is one of the major failings of
educational assessment at all levels of education.
STEP 5: EXAM DESIGN AND ASSEMBLY
• Assembling a collection of exam items into a complete exam form.
• The validity of the final exam score interpretation very much relies on the
competent and accurate exam assembly process.
• The specific method and process of assembling test items into final exam forms
depends on the mode of examination delivery.
• If a single exam form is to be administered in paper-and-pencil mode, the
exam can be assembled manually, by skilled exam developers (perhaps using
computer software to assist manual item selection and assembly).
• The most important validity-related issues in exam assembly are the
correspondence of the content actually tested to the content specifications and
the high-level quality control of this entire process. Where as;
• Formatting issues are more psychometric in nature.
STEP 6: EXAM PRODUCTION
• The production, printing, or publication of examinations
Security  exam production.
precautions must  shipping of printed exam copy and printed booklets.
be taken during  destruction of excess printed materials.

• Exam production security standards and policies must be developed and implemented.
• Independent audits of these security procedures should be carried out periodically, by security
professionals.
• All secure exam materials must be locked in limited-access files at all times.
• If an exam is being printed by a printing company, exam development staff (in addition to
printer’s staff) must take responsibility for many quality assurance procedures.

Items must be items or visual stimuli are unclearly printed.

invalidated and booklet pages are omitted.
omitted from final items are mis-ordered/with no key.
scoring if options are out of order/repeated.
Step 7. Field Testing/ Pilot Exam/ Tryout
• Despite rigorous procedures for development, review, and editing of test items, one cannot be sure
how items will work until they have been tried out with a sample of the intended population of exam
takers.
• For large-scale examinations, items should be pretested by administering to a group of exam takers
like those who will take the exam, before the items are included in an actual form of the exam.
• The value of the tryout often depends on the adequacy of the sample.
• The sample(s) should be as representative of the population(s) for which the exam is intended.
• The information provided by item analysis helps test developers to
 Identifyand correct any flawed items before they are included in an
operational version of the test
Select the items to be accepted or rejected
 Identify items that need to be revised/ modified
 Serves as a quality-control step as last chance to catch errors
Learn what types of items tend to work well or not well in a particular
type of exam
 Things to be
First
Third considered in the
Tryout
Tryout tryout cycle
 Difficulty Level
 Discrimination
Power
 Effectiveness of
alternatives
Second
 Reliability
Tryout
 Validity

Write Review Edit Pilot Test Analysis

Validation Effective Items Item Bank
 Tips to construct Multiple Choice Questions
• A “multiple choice question” (MCQ) is a question in
which students are asked to select one alternative from a
given list of alternatives in response to a question.
• Advantages:
• Very effective
• Versatile/Useful at all levels
• Minimum of writing for student
• Guessing reduced
• Can cover broad range of content
Multiple choice tests can be used to test the ability to:

1. recall memorized information

2. apply theory to routine cases

3. apply theory to novel situations

4. use judgment in analyzing and evaluating

 •Difficult to construct
good test items
Disadvantages: •Difficult to come up
with plausible
distractors
 Anatomy of a Multiple-
Choice Item

A standard multiple-
choice test item • a problem (stem) and
consists of two basic • a list of suggested solutions (alternatives).
parts:

The stem may be in the • a question

form of either: • an incomplete statement,
 The list of • one correct or best alternative (answer) and
alternatives • a number of incorrect or inferior alternatives
contains: (distractors).

Example
What process is most nearly the opposite of photosynthesis?-----Stem
Distractor--- A. Digestion
Distractor--- B. Circulation
Key---------- C. Respiration alternatives
Distractor-- -D. Exertion
• When possible, state the stem as a direct question rather than as an incomplete
statement.
• Poor: Alloys are ordinarily produced by…
• Better: How are alloys ordinarily produced?
• Present a definite, explicit and singular question or problem in the stem.
• Poor: Psychology…
• Better: The science of mind and behavior is called…
• Eliminate excessive verbiage or irrelevant information from the stem.
• Poor: While ironing her formal, Jane burned her hand accidently on the hot iron. This
was due to a transfer of heat between...
• Better: Which of the following ways of heat transfer explains why Jane’s hand was
burned after she touched a hot iron?
• Include in the stem any word(s) that might otherwise be repeated in each alternative.

•Poor: In national elections in the United States the President is officially:

A. chosen by the people.
B. chosen by members of Congress.
C. chosen by the House of Representatives.
D. chosen by the Electoral College.
•Better: In national elections in the United States the President is officially chosen by
A. the people.
B. members of Congress.
C. the House of Reps.
D. the Electoral college.

• Use negatively stated stems economically. When used, underline

and/or capitalize the negative word.
Poor: Which of the following is not cited as an accomplishment of the
Kennedy administration?
Better: Which of the following is NOT cited as an accomplishment of the
Kennedy administration?
• Make all alternatives plausible and attractive to the less knowledgeable or skillful
student.
Poor: What process is most nearly the opposite of photosynthesis?
A. Digestion
B. Assimilation
C. Respiration
D. Catabolism
Better: What process is most nearly the opposite of photosynthesis?
A. Digestion
B. Relaxation
C. Respiration
D. Exertion
• Make the alternatives mutually exclusive
Poor: The daily minimum required amount of milk that a 10 Better: What is the daily minimum required amount of
year old child should drink is milk a 10 year old child should drink?
A. 1-2 glasses. A. 1 glass.
B. 2-3 glasses. " B. 2 glasses.
C. 3-4 glasses. " C. 3 glasses. "
D. at least 4 glasses. D. 4 glasses.
 • Poor: The most general cause of low individual incomes in
the United States is:
Make • A. lack of valuable productive services to sell. "
• B. unwillingness to work.
alternatives • C. automation.
approximately • D. inflation.
equal in • Better: What is the most general cause of low individual
incomes in the United States?
length/Sequence • A. A lack of valuable productive services to sell. "
• B. The population’s overall unwillingness to work.
by length. • C. The nation’s increased reliance on automation.
• D. An increasing national level of inflation.
Summary of Effective Exam Development
Overall
Content Exam Item
plan
definition specifications development

Selecting
Exam
items for Item Pilot Exam design
large scale production
examination bank exam and
assembly

Exam Scoring exam Passing Reporting Exam technical

administration responses scores exam results report