+
How to make tests more valid
Write explicit specifications
Use direct testing
Relate the scoring to what is being
tested
Make the test reliable
+
CHAPTER 5:
RELIABILITY
+ Take a look on two scores below!
Which one is more reliable?
Note the size of difference between the two scores
for each student.
Scores on Test A Scores on Test B
Reliability refers to the consistency of
the scores obtained
It means that if the test is administered to
the same students on different occasions
(with no language practice work between
these occasions) then it produces (almost)
the same results.
Reliability actually does not really deal
with the test itself. It deals with the results
of the test. The test results should be
consistent.
Student-
related
Reliability
Test Kinds of Scorer
Reliability
reliability Reliability
Test
administration
Reliability
+
Student-Related
Reliability
refers to temporary illness, fatigue, a bad day,
anxiety and other physical or psychological
factors of the students.
+ Scorer Reliability
When the test is When the test is
objective, the subjective, the
scoring requires scoring requires
no judgment, judgment, and
and the scores
should always be the scores will
the same. not be the same.
+ Test Administration Reliability
Test administration reliability concerns with the condition
and situation in which the test is administered.
To increase the degree of this kind of reliability test,
teachers as the administrators should consider all the
things related to the test administration.
+For instance, if we want to conduct a listening test, we
should provide a room which is…????
+
Test Reliability
Tests reliability refers to the test itself.
Whether the test fits into the time
constraints.
It means that the test should not be too long
or short.
The items of the test should be crystal clear
that it will not end with ambiguity.
+
HOW TO MAKE TESTS
MORE RELIABLE?
+ 1. Take enough samples
of behaviour
The
more items that you have on a test, the
more reliable that test will be.
Theadditional items should be independent of
each other and of existing items. -> each
additional item = a fresh start
+For example, in a test writing where candidates
have to produce a number of passages, each of
those passages is to be regarded as an item. The
more independent passages there are, the more
reliable will be the test.
+
+
+
+
+
+
+ 2. Exclude items which do not
discriminate well between
weaker and stronger students
Tooeasy or too difficult items for the
candidates
A small number of easy, non-discriminating
items may be kept at the beginning of a
test to give candidates confidence and
reduce the stress they feel
+ 3. Do not allow candidates
too much freedom
The more freedom that is given, the greater
is likely to be the difference between the
performance actually elicited
For example, in a test of writing the
candidates are simply given a selection of
titles to choose
W
+ rite an essay based on one of the
given titles below.
1. The importance of learning a foreign
language in today’s world
2. Describe a memorable event from your
childhood
3. Do you agree or disagree that technology
makes people less social?
+
4. Write unambiguous items
Avoid items whose meaning is not
clear or to which there is an
acceptable answer which the test
writer has not anticipated.
Question: What is the meaning of bank?
a) The side of a river
b) A place where you keep money
c) Both a and b
+
5. Provide clear and explicit instructions
“Read the text carefully. Then answer the
five multiple-choice questions that follow by
circling the letter (A, B, C, or D) of the
correct answer.”
6. Ensure that tests are well laid out and
perfectly legible
A test paper uses 12-point font, double spacing,
clear headings, and numbering so that all
questions are easy to read and no words are cut off
or overlapping.
+
7. Make candidates familiar with format and
testing techniques
Before the exam, students are given a
sample test with the same format (multiple
choice, matching, short answer) so they know
how to respond during the real test.
+
8. Provide uniform and non-distracting
conditions of administration
The test is conducted in a quiet room with good
lighting, all candidates start and finish at the
same time, and no one is disturbed by outside
noise or interruptions.
+
9. Use items that permit scoring
which is as objective as possible
Use multiple choice items
Use open-ended item which has a
unique, possibly one word, correct
response which the candidates
produce themselves
+For example: …… was more closely
associated with ……
+
+
10. Make comparisons between
candidates as direct as possible
Candidates should not be given a
choice of items
Scoring the compositions all on one
topic will be more reliable than if the
candidates are allowed to choose
from 6 topics
+ [Link] a detailed scoring key
+
[Link] a detailed scoring key
[Link] scorers
[Link] acceptable responses and
appropriate scores at outset of scoring
+
+
[Link] a detailed scoring key
[Link] scorers
[Link] acceptable responses and
appropriate scores at outset of scoring
[Link] candidates by number not name
[Link] multiple, independent scoring:
where testing is subjective, all scripts
should be scored by at least 2
independent scorers
+
Reliability and Validity
+
Reliability and Validity
WHY ?
A reliable test is not always valid
and vice versa.