18/03/2025
WHAT IS UTILITY?
• usefulness or practical value of testing to improve efficiency
Factors That Affect a Test’s Utility
UTILITY Psychometric soundness
reliability and validity of a test
Lesson 11, Psychological Assessment index of utility can tell us something about the PRACTICAL
VALUE of the information derived from scores on the test
Test scores are said to have utility if their use in a particular
situation helps us to make better decisions
WHAT IS UTILITY? WHAT IS UTILITY?
Factors That Affect a Test’s Utility Factors That Affect a Test’s Utility
Psychometric soundness Cost
A test may be PSYCHOMETRICALLY SOUND, it may have economic, financial, or budget-related
little utility— particularly if the targeted testtakers demonstrate a one of the most basic elements in any utility analysis is the
tendency to “bend, fold, spindle, mutilate, destroy, tamper FINANCIAL COST of the selection device
with,” or otherwise fail to scrupulously follow the test’s directions.
COST in the context of test utility refers to disadvantages,
losses, or expenses in both economic and noneconomic terms
1
18/03/2025
WHAT IS UTILITY? WHAT IS UTILITY?
Factors That Affect a Test’s Utility
Factors That Affect a Test’s Utility Cost
Cost (1) payment to professional personnel and staff associated with test
expenditures associated with testing or not testing administration, scoring, and interpretation, (2) facility rental,
mortgage, and/or other charges related to the usage of the test
allocate funds to purchase (1) a particular test, (2) a supply of blank facility, and (3) insurance, legal, accounting, licensing, and other
test protocols, and (3) computerized test processing, scoring, and routine costs of doing business
interpretation
WHAT IS UTILITY? WHAT IS UTILITY?
Factors That Affect a Test’s Utility Factors That Affect a Test’s Utility
Cost Benefits
Non Economic Costs: pain, suffering, inconvenience, emotional whether the benefits of testing justify the costs of administering,
distress, loss of companionship,loss of enjoyment of life scoring, and interpreting the test
BENEFIT refers to profits, gains, or advantages
thoughtfully designed and well-run testing programs
a) an increase in the quality of workers’ performance;
2
18/03/2025
WHAT IS UTILITY? WHAT IS UTILITY ANALYSIS?
Factors That Affect a Test’s Utility UTILITY ANALYSIS may be broadly defined as a family of
Benefits techniques that entail a COST–BENEFIT ANALYSIS designed
to yield information relevant to a decision about the usefulness
b) an increase in the quantity of workers’ performance
and/or practical value of a tool of assessment
c) a decrease in the time needed to train workers
d) a reduction in the number of accidents
UTILITY ANALYSIS may be undertaken for the purpose of
e) a reduction in worker turnover. evaluating whether the benefits of using a test (or training program
Admission Test: Graduates or intervention) outweigh the costs.
Diagnostic Test: Responsive Interventions
WHAT IS UTILITY ANALYSIS? WHAT IS UTILITY ANALYSIS?
If undertaken to evaluate a test, the utility analysis will help make If undertaken to evaluate a test, the utility analysis will help make
decisions regarding whether: decisions regarding whether:
one test is preferable to another test for use for a specific purpose the addition of one or more tests (or other tools of assessment) to
one or more tests (or other tools of assessment) that are already in
use is preferable for a specific purpose
one tool of assessment (such as a test) is preferable to another tool
of assessment (such as behavioral observation) for a specific purpos
no testing or assessment is preferable to any testing or assessment
3
18/03/2025
HOW IS A UTILITY ANALYSIS HOW IS A UTILITY ANALYSIS
CONDUCTED? CONDUCTED?
EXPECTANCY DATA
EXAMPLE: A school psychologist uses an expectancy table to
EXPECTANCY DATA determine the likelihood that students who score within a particular
An expectancy table can provide an indication of the likelihood range on an aptitude test will succeed in regular classes as opposed
that a testtaker will score within some interval of scores on a to special education classes.
criterion measure—an interval that may be categorized as “passing,”
“acceptable,” or “failing.” Likelihood that individuals who score within a given range on the
predictor will perform successfully on the criterion
HOW IS A UTILITY ANALYSIS HOW IS A UTILITY ANALYSIS
CONDUCTED? CONDUCTED?
EXPECTANCY DATA TAYLOR-RUSSELL TABLES
Taylor-Russell tables provide an estimate of the extent to which
inclusion of a particular test in the selection system will
improve selection.
New Test Existing Test or Tool Improved Selection
4
18/03/2025
HOW IS A UTILITY ANALYSIS HOW IS A UTILITY ANALYSIS
CONDUCTED? CONDUCTED?
Original Base Rate of .60, meaning that 60% of those hired under the
existing system are successful in their work.
New Test: validity coefficient of .55
TAYLOR-RUSSELL TABLES Selection Ratio: 0.20
provide an estimate of the percentage of employees hired by New Base Rate: 0.88.
the use of a particular test who will be successful at their jobs, Selection Ratio: 5% or 0.05
(the test’s validity, the selection ratio used, and the base rate) New Tests: validity coefficient of .15
New Base Rate: ???
THE CUT SCORE IN USE
CUT SCORE as a (usually numerical) reference point derived
as a result of a judgment and used to divide a set of data into
two or more classifications.
5
18/03/2025
THE CUT SCORE IN USE THE CUT SCORE IN USE
RELATIVE CUT SCORE (NORM-REFERENCED CUT
SCORE) may be defined as a reference point—in a distribution of
test scores used to divide a set of data into two or more FIXED CUT SCORE is a pre-determined, minimum
classifications—that is set based on norm-related passing score for a test or assessment, independent of the
considerations rather than on the relationship of test scores to a performance of other test-takers, used to determine whether a
criterion. candidate passes or fails.
THE CUT SCORE IN USE THE CUT SCORE IN USE
MULTIPLE CUT SCORES refer to the use of two or more cut MULTISTAGE (OR MULTIPLE HURDLE) method
scores with reference to one predictor for the purpose of involves a series of sequential selection steps, where candidates
categorizing testtakers. must pass each "hurdle" (or stage) to advance to the next, with
failure at any stage resulting in disqualification.
class member will be assigned to one category (e.g.,A, B, C, D, or F)
6
18/03/2025
YES = 1
METHODS FOR SETTING CUT SCORES METHODS FOR SETTING CUT SCORES
NO = 0
THE ANGOFF METHOD THE ANGOFF METHOD
setting fixed cut scores can be applied to personnel selection TASKS the judgments of the experts are averaged to yield cut scores for
as well as to QUESTIONS regarding the presence or absence of a the test
particular trait, attribute, or ability Item SME 1 SME 2 SME 3
Number
relating to the determination of whether or not testtakers possess 1 1 0 1 2/3 = 0.66
a particular trait, attribute, or ability, an EXPERT PANEL MAKES 2 1 1 1 3/3 = 1.00
3 0 0 1 1/3 = 0.33
JUDGMENTS concerning the way a person with that trait,
4 0 0 0 0/3 = 0.00
attribute, or ability would respond to test items.
5 1 1 1 3/3 = 1.00
3/5 = 0.60 2/5 = 0.40 4/5 = 0.80 9/15 = 0.60
METHODS FOR SETTING CUT SCORES METHODS FOR SETTING CUT SCORES
THE KNOWN GROUPS METHOD THE KNOWN GROUPS METHOD
administered to all incoming freshmen
Collection of data on the predictor of interest from groups
known to possess, and not to possess, a trait, attribute, or Analyzes the scores: a group who passed the course and
ability of interest. earned credit, and a group who did not earn credit for the
“Who Needs to Take Remedial Math? Test” (WNTRMT) course because their final grade was a D or an F
setting a cut score based on the performance of two score at the point of least difference between the two
contrasting groups: (1) students who successfully completed groups
college-level Math, and (2) students who failed college-level
math
7
18/03/2025
METHODS FOR SETTING CUT SCORES METHODS FOR SETTING CUT SCORES
IRT-BASED METHODS
IRT framework, each item is associated with a particular
level of difficulty
“Pass” --- must answer items (above minimum level of
difficulty,) --- experts and serves as the cut score
METHODS FOR SETTING CUT SCORES METHODS FOR SETTING CUT SCORES
ITEM-MAPPING METHOD BOOKMARK METHOD
arrangement of items in a histogram begins with the training of experts with regard to the
judges who have been trained regarding minimal minimal knowledge, skills, and/or abilities that testtakers
competence required for licensure are presented with should possess in order to “pass”
sample items from each column
asked whether or not a minimally competent licensed the experts are given a book of items, with one item
individual would answer those items correctly about half the printed per page, such that items are arranged in an
time ascending order of difficulty
8
18/03/2025
METHODS FOR SETTING CUT SCORES
BOOKMARK METHOD
the expert then places a “bookmark” between the two
pages (or, the two items) that are deemed to separate
testtakers who have acquired the minimal knowledge, skills,
and/or abilities from those who have not.
Additional rounds of bookmarking with the same or other
judges may take place as necessary.