UTILITY

1
 WHAT IS UTILITY?

utility refers to the practical value or usefulness of a

test or assessment program. It measures how
effectively a test or assessment helps achieve a
specific goal.

2
 FACTORS THAT AFFECT A
TEST'S UTILITY

Psychometric
Cost Benefits
Soundness

3
 PSYCHOMETRIC SOUNDNESS

This refers to the technical quality of the

test. A psychometrically sound test is
reliable and valid.

4
 COST

Economic costs encompass direct expenses

like test materials, processing fees, personnel
salaries, facility usage, and administrative
overhead.

5
 BENEFITS

Evaluating test utility involves comparing the costs of testing

against its benefits. These benefits can be economic (e.g.,
increased profits) or non-economic (e.g., improved decision-
making), and are weighed against the costs to determine the
overall value of the test.

6
 WHAT IS UTILITY
ANALYSIS?

A utility analysis is a broad term encompassing various

methods for evaluating the cost-effectiveness of assessment
tools (tests, training programs, or interventions). It helps
determine if the benefits outweigh the costs, guiding decisions
on which tool or approach is optimal for a specific purpose.

7
 IF UNDERTAKEN TO EVALUATE A TEST, THE
UTILITY ANALYSIS WILL HELP MAKE DECISIONS
REGARDING WHETHER:

one test is preferable to another test for use for a specific purpose;
one tool of assessment (such as a test) is preferable to another tool of
assessment (such as behavioral observation) for a specific purpose;
the addition of one or more tests (or other tools of assessment) to one
or more tests (or other tools of assessment) that are already in use is
preferable for a specific purpose;
no testing or assessment is preferable to any testing or assessment.
8
 IF UNDERTAKEN FOR THE PURPOSE OF
EVALUATING A TRAINING PROGRAM OR
INTERVENTION, THE UTILITY ANALYSIS WILL HELP
MAKE DECISIONS REGARDING WHETHER:

one training program is preferable to another training program;

one method of intervention is preferable to another method of
intervention;
the addition or subtraction of elements to an existing training
program improves the overall training program by making it
more effective and efficient;

9
the addition or subtraction of elements to an existing
method of intervention improves the overall
intervention by making it more effective and
efficient;
no training program is preferable to a given training
program;
no intervention is preferable to a given intervention.
 HOW IS A UTILITY
ANALYSIS CONDUCTED?

The specific objective of a utility analysis will

dictate what sort of information will be
required as well as the specific methods to be
used studies.
11
TWO GENERAL APPROACHES TO
UTILITY ANALYSIS

Brogden-Cronbach-
Expectancy Data
Gleser (BCG)
Approach Formula
 EXPECTANCY DATA

This approach uses data on the expected outcomes or

consequences of using an assessment. It involves gathering
information on how well the assessment predicts future
performance or outcomes.
Taylor-Russell Tables – Estimate the increase in the base rate
of successful performance when using a test.
Naylor-Shine Tables – Estimate the likely average increase in
performance due to using a test.
TABLE 1: DECISION THEORY TERMS
What It Means in
Term General Definition Implication
This Study

Hit A test score A passing score on the A qualified driver is

correctly identifies FERT is associated hired.
a condition of with satisfactory
interest. performance during
training.

Miss The test score fails A failing score on the A qualified driver is
to identify a trait FERT is associated with not hired.
when it exists. satisfactory
performance during
training.
14
False Alarm The test score A passing score on the FERT An unqualified
incorrectly identifies is associated with driver is hired.
the condition of unsatisfactory performance
interest. during training.

Correct The test score A failing score on the FERT is An unqualified

Rejection correctly identifies associated with driver is not
the absence of the unsatisfactory performance hired.
condition of interest. during training.

Sensitivity (Hit If a person has the Among drivers with The proportion
Rate) condition of interest, satisfactory performance of qualified
what is the probability during training, what drivers who
that the test will proportion had passing would be hired
correctly indicate the scores on the FERT? based on
condition? passing scores
15
on the FERT.
Specificity (True If a person lacks a Among drivers with The proportion
Negative Rate) condition, what is the unsatisfactory performance of unqualified
probability the test during training, what drivers who
will correctly indicate proportion had failing would not be
the condition is scores on the FERT? hired based on
absent? failing scores
on the FERT.

Positive If a test score Among drivers with passing The proportion

Predictive Value indicates the scores on the FERT, what of people hired
presence of a proportion of them had based solely on
condition, what is the satisfactory performance passing scores
probability the during the training period? on the FERT
condition is truly that would turn
present? out to be
qualified drivers
16
after training.
Negative If a test score indicates Among drivers with The proportion of
Predictive the absence of a failing scores on the people not hired
Value condition, what is the FERT, what proportion of based solely on
probability that the them had unsatisfactory failing scores on the
condition is truly performance during the FERT that would have
absent? training period? turned out to be
unqualified drivers
after training.

Base Rate The proportion of The proportion of drivers The proportion of

(Prevalenc individuals with the with satisfactory drivers who would
e) condition of interest. performance during have satisfactory
training. performance during
training if employees
were chosen at
random 17
Selection The proportion of The proportion of drivers The drivers who
Ratio individuals with test with passing scores on would be hired based
scores indicating the the FERT. on FERT scores.
presence of the
condition of interest.

Overall The proportion of The proportion of drivers The proportion of

Accuracy decisions that are with either passing scores correct decisions the
correct (i.e., true on the FERT and FERT allows in
positives and true satisfactory performance employee selection.
negatives). during training or failing
scores on the FERT and
unsatisfactory
performance during
training. 18
(1) Limit the cost of selection by not using the
FERT.
(2) Ensure that qualified candidates are not
rejected.
(3) Ensure that all candidates selected will prove to
be qualified.
(4) Ensure, to the extent possible, that qualified
candidates will be selected and unqualified
candidates will be rejected.
MOST EVERYTHING YOU EVER WANTED TO KNOW ABOUT
UTILITY TABLES

Instrument What It Tells Us Advantages Disadvantages

Expectancy table or Likehood that Easy-to-use Unrealistically

chart individuals who graphical display dichotomizes
score within a Can aid in decision performance into
given rage on the making regarding unsuccessful
a specific categories
predictor will
individual or a Does not address
perform
group of monetary issues
successfully on the
individuals scoring (i.e. cost of testing
criterion in a given range on investment of
the predictor testing) 20
Taylor- Increase in base Easy-to-use Requires linear
Russel rate of successful Shows the relationship between,
tables performance that is relationships predictor and criterion
associated with a between selection Does not indicate the likely
particular level of ratio, criterion- average increase in
related validity, performance with use of
criterion-related
and existing base the test
validity
rate Difficulty identifying a
Facilitates decision criterion value to separate
making with successful and
regard to test use unsuccessful performance
Unrealistically
and/or recruitment
dichotomizes performance
to lower the
into successful-versus
selection ratio
unsuccessful
Does not consider the cost
of testing in comparison to
21
benefits.
Naylor- Likely average increase in Provides information (or, Overestimates
Shine criterion performance as average performance gain) utility unless top-
tables a result of using a needed to use the Brogden- down selection
particular test or Cronbach-Gleser utility Utility expressed
intervention; also formula in terms of
provides selection ratio Does not dichotomize performance gain
needed to achieve a criterion performance based on
particular increase in Useful either for showing standardized
criterion performance average performance gain units, which can
or show selection ratio be difficult to
needed for particular interpret in
performance gain practical terms
Facilitates decision making Does not address
with regard to likely monetary issues
increase in performance (i.g., cost testing
with test use and/or or return on
recruitment needed to investment)
22
lower the selection ratio
 BROGDEN-CRONBACH-GLESER
FORMULA

Used to calculate the dollar amount of a utility

gain resulting from the use of a particular
selection instrument under specified conditions.
 UTILITY GAIN

Utility gain refers to an estimate of the benefit

(monetary or otherwise) of using a particular test
or selection method.
Utility Gain=(N)(T)(rxy​)(SDy​)(Zm​)−(N)(C)
N = Number of people hired
T = Average tenure of employees (in years)
rxy = Validity coefficient of the test
SDy = Standard deviation of job performance in monetary
terms
Zm = Average test score of the selected group (in standard
deviation units)
C = Cost per applicant for using the test
 STEP-BY-STEP CALCULATION

Step 1: Gather the necessary data

Step 2: Substitute the given values into the
formula
Step 3: Calculate the benefit of using the test
Step 4: Subtract the cost of the test
Step 5: Compute the Final Utility Gain
PhilTech Industries, a manufacturing company, is considering enhancing its
hiring process by incorporating a cognitive ability test to better predict job
performance. Currently, the company selects employees based on
interviews alone, but they want to evaluate if adding the test would be
financially worthwhile. Each year, PhilTech plans to hire 100 new employees,
and these employees typically stay with the company for an average of five
years. The cognitive ability test being considered has a validity coefficient of
0.5, indicating it is moderately effective in predicting how well employees
will perform on the job. The company estimates that the variation in
employee productivity, measured in terms of money, is $10,000 per year.
Additionally, candidates selected through the test tend to score 1.0 standard
deviation above the average in their performance. The cost of administering
the test is $500 per applicant. PhilTech wants to calculate whether the
financial benefits of using this test outweigh the costs involved.
 PRODUCTIVITY GAIN

Refers to an estimated increase in work output. The

result is a formula that helps estimate the percent
increase in output expected through the use of a
particular test.
Revised formula:
Productivity Gain=(N)(T)(rxy​)(SDp​)(Zm​)−(N)(C)
TechNova Solutions, a software development company, is
considering using a cognitive ability test to improve its hiring
process and employee productivity. By hiring 100 new developers
each year with an average tenure of 5 years, the company expects
the test to predict job performance effectively, given its validity
coefficient of 0.7 and a standard deviation of productivity at
$20,000. After calculating the productivity gain using the formula,
the company estimates a productivity increase of $10,500,000,
minus the cost of administering the test ($80,000), resulting in a net
gain of $10,420,000 annually. This indicates that the test would
significantly enhance work output and be a valuable investment for
TechNova.
Some Practical
Considerations

30
 A number of practical matters must be considered
when conducting utility analyses.
1. The pool of job applicants- refers to the total number
of individuals applying for a given position.
2. Complexity of the job-The complexity of a job refers
to the degree of skill, knowledge, and cognitive ability
required to perform tasks effectively.
3. Cut Score in Use – A reference point in a test score
distribution that determines classifications. It can be:
Relative Cut Score – Based on norm-related
considerations rather than direct criteria.
Fixed Cut Score – Based on a minimum required
level of proficiency for a classification.
 Multiple cut scores– refers to the use of two or more cut
scores with reference to one predictor for the purpose of
categorizing testtakers.
Multiple hurdle- the cut score used for each predictor
will be designed to ensure that each applicant possess
some minimum level of a specific attribute or skill.
Compensatory Model of Selection- A model where high
scores in one area can balance out lower scores in
another, allowing individuals with different strengths to
perform at a similar level.
METHODS FOR SETTING CUT
SCORES

Known Groups
Angoff Method
Method

Item Response
Theory (IRT) Other Methods
Benefits
Methods
34
 THE ANGOFF METHOD
A method where experts estimate the probability
that minimally competent candidates will answer
each test item correctly.
 KNOWN GROUPS METHOD

A method that collects test data from groups known to

possess and not possess a specific trait, then sets a cut score
that best differentiates the two groups.
 ITEM RESPONSE THEORY (IRT) METHODS

Methods based on advanced statistical models that analyze

the difficulty level of individual test items.
Types of IRT-Based Methods:
Item Mapping Method- Organizes items into a
histogram based on difficulty levels.
Bookmark Method- Experts rank items in
ascending order of difficulty and place a
"bookmark" at the point where minimal
competency is achieved.
 Other Methods

Predictive Yield Method

A technique for setting cut scores which took into account the
number of positions to be filled, projections regarding the
likelihood of offer acceptance, and the distribution of applicant
scores.
 Discriminant Analysis
A statistical method that determines the cut score
by analyzing the relationship between test scores
and performance outcomes.
Concept of Utility

41
 Applications of Utility

Decision-Making- Examines how individuals

make choices under uncertainty and risk.
Behavioral Economics- Analyzes economic
decisions influenced by cognitive biases and
emotions.
Health Psychology- Explores health-related
decisions, such as treatment adherence and
health-promoting behaviors.
Clinical Psychology- Assesses subjective well-
being and quality of life based on personal values
and preferences.
 Applications of Utility in Psychological
Testing
Clinical Assessments- Determine the effectiveness of diagnostic
tools in identifying mental health conditions and aiding treatment
planning..
Educational Testing- Evaluate standardized tests used to
measure intelligence, learning disabilities, and academic
aptitude.
Employment Testing: Assesses the value of aptitude and
personality tests in predicting job performance.
Industrial-Organizational Psychology
(Employment Testing)- Assesses health behaviors,
stress levels, and adherence to treatments to
guide interventions and improve health outcomes.
Health Psychology- Measures health behaviors,
stress, and adherence to treatments to promote
better health outcomes.

45
 Challenges in Ensuring High Utility in
Psychological Testing
Test Misuse or Misinterpretation
•IIf tests are used improperly (e.g., diagnosing without professional oversight),
their utility decreases.
Cultural and Contextual Limitations
•Some tests may not be valid across different cultural groups, reducing their
effectiveness.
Ethical Concerns and Biases
•Ethical issues (e.g., privacy, consent) must be considered to maintain a test's
credibility and fairness.
Technological Changes
•With Al and online assessments becoming more common, ensuring test
security and accuracy is a growing concern. 46
THANK YOU

47