What is Utility?

Utility Analysis

Utility Utility Analysis

➢
Index of Utility

➢
→

FACTORS THAT AFFECT A TEST’S UTILITY →

Psychometric Soundness
→

→
→

→
→

➢
Cost

➢ Economic
HOW IS A UTILITY ANALYSIS CONDUCTED?
➢
➢ Noneconomic
Expectancy Data

Benefits

→
➢

Taylor-Russell Tables

➢
→
 Test’s Validity
➢
concurrent validation
Selection Ratio procedures

Brogden-Cronbach-Gleser Utility Formula

E.g., If there are 50 positions and 100
applicants, then the selection ratio is 50/100,
or .50.
Base Rate ➢

E.g., If a firm employs 25 computer ➢

programmers and 20 are considered
successful, the base rate would be .80.
→

Naylor-Shine Tables

→
𝐒𝐚𝐯𝐢𝐧𝐠 = (n)(t)(r)(SDy )(m)
𝐂𝐨𝐬𝐭 𝐨𝐟 𝐭𝐞𝐬𝐭𝐢𝐧𝐠 = (# of applicants × cost per applicant)
Savings – Cost of Testing
E.g., Suppose we hire 10 auditors per year, the average person in
→ this position stays two years, the validity coefficient is .30, and the
average annual salary for the position is $30,000, and we have 50
applicants for 10 openings. Thus,
n = 10; t = 2; r = .30; SDy = $30,000 x .40 = $12,000
m = 10/50 = .20 = 1.40 (converted using standardized table)
➢ cost of testing = 50 applicants x $10

(10) (2) (.30) (12,000) (1.40) – (50) (10) = $100,300 of savings

Decision Theory and Test Utility SOME PRACTICAL CONSIDERATIONS

The Pool of Job Applicants.

The Complexity of the Job.

Base Rate

The Cut Score in Use.

Hit Rate
Reference Point or Cut Score

→ Relative or Norm-Referenced Cute Score

→ Fixed or Absolute Cut Score

Miss Rate
➢

→ False Negative Multiple Cut Scores

→ False Positive

Multistage or Multiple Hurdle

→

→ →
➢

→
Compensatory Model of Selection IRT-BASED METHODS

IRT-Based Methods

→
➢

➢
Methods for Setting Cut Scores
➢
ANGOFF METHOD

Angoff Method

Item-Mapping Method

→ →
→

→
→

→
Disadvantages.

KNOWN GROUPS METHOD

Bookmark Method
Known Groups Method or Method of Contrasting Groups

Disadvantage.
→
 OTHER METHODS

Method of Predictive Yield

Discriminant Analysis