Module 3: Signal Detection Theory
Module Overview
This module the students will learn about the Signal Detection Theory and
basic concepts such as hits, false alarms, criterion, d’, and ROC curves.
Motivation Question
Why do we need to study signa detection theory?
Module Pretest
This will be posted in the official virtual classroom.
Lesson 3.1: Signal Detection Theory
Lesson Summary
This lesson discusses the basic concepts of signal determination theory.
Learning Outcomes
At the end of the lesson, the students should be able to:
1. Discuss the signal detection theory.
2. compute the sensitivity and bias in a signal detection experiment/problem.
Motivational Questions
What can you say about the figure?
Photo take from:
[Link]
q=supplier+appraisal+cartoon&sxsrf=AOae
mvKeUVpgMYLn8VQ896SU1W1zJGLqfw:163
2800755190&source=lnms&tbm=isch&sa=
X&ved=2ahUKEwiu4oHo4KDzAhXSc3AKHZF
3DCYQ_AUoAXoECAEQAw&biw=1536&bih=7
04&dpr=1.25#imgrc=D_BcFdWLfB77cM
Discussion
While reading in a coffee shop,
you notice a familiar-looking person. Have you previously met him? You start
conversing with someone at the coffee shop a short while later, and they are
telling you an amazing story. Is she lying to you, or is the narrative true?
These are a few instances of detecting methods. Uncertainty over the
existence of a signal is a common aspect of these circumstances. In other
words, have you or have they met before? Is someone being truthful with
you, or are they lying? Should you approach the person who seems familiar
even if you could end up embarrassing yourself if you turn out to be wrong?
At the risk of upsetting this friend, should you pretend to ignore him? There
could be costs and benefits associated with each course of action, so it's
unclear which is the best choice. Additionally, your decision may be
influenced by past experiences that you have had. For example, you might be
less inclined to wave to someone who appears somewhat familiar if you have
ever been embarrassed when you unintentionally waved "Hello" to a
stranger.
Signal Detection Theory (SDT) is a method of modeling the decision-making
process for someone who decides between different classes of items (e.g., friend
or stranger) and their bias to favor a particular type of response. When working
with SDT, performance is described in terms of hit and false alarm rates. If a
signal is present and a person correctly identifies the signal, then she has made a
‘hit.’ However, if the signal is absent and she says that the signal is present, then
she has made a ‘false alarm.’
The underlying model of SDT consists of two normal distributions, one
representing a signal and another representing noise. In this tutorial, we refer to
the signal distribution as “Signal Present” and the noise distribution as “Signal
Absent.” How well a person can discriminate between Signal Present and Signal
Absent trials is represented by the difference between the means of the two
distributions, d’. The willingness of the person to say ‘Signal Present’ in response
to an ambiguous stimulus is represented by the criterion.
The logic of the SDT model is very similar to statistical hypothesis testing. The
Signal Absent distribution corresponds to the null hypothesized distribution, the
Signal Present is the alternative distribution, and the criterion is the alpha error
rate set by the analyst.
“Yes-No” paradigms
A research domain where SDT has been successfully applied is in the study of
memory. Typically, in memory experiments, participants are shown a list of
words and later asked to make a “yes” or “no” statement as to whether they
remember seeing an item before. Participants may respond “old” or “new” to
words they are shown. The outcome of each decision can be portrayed in what is
called a decision matrix.
Old New
Say “Old” Hit False Alarm
Say “New” Miss Correct Rejection
The hit rate is defined as the proportion of “old” responses given for items that
are Old and the false alarm rate is the proportion of “old” responses given to
items that are New.
Does this look familiar? In hypothesis testing, the same decision matrix would
have the following labels:
Ho False Ho True
Reject the Null
Hypothesis Correct Decision Type I Error
Fail to Reject the Null
Hypothesis Type II Error Correct Decision
Example 1:
If we wish to compare memory ability for two people, why can’t we simply use
the percent correct on identifying old items (i.e., hit rate) as the measure of
ability?
Consider two participants in a recognition memory test. Participant A has a hit
rate of .70 and Participant B has a hit rate of 0.75.
Table 1. Decision Matrix
Old New
Say “Old” Hit False Alarm
Say “New” Miss Correct Rejection
Can we be sure that B has a better memory than A? What if participant A had a
false alarm rate of .05 and participant B had a false alarm rate of .24? This
indicates that B is more willing to say “old.” ‘Would you still think that B is doing
better than A? If we look at only the hit rate, we overlook the fact that Participant
B might have a higher hit rate just because he is willing to say “old” more often,
and hence, he is getting more hits at the expense of more false alarms.
Participant Hit Rate False Alarm Rate
A .70 .05
B .75 .24
C 1.00 1.00
Participant C is an extreme case, with a hit rate of 1.00. He attains this rate by
saying “old” to every item. Great Job! Or is it? This performance wouldn’t really
say anything about his memory ability, however, because his false alarm rate
would also be 1.00. This performance wouldn’t demonstrate any discrimination
between old and new items.
In the following sections, we will learn how the SDT model uses the relationship
between hit and false alarm rates to compute separate measures of
discrimination ability and criterion (willingness to say “old”).
An Application to Memory
Below is a decision matrix filled in with the frequencies of each response for a
hypothetical memory experiment with 50 Old items and 50 New items.
Old New
Say “Old” 40 5
Say “New” 10 45
TOTAL 50 50
The hit rates.
The hit rate (say “Old” for old items) is 40/50 or as a proportion 0.80.
The false alarm rates.
The false alarm rate (say “Old” for new items) is 5/50 or 0.10.
Miss rate and rate of correction rejection
Note that Misses and Correct Rejections are redundant with Hits and False
Alarms. The miss rate is 10/50 which is .20 or simply (1 – “hit rate”) and the
Correct Rejection rate is 45/50 or .90 or (1 – “false alarm rate”). Therefore, you
can perfectly describe all four measures of a person’s performance in a signal
detection experiment through their Hit and False Alarm rates.
Before we continue, we should take a moment to review a few simple statistical
transformations that are necessary for SDT calculations.
In the model we are using here, SDT is based upon two normal distributions
whose variances are equal. To calculate SDT measures, we need to convert p-
values to z-scores, and vice versa. In SDT, a z-score measures performance in
terms of the number of standard deviations that the signal distribution is above
the noise distribution, and a p-value represents the probability of observing a
score greater than the observed score if we were sampling from the noise
distribution.
Table 2. Z-score table
Adapted from: Statistics: 1001 Practice Problems For Dummies (+ Free Online
Practice)
All the measures used to describe performance in SDT are derived from the
relationships between the Signal Present and Signal Absent distributions.
d’
The most used SDT measure of sensitivity is d’ (d prime), which is the
standardized difference between the means of the Signal Present and Signal
Absent distributions. To calculate d’, we need to know only a person’s hit rate
and false alarm rate.
The formula for d’ is as follows: d’ = z(FA) – z(H)
where FA and H are the False Alarm and Hit rates, respectively, that correspond
to right-tail probabilities on the normal distribution. Thus, z(FA) and z(H) are
the z-scores that correspond to these right-tail p-values represented by FA and H.
Larger absolute values of d’ mean that a person is more sensitive to the
difference between the Signal Present and Signal Absent distributions. d’ values
near zero indicate chance performance.
Some statisticians use z(FA) and z(H) to indicate the z-scores for the left-tail p-
values from the normal distribution. However, in signal detection applications, hit
rate and false alarm rates refer to the right-tail of the normal distribution. Thus,
we follow Swets (1996) and use z(FA) and z(H) to indicate z-scores for the right-
tail probabilities. If we were to use the left-tail p-values, then the correct formula
for d’ would be z(H) – z(FA).
What if p = 0 or p = 1?
It is possible that a respondent detects every signal, giving a hit rate of H = 1.00,
or a respondent might make no false alarms giving FA = 0. A right-tail p-value of
0 corresponds to z = infinity while a p-value of 1 corresponds to z =negative
infinity. This poses a problem for computing d ‘
A conventional adjustment is to set the minimum p = 1/N where N is the number
of trials used in the calculation of p. Similarly, we can set the maximum value
for p = (N-1)/N.
Ideally, we design the study to avoid a large portion of p-values at the extremes.
Exercise 3: A subject has a false alarm rate of .30 and a hit rate of .90.
Calculate d’ for this subject using the formula: d’ = z(FA) – z(H).
The False Alarm rate of .30 indicates that when the signal is absent the subject
says ‘Yes’ 30% of the time. Thus, the right-tail p-value on the red ‘Signal Absent’
distribution is .30. Use the p-z converter to find z(FA), the z value that
corresponds to this p-value.
The Hit rate of .90 indicates that when the signal is present the subject says
“Yes” 90% of the time. Thus, the right-tail p-value on the green ‘Signal Present’
distribution is .90. Use the p-z converter to find z(H), the z-value that corresponds
to this p-value.
We can now compute d’ = z(FA) – z(H).
An observer has a hit rate of .90 and a false alarm rate of .30.
Calculate d’, z(Hits), and z (False Alarms) for this person.
Z (False Alarms) = z (.30) = .524
Z (Hits) = z (.90) = -1.282
d’ = z (False Alarms) – z(Hits)
= .524 – (-1.282) = 1.806
Exercise 4 (Sensitivity):
The value of d‘ is a measure of the sensitivity of a respondent to the presence of
the signal. In this exercise, we will compare the sensitivity of two respondents to
the presence of the signal.
a. Find d’ for Anita who had a hit rate of .90 and a false alarm rate of .30.
Solution: Anita’s d’ = 1.81
Measure of Decision Criterion
Criterion is a measure of the willingness of a respondent to say ‘Signal
Present’ in an ambiguous situation. The choice of a criterion may depend on
perceived consequences of outcomes. For example, if the consequences are
costly for saying ‘Signal Present’ when the signal actually is absent, then a
respondent may generally be less willing to say ‘Signal Present.’ On the other
hand, if the consequences are more costly for failing to detect a signal when
it is present, then a respondent may be more willing to say ‘Signal Present.’
Researchers may be interested in comparing response bias (i.e., criterion) for
groups of individuals who differ in various ways. For example, are older
people more reluctant to say ‘Signal Present’ when the situation is
ambiguous?
Setting a criterion is equivalent to setting alpha error in a hypothesis testing
situation. Alpha error is the chance a researcher is willing to take that a test
will be found ‘statistically significant’ when there is no real effect. We define
the Criterion as the z score on the Signal Absent distribution. A larger value of
the Criterion implies that the respondent requires stronger evidence before
saying that the signal is present.
Refer to this link for the SDT applet
[Link] .
Exercise 5 (Criterion):
In the SDT applet, Move the “d’ =” box in the Normal Distributions panel to
obtain a value of d’ near 1.8. Now move the Criterion box to the far right.
a. Slowly move the Criterion box in the Normal Distributions panel from the
far right to the far left, and back again. Explain what happens to the hit and
false alarm rates as you shift the criterion. How do the rates of change in hits
and false alarms differ? What happens to d’ as you shift criterion?
Answer:
When the criterion is at the far right, the respondent will never say ‘Signal
Present.’ Consequently, the hit and the false alarm rates are both zero. When
the criterion is at the far left, the respondent will always say ‘Signal Present,’
and the hit and false alarm rates are both 1.00. As the criterion moves from
the far right to the left, the respondent becomes more willing to say, ‘Signal
Present.’ When d’ is positive, as it is here, the hit rate increases more rapidly
than the false alarm rate until the criterion reaches the point where the two
normal curves intersect. From there on, the false alarm rate grows more
quickly. d’ is not affected by changes in the criterion.
b. Use the applet to model the performance of Anita, who had hit and false
alarm rates of .90 and .30, respectively. What is her d’ and her Criterion
(shown in the boxes)?
Answer: Anita has d’ = 1.81 and Criterion = .52.
c. Use the applet to model the performance of Bob, who had hit and false
alarm rates of .68 and .09, respectively. What is his d’ and his Criterion?
Bob has d’ = 1.81 and Criterion = 1.34.
d. Compare the performance of Anita and Bob in simple language, avoiding
technical SDT terminology.
Answer: Anita and Bob have equal sensitivity to the signal, but compared to
Bob, Anita will say ‘Signal Present’ at a lower level of stimulation.
e. How is the Criterion value related to the Signal Absent distribution?
Answer: When the signal is absent, the stimulation on any given trial may be
more or less like a signal. The criterion represents the minimum level of
stimulation needed for the respondent to say, ‘Signal Present.’ The computed
Criterion is the z-score of this stimulation value on the Signal Absent
distribution. Thus, the criterion reflects the willingness of the respondent to
make a false alarm (i.e., say ‘Signal Present’ when it is absent).
f. Set d’ = 1.8 and move the Criterion to about 3. What are the hit and false
alarm rates?
Answer: With d’ = 1.8 and Criterion = 3, the hit rate is about .115 and the
false alarm rate is about .001.
g. Continue with d’ = 1.8 and move the Criterion to about 0. What are the hit
and false alarm rates?
Answer: With d’ = 1.8 and Criterion = 0, the hit rate is about .96 and the false
alarm rate is about .49.
h. If a false alarm is very costly relative to the benefit of a hit, which Criterion
is better, 3 or 0, and why?
Answer: If false alarms are very costly, the higher criterion is better. The ratio
of hits to false alarms is greater than 100:1, while with the second criterion,
the ratio is only about 2:1.
Learning Task
Solve the following:
1) Participant A had a hit rate of .82 and a false alarm rate equal to .24.
Participant B had a hit rate of .82 and a false alarm rate of .40. Compare
these two participants in terms of the d’ and criterion scores.
2. Compute the following:
a) Hit rate = .85, False Alarm rate = ?
b) False Alarm rate = .40, Hit rate = ?
c) Hit rate = .30, False Alarm rate = ?
3. A construction worker sued his employer over a work-related injury. He
contended that loud noises at the construction site ruined his hearing, leaving
him totally deaf. The insurance company representing the construction
company wanted to find out how damaged his hearing really is. To do so,
they constructed a “yes-no” hearing test in which half of the trials contain a
signal that is quiet, but is readily perceivable to someone with normal
hearing, and half contain no signal. The response rates are listed in the table
below. Calculate d’ and give your conclusion as to whether the worker really
is totally deaf.
Proportions:
Signal Present Signal Absent
Say “yes” .10 .50
Say “no” .90 .50
What is the Hit rate, False Alarm rate, d’, and Criterion measure?
Fun fact: have you ever heard that eating carrots improves eyesight and/or
confers the ability to see in the dark? That’s the result of a WWII British
intelligence campaign to prevent Germany from figuring out that Royal Air
Force pilots had on-board radar . Carrots are still good for you, though, and
can help in maintaining (but not super-powering) eyesight.
Assessment
Assessment will be posted/announced in our official E-learning portal.
Instructions on how to submit student output.
Instructions will be posted in our VSUEE learning portal.
Module Posttest
This will be posted in the official virtual classroom.
References and Additional Resources
Middlesworth, Matt (March 15, 2021). Cognitive Ergonomics 101: Definition,
Applications, and Disciplines. Retrieved from [Link]
ergonomics/
The Theory of Signal Detection. (n.d.). Retrieved January 16, 2024, from
[Link]
[Link]
Chapter 8 Signal Detection Theory | Advanced Statistics I & II. (n.d.).
[Link]
[Link]#distinguishing-signal-from-noise
Answers to the Pretest
This will be posted in the official virtual classroom.
Answers to the Post Test
This will be posted in the official virtual classroom.