Week 9: Sign Test

March 6th, 2025

Slides adapted from Dr. Grace Truong
Updated Announcements & Notes
• SPSS Assignment 3
• Demo to be uploaded to canvas soon
• Assignment to be published soon
• Consider doing the review questions in Chapter 10 of the Pagano
Textbook
Learning Goals
• Define Type I and Type II errors and explain why it is important to
discuss these possible errors
• Specify the relationship between Type I and Type II errors
• Specify the relationship between the alpha level and Type I and Type
II errors.
• Apply the steps of hypothesis testing to sign tests (form hypotheses,
make decisions about Ho, form conclusions)
• Use the binomial distribution to evaluate the results of a sign test
Type 1 and Type 2 Errors
Type I Error (False Positive)

• We reject H0 when it is actually true.

• Example: We conclude that dogs can predict earthquakes, but
in reality, their barking is just random.

• Consequence: People might rely on dog barking as an

earthquake warning, leading to false alarms and unnecessary
panic.
Type 1 and Type 2 Errors
Type II Error (False Negative)

• We fail to reject H0 when it is actually false.

• Example: We conclude that dogs cannot predict earthquakes
(barking is random), but in reality, dogs do sense something
before quakes.

• Consequence: We ignore a real warning system, missing an

opportunity to study how dogs might detect earthquakes early.
Type 1 and Type 2 Errors – A balancing act
When we design our experiments, we want to reduce the chance of making
a Type 1 and Type 2 error

Type 1: False Positive Type 2: False Negative

Set a more stringent (lower) alpha Increase “power”

level • Increase sample size
• Might increase the probability of
making a Type 2 error
An introduction to hypothesis
testing
With the Sign Test
Goals in hypothesis testing
• To objectively verify

• We evaluate the null hypothesis by assuming it is true and

testing the reasonableness of this assumption by calculating
the probability of getting the results, if chance alone is
happening
What is the Sign Test?
• Compute the difference between conditions for each difference –
obtain plusses and minuses
• Does not care about magnitude differences

• Treat plusses and minuses like heads and tails and use BINOMIAL
DISTRIBUTION to calculate probability of obtaining observed result
(or a more extreme result)
• Pluses/minuses are equally likely under the null hypothesis so use probability
= 0.50
Formulating hypothesis
When we have a research question in mind, we must first create two
hypothesis
Alternative
Null Hypothesis
Hypothesis
• Can be directional or • Is the “opposite” of
non-directional your alternative
hypothesis

Together, the null and alternative hypothesis must encompass all

possible outcomes
 That is why it is
recommended to write
Formulating hypothesis your Alt. hypothesis first
before your null

When we have a research question in mind, we must first create two

hypothesis
Alternative
Null Hypothesis
Hypothesis
• Can be directional or • Is the “opposite” of
non-directional your alternative
hypothesis

Together, the null and alternative hypothesis must encompass all

possible outcomes
 Sleep Experiment
ID Hrs of sleep Pre- Hrs of sleep Post- Imagine you are a Health
Protocol protocol
1 5 3
Psychologist.
2 3 5
You are interested in whether a
new health protocol influences
3 7 9
how much sleep a person gets
4 4 9
each night.
5 6 4
What is my hypothesis?
6 8 10
• What is my null hypothesis
7 2 2
8 5 3
• What is my alternative
hypothesis?
9 7 10
 Is the sign test appropriate?
ID Hrs of sleep Pre- Hrs of sleep Post- • Specific hypothesis testing
Protocol protocol
1 5 3
technique
2 3 5
• Used for 2 condition repeated
measures designs
3 7 9
• Paired scores, differences are
4 4 9 being analyzed
5 6 4
6 8 10
7 2 2
8 5 3
9 7 10
 Formulating my Hypothesis
ID Hrs of sleep Pre- Hrs of sleep Post- You are interested in whether a
Protocol protocol
1 5 3
new health protocol influences
how much sleep a person gets
2 3 5
each night.
3 7 9
What is my hypothesis?
4 4 9
1. What is my alternative
5 6 4
hypothesis?
6 8 10
7 2 2
2. What is my null hypothesis?
8 5 3
9 7 10
 Formulating my Hypothesis
ID Hrs of sleep Pre- Hrs of sleep Post- You are interested in whether a new
Protocol protocol health protocol influences how
1 5 3 much sleep a person gets each night.
2 3 5 What is my hypothesis?
3 7 9 1. What is my alternative
4 4 9
hypothesis?
• The new health protocol influences
5 6 4 how much sleep a person gets each
night
6 8 10
2. What is my null hypothesis?
7 2 2
• The new health protocol does not
8 5 3 influence how much sleep a person
gets each night
9 7 10
 How to do conduct a sign test
ID Hrs of sleep Hrs of sleep Difference Sign • We count the number of
Pre-Protocol Post-protocol (Post-Pre) positive differences and the
1 5 3 -2 - number of negative
2 3 5 +2 + differences
3 7 9 +2
• Pos. Differences: 5
+
• Neg. Differences: 3
4 4 9 +5 +
• What do we when the
5 6 4 -2 - difference is neither
6 8 10 +2 + positive or negative?
7 2 2 0 • We remove them from
NA analysis
8 5 3 -2 - • N decreases
9 7 10 +3 +
What do + and - mean • Depends on how you
defined it
ID Hrs of sleep Hrs of sleep Difference Sign
Pre-Protocol Post-protocol (Post-Pre) • For this data, we did post-
1 5 3 -2
pre
-
• Plus = positive difference –
2 3 5 +2 + more sleep after the protocol
3 7 9 +2 + • Minus = negative difference –
less sleep after the protocol
4 4 9 +5 +
5 6 4 -2
• Does it matter which why I
- calculate?
6 8 10 +2 + • No, as long as you are
7 2 2 0 NA consistent (within a dataset)
and can interpret plusses and
8 5 3 -2 - minuses correctly
9 7 10 +3 +
Sign Test • Based on these results,
what should we
ID Hrs of sleep Hrs of sleep Difference Sign
Pre-Protocol Post-protocol (Post-Pre) conclude?
1 5 3 -2 - • Might be tempting to
conclude that the health
2 3 5 +2 + protocol influences sleep
3 7 9 +2 +
4 4 9 +5 +
• As researchers we should
5 6 4 -2 - ask
6 8 10 +2 + • What is the probability of
7 2 2 0 NA obtaining a value as
8 5 3 -2
extreme or more extreme
- than what we observe, if
9 7 10 +3 + there is no effect
What is the probability of obtaining a value as extreme – or more
extreme – than 5 pluses?

0.3000 0.2734
0.2500 0.2188 0.2188
0.2000
0.1500
0.1094 0.1094
0.1000
0.0500 0.0312 0.0312
0.0039 0.0039
0.0000
0 1 2 3 4 5 6 7 8
Number of Pluses
Why should we look at 0+, 1+, 2+, 3+, and 6+, 7+, 8+?
• What is the probability of obtaining a value as extreme - or
more extreme than 5 pluses?
0.3000 0.2734
“Or more extreme”
0.2500 0.2188 0.2188 Need to find all
0.2000 outcomes that are
even less probable
0.1500
0.1094 0.1094
0.1000 “As extreme”
0.0500 0.0312 0.0312 Need to find opposite
0.0039 0.0039 outcome to observed
0.0000 outcome
0 1 2 3 4 5 6 7 8
• Why?
Number of Pluses
 Add
probabilities
together
How do we evaluate our hypothesis?
• Given that there are 8 N
No. of P or Q
0.5
Events
participants, N = 8 0 0.0039
• We can let P = probability of 1 0.0312
2 0.1094
getting a plus with any 3 0.2188
participant 8 4 0.2734
5 0.2188
• If chance alone is operating 6 0.1094
• p(+) = p(-) 7 0.0312
• Thus P = 0.50 8 0.0039

5 Pluses or More sum(0+, 1+, 2+, 3+,

Extreme 5+, 6+, 7+, 8+)
How do we evaluate our hypothesis?
• Given that there are 8 N
No. of P or Q
0.5
Events
participants, N = 8 0 0.0039
• We can let P = probability of 1 0.0312
2 0.1094
getting a plus with any 3 0.2188
participant 8 4 0.2734
5 0.2188
• If chance alone is operating 6 0.1094
• p(+) = p(-) 7 0.0312
• Thus P = 0.50 8 0.0039

5 Pluses or More sum(0+, 1+, 2+, 3+,

0.7266
Extreme 5+, 6+, 7+, 8+)
How do we evaluate our hypothesis?
• Probability of obtaining 5+ or N
No. of P or Q
0.5
Events
more extreme
0 0.0039
• P(5+ as/or more extreme) = 1 0.0312
0.7266 2 0.1094
• p-value (probability value) of .72 is 3 0.2188
pretty high 8 4 0.2734
• How do we decide if the 5 0.2188
protocol influenced sleep 6 0.1094
7 0.0312
length? 8 0.0039
• Compare with a pre-specified
“tolerance” level
• alpha level (α) 5 Pluses or More sum(0+, 1+, 2+, 3+,
0.7266
Extreme 5+, 6+, 7+, 8+)
What is an alpha level?
• alpha level (α) – is the maximum probability of rejecting the null
hypothesis when it is actually true
• Most common alpha used/selected is 0.05

• How willing are you to risk a Type 1 error?

• Higher alpha level (α) means greater risk of Type 1 error because it
makes you more likely to reject the null hypothesis
What is an alpha level?
• You decided on the alpha level before you run your statistical test
• You cannot change alpha after you see the results of your test in
order to get the decision you want

α = 0.05 α = 0.05 α = 0.10

p = 0.07 p = 0.07

You must RETAIN Ho! You can REJECT Ho!

(conclude no effect) (conclude effect is
present)
Not allowed!!
How do we evaluate our hypothesis?
• Pre-set alpha value = .05 N
No. of P or Q
0.5
Events
• Obtained p-value = 0.7726
0 0.0039
1 0.0312
• Do I reject or retain my null 2 0.1094
hypothesis? 3 0.2188
• If p is greater than α, retain null 8 4 0.2734
hypothesis 5 0.2188
• If p is less or equal to than α, reject null 6 0.1094
hypothesis 7 0.0312
8 0.0039
• What can I conclude?
• Fail to reject my null hypothesis: The 5 Pluses or More sum(0+, 1+, 2+, 3+,
protocol does not influence sleep Extreme 5+, 6+, 7+, 8+)
0.7266
length.
Non-Directional vs. Directional Hypothesis
• For a non-directional hypothesis (like “protocol affects sleep
efficiency”), both tails of the distribution are evaluated
• Two-tailed test

• For a directional hypothesis (like “protocol increases sleep

efficiency”), only one tail of the distribution is evaluated
• One-tailed test
 One tail or two tails: Depends on your
hypothesis
• Ha: protocol affects sleep • Ha: protocol increases sleep (predict plusses)
• 0, 1, 2, 3, or 5,6,7,8 plusses: two- • 5,6,7,8 plusses : one-tailed test (p = 0.3863)
tailed test
0.3000 0.2734 0.3000 0.2734
0.2500 0.2500 0.2188 0.2188
0.2188 0.2188
0.2000 0.2000

0.1500 0.1500
0.1094 0.1094 0.1094 0.1094
0.1000 0.1000

0.0500 0.0312 0.0312 0.0500 0.0312 0.0312

0.0039 0.0039 0.0039 0.0039
0.0000 0.0000
0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8
Number of Pluses Number of Pluses
Sign Test – 6 steps
Step 1: Formulate clear, specific null and alternate hypotheses
• Directional or non-directional Ha/H1 ??

Step 2: Set your alpha value

Step 3: Gather data and calculate the number of pluses and minuses
obtained
Sign Test – 6 steps
• Step 4: Determine the probability of getting an outcome this extreme
or any more extreme
• An extreme value is one that favors the alternate hypothesis
• If you had a non-directional H1, you will be doing a two-tailed evaluation
• If you had a directional H1, you will be doing a one-tailed evaluation
Sign Test – 6 steps
Step 5: Compare p-value to alpha, make a decision
If p is greater than α, retain null hypothesis
If p is less or equal to than α, reject null hypothesis

Step 6: Write out a clear, specific conclusion that is consistent with

the wording of your original hypotheses
Sign Test – Spelling example
• You are interested in whether children’s spelling improves over the
course of the year. You gave children the same spelling test at the
beginning of the year and at the end of the year.

• Alternative Hypothesis: Children performance is better at the end-of-

year spelling test than at the beginning-of-the-year test
• Null hypothesis: Children’s performance is not better at the end-of-
year spelling test than at the beginning-of-the-year test

• Directional or non-directional? One-tailed test or two-tailed test?

Sign Test – Spelling example
• You are interested in whether children’s spelling improves over the
course of the year. You gave children the same spelling test at the
beginning of the year and at the end of the year.

• Alternative Hypothesis: Children performance is better at the end-of-

year spelling test than at the beginning-of-the-year test
• Null hypothesis: Children’s performance is not better at the end-of-
year spelling test than at the beginning-of-the-year test

• Directional or non-directional? One-tailed test or two-tailed test?

Sign Test – Spelling example
Alpha level: 0.05 (directional, one-tailed test)
Subject ID Beginning of the Year End of the Year
1 80 70
2 100 80
3 110 90
4 120 90
5 80 70
6 100 110
7 110 80
8 125 120
9 110 80
10 90 70
Sign Test – Spelling example
Alpha level: 0.05 (directional, one-tailed test)
Subject ID Beginning of the Year End of the Year Difference (Beginning-End)

1 80 70 +10
2 100 80 +20
3 110 90 +20
4 120 90 +30
5 80 90 -10
6 100 110 -10
7 110 80 +30
8 125 120 +5
9 110 80 +30
10 90 70 +20
Sign Test – Spelling example
• Determine probability of observed outcome or more extreme
outcome (8+, 9+, 10+)

0.0439
0.0098
+ 0.0010
0.0547
Sign Test – Spelling example
• Compare p-value to alpha (0.05)

p = 0.0547 > α = 0.05

• Retain null hypothesis

• Conclude that children’s performance is not better

(the same or worse) at the end-of-year spelling test
than at the beginning-of-the-year test
But what if you predicted in the wrong
direction?
• Imagine you hypothesize that people liked Barbie more than
Oppenheimer. Minus difference scores = prefers Barbie
Subject ID Oppenheimer Barbie Sign
1 9 5 +
2 7 6 +
3 6 4 +
4 5 8 -
5 5 7 -
6 8 6 +
But what if you predicted in the wrong
direction?
• Imagine you hypothesize that people liked Barbie more than
Oppenheimer. Minus difference scores = prefers Barbie
Subject ID Oppenheimer Barbie Sign
1 9 5 +
2 7 6 +
3 6 4 +
What does 2-
4 5 8 - and 4+ tell us?
5 5 7 -
6 8 6 +
What if you predict the wrong direction?
• Determine probability of observed outcome or more
extreme outcome:
• 2 minuses, 3 minuses, 4 minuses, 5 minuses, or 6 minuses

0.2344
0.3125
0.2344
0.0938
+0.0156
0.8907

p = 0.8907, retain the null hypothesis