Analyzing Experimental Data
Chapter 11
Null Hypothesis Significance Testing (NHST)
Treatment condition means can differ because of:
The treatment effect Error variance (e.g., individual differences,
measurement error)
NSHT allows us to make a decision:
Is the difference large enough to be highly unlikely
(e.g., p < .05) to be due to chance? OR, is the difference small enough that it could reflect error variance?
Null Hypothesis Significance Testing (NHST)
Null hypothesis (H0) the independent variable has
no effect on the dependent variable
Experimental or alternate hypothesis (H1) the
independent variable does have an effect on the dependent variable
Even though we are interested in H1, NHST tests the
null hypothesis (H0).
Null Hypothesis Significance Testing (NHST)
Estimate how much the means might differ due to
error variance even if the independent variable has no effect (H0). If the obtained difference exceeds that amount, reject H0. (the difference is highly unlikely [e.g., p <
.05] to be due to chance)
Otherwise, fail to reject H0 (no significant
difference).
Any differences may have been due to chance (p > .05) Still, we cannot logically conclude that there is no
effect. We might not have enough power.
Errors in Hypothesis Testing
Type I Errors:
Reject H0 (the null hypothesis) when it is actually
true Concluding that there is an effect when there actually is not Alpha level (a) is the probability of making a Type I error.
Type II Errors:
Fail to reject H0 (the null hypothesis) when it is
actually false Concluding that there is not an effect when there actually is Beta () is the probability of making a Type II error
Type I and Type II Errors
Actual Situation No Effect (H0 True) Experimenters Decision Effect Exists (H0 False)
Correct Decision
Reject H0 Type I error
Retain H0
Correct Decision
Type II error
Statistical Power
The power of a test is the probability that the test will
correctly reject a false null hypothesis
i.e., it will detect a treatment effect if one exists Power = 1 , or 1 the probability of making a Type II error
Power is usually estimated before starting a study, using:
Effect size Sample size Alpha level Type of test (one- vs. two-tailed)
Power analysis suggests how many participants are
needed to detect a significant effect at a specific power level.
Influences on Power
Power is increased with larger effect sizes larger sample sizes larger (less stringent) alpha levels (e.g., a = .05) one-tailed (instead of two-tailed tests) tests
Estimating sample size needed a priori with G*Power
Can also estimate post hoc power
Power Illustration
Sampling distribution if H0 is true Sampling distribution if H0 is false
a = probability of making a Type I error = probability of making a Type II error
Power = 1
Power =.9 1-tailed z-test, a = .05
H0
H1
Power =.9 1-tailed z-test, a = .05
H0
H1
Power (1-b)
Power =.85 2-tailed z-test, a = .05
H0
H1
Power =.85 2-tailed z-test, a = .05
H0
H1
Power (1-b)
Effect Size
Statistical significance (rejecting H0) does not
necessarily tell you about the size or practical significance of an effect Statistically significant effects are not always practically significant!
Effect size measures the absolute magnitude of a
treatment effect, independent of sample size
Two Kinds of Effect Size Indicators
Proportion of variance effect size (r2) the
proportion of variance in the dependent variable that is due to the independent variable; ranges from .00 to 1.00
Mean difference effect size (Cohens d) size of
the difference between two means relative to the standard deviation of the data; typically ranges from -3.0 to +3.0
Cohens d (size of difference in standard deviatio
Magnitude of d Evaluation of Effect Size
d = .20
d = .50 d = .80
Small effect
Medium effect Large effect
r2 (percentage of variance accounted for)
Magnitude of r2 Evaluation of Effect Size
r2 = .01 r2 = .09 r2 = .25
Small effect Medium effect Large effect
What power is adequate?
Power of .80 (80% chance of detecting an effect)
is generally considered desirable In psychology, average power levels are about:
.20 for detecting small effect sizes .50 - .60 for detecting medium effect sizes (Clark-Carter, 1997; Cohen, 1962; Rossi, 1990;
Sedlmeier & Gigerenzer, 1989)
Power examples
All 2-tailed tests with a = .05, calculated with GPower
Type of Test Effect size Small, |r|=.10 Correlation Medium, |r|=.30 Number of participants needed to get power = .80 N = 779 N = 82
Large, |r|=.50
Small, d =.20 t-test with equal Ns Medium, d =.50 Large, d =.80 Small, f = .10 2x2 ANOVA Medium, f = .25 (interaction term) Large, f = .40
N = 26
N = 788 N = 128 N = 52 N = 787 N = 128 N = 52
Issues with Null Hypothesis Significance Testing (NHST)
There is nothing sacred about p < .05
What about p = .051? False dichotomy (effect vs. no effect)
NHST cannot directly tell us the probability that
the null hypothesis (H0) is true
We may just not have enough power to detect the
(small) effect In reality, most null hypotheses may be false (Loftus, 1996)
Example H0: m1 = m2 = m3 = m4 This means there are ZERO differences (not even .00001)
in the population
How can we ameliorate the problems with NHST?
Always report effect size in addition to statistical
significance. Use meta-analysis of effect sizes across studies, when possible Try to maintain adequate power (ideally > .80). Report plots and graphs of data, when appropriate. Report confidence intervals (CIs)
E.g., We can be 95% confident that the population
mean difference lies between 0.04 and 1.36 CIs can be used to test hypotheses (does the CI include 0?) CIs also provide a direct estimate of the population means
t-tests
t-tests are used to test the difference between two
means
One-sample t-test: Compare to population mean or
baseline Independent samples t-test: Compare means for two independent groups Related samples or paired t-test: Compare means for two related groups (matched subjects or repeated measures)
Error variance is used to estimate how much the
means of the conditions would differ due to error variance alone. The observed difference between the means is compared with this estimate.
Conducting an independent measures t-test
Calculate a critical t-value using the t-table
Pick alpha level (usually a = .05) Find df
Calculate the obtained t-statistic
Difference between means Standard error of the differences between means
Compare the obtained and critical t-statistics
If obtained t exceeds critical t, reject the null
hypotheses If not, fail to reject the null hypothesis
Hypotheses
Directional hypothesis states which of the two
condition means is expected to be larger
Use a one-tailed test
Nondirectional hypothesis states that the two
means are expected to differ but does not specify which will be larger
Use a two-tailed test