DATA ANALYSIS
HYPOTHESIS TESTING
RECAP & OUTLOOK
BAYESIAN PARAMETER ESTIMATION FREQUENTIST HYPOTHESIS TESTING
▸ model M captures prior beliefs ▸ model M captures a hypothetically
about data-generating process assumed data-generating process
▸ prior over latent parameters ▸ fix parameter value of interest
▸ likelihood of data ▸ likelihood of data
▸ Bayesian posterior inference using ▸ single out some aspect of the data
observed data Dobs as most important (test statistic)
▸ compare posterior beliefs to some ▸ look at distribution of test statistic
parameter value of interest given the assumed model
(sampling distribution)
▸ check likelihood of test statistic
applied to the observed data Dobs
LEARNING GOALS
▸ understand basic idea of frequentist hypothesis testing
▸ understand what a p-value is
▸ definition, one- vs two-sided
▸ test statistic & sampling distribution
▸ relation to confidence intervals
▸ significance levels & α -error
p-value
PRELIMINARIES
▸ research hypothesis: theoretically implied answer to a main question of interest for
research
▸ e.g., is the coin fair?
▸ e.g., faster reactions in reaction time trials than in go/No-go trials? (Mental
Chronometry)
▸ null hypothesis: specific assumption made for purposes of analysis
▸ fix parameter value in a data-generating model for technical reasons
▸ analogy: useful assumption in mathematical proof (e.g., in reductio ad absurdum)
▸ alternative hypothesis: the antagonist of the null hypothesis, specified to relate the
null hypothesis to the research hypothesis
P-VALUE
Binomial
Model
BAYESIAN BINOMIAL MODEL
N θ θ ∼ Beta(…)
k ∼ Binomial(θ, N)
k
BAYESIAN BINOMIAL MODEL
θ ∼ Beta(…)
N xi xi ∼ Bernoulli(θ0)
k
FREQUENTIST BINOMIAL MODEL
[doted line = “working assumption”]
θ0 xi ∼ Bernoulli(θ0) [likelihood of “raw” data]
[test statistic (derived from “raw” data)]
N xi
FACT:
The sampling distribution of k is:
k k ∼ Binomial(θ0, N)
FREQUENTIST BINOMIAL MODEL
θ0 ▸ null-hypothesis: θ = θ0
▸ test statistic: k derived from “raw” data x
▸ the most important (numerical) aspect of the
data for the current testing purposes
N xi ▸ sampling distribution: likelihood of observing a
particular value of k in this model
▸ notice: the observed data Dobs has not yet made
any appearance
k ▸ remark: sometimes summary statistics of Dobs other than the
test statistic might be used in the model
FREQUENTIST BINOMIAL MODEL
|H
θ0 ▸ likelihood of data: random variable 𝒟 0
|H0
▸ sampling distribution: random variable T
N xi
P(T |H0 = k) = Binomial(k, θ0, N)
k
Binomial
p-values
BINOMIAL TEST
▸ 24/7 example: N = 24 and k =7
▸ t(Dobs) = 7
|H0 = k) = Binomial(k, θ , N)
▸ P(T 0
▸ p-value definition:
p(Dobs) = |H
P(T 0 ⪰ 0,a
H t(Dobs))
we know this ??? we know this
What counts as “more extreme evidence against the null hypothesis” is a
context-sensitive notion that depends on the null-hypothesis and the
alternative hypothesis because only when put together do null- and
alternative hypothesis address the research question in the background.
BINOMIAL TEST
▸ compare two research questions ▸ we still use a point-valued null-
hypothesis for technical reasons
1. Is the coin fair?
▸ the alternative hypothesis is
▸ H0 : θ = 0.5 H0,a
important to fix the meaning of ⪰
▸ Ha : θ ≠ 0.5
2. Is the coin biased towards heads?
▸ H0 : θ = 0.5
▸ Ha : θ < 0.5
BINOMIAL TEST
▸ Case 1: Is the coin fair?
▸ H0 : θ = 0.5
▸ Ha : θ ≠ 0.5
▸ which values of k are
more extreme evidence
against H0?
BINOMIAL TEST
▸ Case 1: Is the coin fair?
▸ H0 : θ = 0.5
▸ Ha : θ ≠ 0.5
▸ which values of k are
more extreme evidence
against H0?
▸ anything that’s even
less likely to occur
BINOMIAL TEST
BINOMIAL TEST
▸ Case 2: Is the coin
biased towards heads?
▸ H0 : θ = 0.5
▸ Ha : θ < 0.5
▸ which values of k are
more extreme evidence
against H0?
BINOMIAL TEST
▸ Case 2: Is the coin
biased towards heads?
▸ H0 : θ = 0.5
▸ Ha : θ < 0.5
▸ which values of k are
more extreme evidence
against H0?
▸ anything even more in
favor of Ha
BINOMIAL TEST
p-value
revisit
P-VALUE
significance
and α-errors
SIGNIFICANCE LEVELS
▸ standardly we fix a significance level α before the test
▸ common values of α are:
▸ α = 0.05
▸ α = 0.01
▸ α = 0.001
▸ if the p-value for the observed data lower than the pre-established
threshold of significance, we say that the test result was significant
▸ a significant test result is conventionally regarded as “strong enough” evidence
against the null-hypothesis, so that we can reject the null hypothesis as a viable
explanation of the data
▸ non-significant results are interpreted differently in different approaches (more
later)
α-ERROR
▸ an α-error (aka type-I error) occurs when we reject a true null hypothesis
▸ by definition this type of error occurs, in the long run, with a proportion of no
more than α
▸ it is in this way that frequentist statistic is subscribed and cherishes a regime of
long-term error control on research results
▸ Bayesian approaches (usually) are not concerned with long-term error control
Exercise