CHAPTER 2.

TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS22

0.15 0.30

1.0
Density

0.8
0.00

0.6
8 10 12 14 16

F(y)
yA

0.4
Density
0.15

0.2
0.0
0.00

8 10 12 14 16 8 10 12 14 16
yB y

Figure 2.5: Histograms and empirical CDFs of the second two hypothetical
samples.
6
0.6

5
Density

Density
3 4
0.4

2
0.2

1
0.0

0 1 2 3 4 0.1 0.2 0.3 0.4 0.5

t statistic KS statistic

Figure 2.6: Randomization distributions for the t and KS statistics for the
second example.
CHAPTER 2. TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS23

Discussion: These last two examples suggest we should abandon gt in favor

of gKS if we are interested in comparing the following hypothesis:

H0 : treatment does not affect response

H1 : treatment does affect response

This is because, as we found, gt is not sensitive all violations of H0 , it is only

sensitive to violations of H0 where there is a difference in means. However,
in many situations we are actually interested in comparing the following
hypotheses:

H0 : treatment does not affect response

H1 : treatment increases responses or decreases responses

In this case H0 and H1 are not complementary, and we are only interested
in evidence against H0 of a certain type, i.e. evidence that is consistent with
H1 . In this situation we may want to use a statistic like gt .

2.5 Basic decision theory

Task: Accept or reject H0 based on data.

truth
action H0 true H0 false
accept H0 correct decision type II error
reject H0 type I error correct decision

As we discussed,
• the p-value can measure of evidence against H0 ;

• the smaller the p-value, the larger the evidence against H0 .

Decision procedure:
1. Compute the p-value by comparing observed test statistic to the null
distribution.

2. Reject H0 if the p-value ≤ α, otherwise accept H0 .

CHAPTER 2. TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS24

This procedure is called a level-α test. It controls the pre-experimental prob-

ability of a type I error, or for a series of experiments, controls the type I
error rate.

Pr(type I error|H0 ) = Pr(reject H0 |H0 )

= Pr(p-value ≤ α|H0 )
= α

Single Experiment Interpretation: If you use a level-α test for your ex-
periment where H0 is true, then before you run the experiment
there is probability α that you will erroneously reject H0 .

Many Experiments Interpretation: If level-α tests are used in a large

population of experiments, then H0 will be declared false in (100 × α)%
of the experiments in which H0 is true.

Pr(H0 rejected|H0 true) = α

Pr(H0 accepted|H0 true) = 1−α
Pr(H0 rejected|H0 false) = ?
Pr(H0 accepted|H0 false) = ?

Pr(H0 rejected|H0 false) is the power. Typically we need to be more specific

than “H0 false” in order to calculate the power. We need to specify how it
is false.
Chapter 3

Tests based on population

models

3.1 Relating samples to populations

If the experiment is

• complicated,

• non- or partially randomized, or

• includes nuisance factors

then a null distribution based on randomization may be difficult to obtain.

An alternative approach to hypothesis testing is based on formulating a sam-
pling model.
Consider the following model for our wheat yield experiment:

• There is a large/infinite population of plots of similar size/shape/com-

position as the plots in our experiment.

• When A is applied to these plots, the distribution of plot yields can be

represented by a probability distribution pA with
R
expectation = E[YA ] = ypA (y)dy = µA ,
variance =Var[YA ] = E[(YA − µA )2 ] = σA2 .

25
CHAPTER 3. TESTS BASED ON POPULATION MODELS 26

• When B is applied to these plots, the distribution of plot yields can be

represented by a probability distribution pB with
R
expectation = E[YB ] = ypB (y)dy = µB ,
variance =Var[YB ] = E[(YB − µB )2 ] = σB2 .
• The plots that received A in our experiment can be viewed as indepen-
dent samples from pA , and likewise for plots receiving B.

Y1,A , . . . , YnA ,A ∼ i.i.d. pA

Y1,B , . . . , YnB ,B ∼ i.i.d. pB

Recall from intro stats:

nA
1 X
E[ȲA ] = E[ Yi,A ]
nA i=1
nA
1 X
= E[Yi,A ]
nA i=1
nA
1 X
= µA = µA
nA i=1

We say that ȲA is an unbiased estimator of µA . Furthermore, if Y1,A , . . . , YnA ,A

are independent samples from population A, then
1 X
Var[ȲA ] = Var[ Yi,A ]
nA
1 X
= Var[Yi,A ]
n2A
1 X 2
= σA = σA2 /nA .
n2A
This means that as nA → ∞,

Var[ȲA ] = E[(YA − µA )2 ] → 0

which, with unbiasedness, implies

ȲA → µA
 CHAPTER 3. TESTS BASED ON POPULATION MODELS 27

‘All possible' A wheat yields Experimental samples

0.10
0.08

0.08
0.06

random sampling

0.06
yA=18.37
sA=4.23
0.04

0.04
0.02

µA 0.02
0.00
0.00

5 10 15 20 25 30 35
yA
‘All possible' B wheat yields Experimental samples
0.08

0.12
0.06

random sampling
0.08
0.04

yB=24.30
sB=5.15
0.04
0.02

µB
0.00
0.00

5 10 15 20 25 30 35
yB

Figure 3.1: The population model

CHAPTER 3. TESTS BASED ON POPULATION MODELS 28

and we say that ȲA is a consistent estimator for µA . Several of our other
sample characteristics are also consistent for the corresponding population
characteristics. As n → ∞,
ȲA → µA
s2A → σA2
x
#{Yi,A ≤ x}
Z
= F̂A (x) → FA (x) = pA (y)dy
nA −∞

Back to hypothesis testing:

We can formulate null and alternative hypotheses in terms of population

quantities. For example, if µB > µA , we would recommend B over A, and
vice versa.
• H 0 : µA = µB
• H1 : µA 6= µB
The experiment is performed and it is observed that
18.37 = ȳA < ȳB = 24.3
This is some evidence that µA > µB . How much evidence is it? Should we
reject the null hypothesis? Consider evaluating evidence against H0 with our
t-statistic:
|ȳB − ȳA |
gt (yA , yB ) = p
sp 1/nA + 1/nB
To decide whether to reject H0 or not, we need to know the distribution of
g(YA , YB ) under H0 . Consider the following setup:
Assume:
Y1,A , . . . , YnA ,A ∼ i.i.d. pA
Y1,B , . . . , YnB ,B ∼ i.i.d. pB
Evaluate: H0 : µA = µB versus H1 : µA 6= µB , i.e., whether or not
Z Z
ypA (y)dy = ypB (y)dy.

To make this evaluation and obtain a p-value, we need the distribution of

g(YA , YB ) under µA = µB . This will involve assumptions about/approxima-
tions to pA and pB .
CHAPTER 3. TESTS BASED ON POPULATION MODELS 29

3.2 The normal distribution

The normal distribution is useful because in many cases,

• our data are approximately normally distributed, and/or

• our sample means are approximately normally distributed.

These are both due to the central limit theorem. Letting P (µ, σ) denote a
population with mean µ and variance σ 2 , then

X1 ∼ P1 (µ1 , σ12 ) 

m
X2 ∼ P2 (µ2 , σ22 ) 
 X
X X 
.. ⇒ X i ∼
˙ normal µ j , σj2 .
. 
 j=1
2

Xm ∼ Pm (µm , σm ) 

Sums of varying quantities are approximately normally distributed.

Normally distributed data

Consider crop yields from plots of land:

Yi = a1 × seedi + a2 × soili + a3 × wateri + a4 × suni + · · ·

The empirical distribution of crop yields from a population of fields with

varying quantities of seed, soil, water, sun, etc. will be approximately normal
(µ, σ), where µ and σ depend on the effects a1 , a2 , a3 , a4 , . . . and the variability
of seed, soil, water, sun, etc..
Additive effects ⇒ normally distributed data

Normally distributed means

Consider the following scenario:

(1) (1)
Experiment 1: sample y1 , . . . , yn ∼ i.i.d. p and compute ȳ (1) ;
(2) (2)
Experiment 2: sample y1 , . . . , yn ∼ i.i.d. p and compute ȳ (2) ;
..
.
CHAPTER 3. TESTS BASED ON POPULATION MODELS 30

(m) (m)
Experiment m: sample y1 , . . . , yn ∼ i.i.d. p and compute ȳ (m) .

A histogram of {ȳ (1) , . . . , ȳ (m) } will look approximately normally distributed

with

sample mean {y (1) , . . . , y (m) } ≈ µ

sample variance {y (1) , . . . , y (m) } ≈ σ 2 /n

i.e. the sampling distribution of the mean is approximately normal(µ, σ 2 /n),

even if the sampling distribution of the data are not normal.

Basic properties of the normal distribution:

• Y ∼ normal(µ, σ 2 ) ⇒ aY + b ∼normal(aµ + b, a2 σ 2 ).

• Y1 ∼ normal(µ1 , σ12 ), Y2 ∼ normal(µ2 , σ22 ), Y1 , Y2 independent

⇒ Y1 + Y2 ∼ normal(µ1 + µ2 , σ12 + σ22 )

• if Y1 , . . . , Yn ∼ i.i.d. normal(µ, σ 2 ), then Ȳ is statistically independent

of s2 .

How does this help with hypothesis testing?

Consider testing H0 : µA = µB (treatment doesn’t affect mean). Then re-

gardless of distribution of data, under H0 :
·
ȲA ∼ normal(µ, σA2 /nA )
·
ȲB ∼ normal(µ, σB2 /nB )
· 2
ȲB − ȲA ∼ normal(0, σAB )
2
where σAB = σA2 /nA + σB2 /nB . So if we knew the variances, we’d have a null
distribution.

3.3 Introduction to the t-test

Consider a simple one-sample hypothesis test:

Y1 , . . . , Yn ∼ i.i.d. P , with mean µ and variance σ 2 .

CHAPTER 3. TESTS BASED ON POPULATION MODELS 31

H 0 : µ = µ0

H1 : µ 6= µ0

Examples:

Physical therapy

• Yi = muscle strength after treatment - muscle score before.

• H0 : E[Yi ] = 0

Physics

• Yi = boiling point of a sample of an unknown liquid.

• H0 : E[Yi ] = 100◦ C.

To test H0 , we need a test statistic and its distribution under H0 .

|ȳ − µ0 | might make a good test statistic:

• it is sensitive to deviations from H0 .

• its sampling distribution is approximately known:

E[Ȳ ] = µ
Var[Ȳ ] = σ 2 /n
Ȳ is approximately normal.

Under H0
(Ȳ − µ0 ) ∼ normal(0, σ 2 /n),
but we can’t use this as a null distribution because σ 2 is unknown. What if
we scale (Ȳ − µ0 )? Then
Ȳ − µ0
f (Y) = √
σ/ n
is approximately standard normal and we write f (Y) ∼ normal(0, 1). Since
this distribution contains no unknown parameters we could potentially use
it as a null distribution. However, having observed the data y, is f (y) a
statistic?

• ȳ is computable from the data and n is known;

CHAPTER 3. TESTS BASED ON POPULATION MODELS 32

• µ0 is our hypothesized value, a fixed number that we have chosen.

• σ is not determined by us and is unknown.

The solution to this problem is to approximate the population variance σ 2

with the sample variance s2 .

One-sample t-statistic:

Ȳ − µ0
t(Y) = √
s/ n
For a given value of µ0 this is a statistic. What is the null distribution of
t(Y)?
Ȳ − µ0 Ȳ − µ0
s ≈ σ so √ ≈ √
s/ n σ/ n
If Y1 , . . . , Yn ∼ i.i.d. normal(µ0 , σ 2 ) then Ȳσ/−µ
√ 0 is normal(0, 1), and so it would
n
seem that t(Y) is approximately distributed as a standard normal distribu-
tion under H0 : µ = µ0 . However, if the approximation s ≈ σ is poor, like
when n is small, we need to take account of our uncertainty in the estimate
of σ.

The χ2 distribution
X
Z1 , . . . , Zn ∼ i.i.d. normal(0, 1) ⇒ Zi2 ∼ χ2n , chi-squared dist with n degrees of freedom
X
(Zi − Z̄)2 ∼ χ2n−1

Y1 , . . . , Yn ∼ i.i.d. normal(µ, σ) ⇒ (Y1 − µ)/σ, . . . , (Yn − µ)/σ ∼ i.i.d. normal(0, 1)

1 X
⇒ 2
(Yi − µ)2 ∼ χ2n
σ
1 X
2
(Yi − Ȳ )2 ∼ χ2n−1
σ
Some intuition: Which vector do you expect to be bigger: (Z1 , . . . , Zn ) or
(Z1 − Z̄, . . . , Zn − Z̄)? (Z1 − Z̄, . . . , Zn − Z̄) is a vector of length n but lies
in an n − 1 dimensional space. In fact, it is has a singular multivariate
normal distribution, with a covariance matrix of rank (n − 1).