CHAPTER 2.

TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS12

1.0

0.06

0.04
0.8

0.04
0.4 0.6

Density

Density
F(y)

0.02
0.02
0.2

0.00

0.00
0.0

15 20 25 10 15 20 25 30 0 10 20 30 40
y y y
1.0

0.00 0.08

0.12
Density
0.8

0.08
0.4 0.6

Density

10 15 20 25
F(y)

yA
0.04
0.00 0.10
Density
0.2

0.00
0.0

15 20 25 10 15 20 25 30 10 15 20 25 30 35
y yB

Figure 2.1: Wheat yield distributions

CHAPTER 2. TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS13

> sd (yA)
[ 1 ] 4.234934
> sd (yB)
[ 1 ] 5.151699

> q u a n t i l e (yA , prob=c ( . 2 5 , . 7 5 ) )

25% 75%
16.850 20.725
> q u a n t i l e (yB , prob=c ( . 2 5 , . 7 5 ) )
25% 75%
24.550 26.825

So there is a difference in yield for these wheat fields.

Would you recommend B over A for future plantings?
Do you think these results generalize to a larger population?

2.2 Hypothesis testing via randomization

Questions:

• Could the observed differences be due to fertilizer type?

• Could the observed differences be due to plot-to-plot variation?

Hypothesis tests:

• H0 (null hypothesis): Fertilizer type does not affect yield.

• H1 (alternative hypothesis): Fertilizer type does affect yield.

A statistical hypothesis test evaluates the plausibility of H0 in light of the

data.
Suppose we are interested in mean wheat yields. We can evaluate H0 by
answering the following questions:

• Is a mean difference of 5.93 plausible/probable if H0 is true?

• Is a mean difference of 5.93 large compared to experimental noise?

To answer the above, we need to compare

{|ȳB − ȳA | = 5.93}, the observed difference in the experiment
CHAPTER 2. TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS14

to
values of |ȳB − ȳA | that could have been observed if H0 were true.
Hypothetical values of |ȳB − ȳA | that could have been observed under H0 are
referred to as samples from the null distribution.

Finding a null distribution: Let

g(YA , YB ) = g({Y1,A , . . . , Y6,A }, {Y1,B , . . . , Y6,B }) = |ȲB − ȲA |.

This is a function of the outcome of the experiment. It is a statistic. Since

we will use it to perform a hypothesis test, we will call it a test statistic.

Observed test statistic: g(11.4, 23.7, . . . , 14.2, 24.3) = 5.93 = gobs

Hypothesis testing procedure: Compare gobs to g(YA , YB ) for values of YA

and YB that could have been observed, if H0 were true.

Recall the design of the experiment:

1. Cards were shuffled and dealt B, R, B, R, . . . and fertilizer types planted
in subplots:

B A B A B B

B A A A B A

2. Crops were grown and wheat yields obtained:

B A B A B B
26.9 11.4 26.6 23.7 25.3 28.5
B A A A B A
14.2 17.9 16.5 21.1 24.3 19.6

Now imagine re-doing the experiment in a universe where “H0 : no treatment

effect” is true:
1. Cards are shuffled and dealt B, R, B, B, . . . and wheat types planted in
subplots:
CHAPTER 2. TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS15

B A B B A A

A B B A A B

2. Crops are grown and wheat yields obtained:

B A B B A A
26.9 11.4 26.6 23.7 25.3 28.5
A B B A A B
14.2 17.9 16.5 21.1 24.3 19.6

Under this hypothetical treatment assignment,

(YA , YB ) = {11.4, 25.3, . . . , 21.1, 19.6}

|ȲB − ȲA | = 1.07

This represents an outcome of the experiment in a universe where

• The treatment assignment is B, A, B, B, A, A, A, B, B, A, A, B;

• H0 is true.

IDEA: To consider what types of outcomes we would see in universes where

H0 is true, compute g(YA , YB ) under every possible treatment assignment
and assuming H0 is true.
Under our randomization scheme, there were
 
12! 12
= = 924
6!6! 6

equally likely ways the treatments could have been assigned. For each one
of these, we can calculate the value of the test statistic that would’ve been
observed under H0 :
{g1 , g2 , . . . , g924 }
This enumerates all potential pre-randomization outcomes of our test statis-
tic, assuming no treatment effect. Along with the fact that each treatment
CHAPTER 2. TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS16

0.12

0.20
0.08
Density

Density
0.10
0.04 0.00

0.00
−10 −5 0 5 10 0 2 4 6 8
YB − YA |YB − YA|

Figure 2.2: Approximate randomization distribution for the wheat example

assignment is equally likely, these value give a null distribution, a probability

distribution of possible experimental results, if H0 is true.

#{gk ≤ x}
F (x|H0 ) = Pr(g(YA , YB ) ≤ x|H0 ) =
924
This distribution is sometimes called the randomization distribution, be-
cause it is obtained by the randomization scheme of the experiment.

Comparing data to the null distribution:

Is there any contradiction between H0 and our data?

Pr(g(YA , YB ) ≥ 5.93|H0 ) = 0.056

According to this calculation, the probability of observing a mean differ-

ence of 5.93 or more is unlikely under the null hypothesis. This probability
calculation is called a p-value. Generically, a p-value is

“The probability, under the null hypothesis, of obtaining a result as or more

extreme than the observed result.”
CHAPTER 2. TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS17

The basic idea:

small p-value → evidence against H0

large p-value → no evidence against H0

Approximating a randomization distribution:

We don’t want to have to enumerate all nAn+n


A
B
possible treatment assign-
ments. Instead, repeat the following S times for some large number S:

(a) randomly simulate a treatment assignment from the population of pos-

sible treatment assignments, under the randomization scheme.

(b) compute the value of the test statistic, given the simulated treatment
assignment and under H0 .

The empirical distribution of {g1 , . . . , gS } approximates the null distribu-

tion:

#(gs ≥ gobs )
≈ Pr(g(YA , YB ) ≥ gobs |H0 )
S
The approximation improves if S is increased.
Here is some R-code:
y<−c ( 2 6 . 9 , 1 1 . 4 , 2 6 . 6 , 2 3 . 7 , 2 5 . 3 , 2 8 . 5 , 1 4 . 2 , 1 7 . 9 , 1 6 . 5 , 2 1 . 1 , 2 4 . 3 , 1 9 . 6 )
x<−c ( ”B” , ”A” , ”B” , ”A” , ”B” , ”B” , ”B” , ”A” , ”A” , ”A” , ”B” , ”A” )

g . n u l l <−r e a l ( )
for ( s in 1:10000)
{
xsim<−sample ( x )
g . n u l l [ s]<− abs ( mean ( y [ xsim==”B ” ] ) − mean ( y [ xsim==”A” ] ) )
}

2.3 Essential nature of a hypothesis test

Given H0 , H1 and data y = {y1 , . . . , yn }:
CHAPTER 2. TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS18

1. From the data, compute a relevant test statistic g(y): The test statistic
g(y) should be chosen so that it can differentiate between H0 and H1 in
ways that are scientifically relevant. Typically, g(y) is chosen so that

small under H0
g(y) is probably
large under H1

2. Obtain a null distribution : A probability distribution over the possible

outcomes of g(Y) under H0 . Here, Y = {Y1 , . . . , Yn } are potential
experimental results that could have happened under H0 .

3. Compute the p-value: The probability under H0 of observing a test

statistic g(Y) as or more extreme than the observed statistic g(y).

p-value = Pr(g(Y) ≥ g(y)|H0 )

If the p-value is small ⇒ evidence against H0

If the p-value is large ⇒ not evidence against H0
Even if we follow these guidelines, we must be careful in our specification of
H0 , H1 and g(Y) for the hypothesis testing procedure to be useful.

Questions:
• Is a small p-value evidence in favor of H1 ?

• Is a large p-value evidence in favor of H0 ?

• What does the p-value say about the probability that the null hypoth-
esis is true? Try using Bayes’ rule to figure this out.

2.4 Sensitivity to the alternative hypothesis

In the previous section we said that the test statistic g(y) should be able
to “differentiate between H0 and H1 in ways that are scientifically relevant.”
What does this mean?
Suppose our data consist of samples yA and yB from two populations A and
B. Previously we used g(yA , yB ) = |ȳB − ȳA |. Let’s consider two different
test statistics:
CHAPTER 2. TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS19

t-statistic:
|ȳ − ȳA |
gt (yA , yB ) = p B , where
sp 1/nA + 1/nB
nA − 1 nB − 1
s2p = s2A + s2
(nA − 1) + (nB − 1) (nA − 1) + (nB − 1) B
This is a scaled version of our previous test statistic, in which we com-
pare the difference in sample means to a pooled version of the sample
standard deviation and the sample size. Note that this statistic is
• increasing in |ȳB − ȳA |;
• increasing in nA and nB ;
• decreasing in sp .
A more complete motivation for using this statistic will be given in the
next chapter.
Kolmogorov-Smirnov statistic:
gKS (yA , yB ) = max |F̂B (y) − F̂A (y)|
y∈R

This is just the size of the largest gap between the two sample CDFs.

Comparing the test statistics:

Suppose we perform a CRD and obtain samples yA and yB like those in

Figure 2.3. For these data,
• nA = nB = 40
• ȳA = 10.05, ȳB = 9.70.
• sA = 0.87, sB = 2.07
The main difference between the two samples seems to be in their variances
and not in their means. Now let’s consider evaluating
H0 : treatment does not affect response
using our two new test statistics. We can approximate the null distributions
of gt (YA , YB ) and gKS (YA , YB ) by randomly reassigning the treatments but
leaving the responses fixed:
CHAPTER 2. TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS20

0.6

1.0
Density
0.3

0.8
0.0

0.6
6 8 10 12 14

F(y)
yA

0.4
Density

0.2
0.10

0.0
0.00

6 8 10 12 14 6 8 10 12 14
yB y

Figure 2.3: Histograms and empirical CDFs of the first two hypothetical
samples.

Gsim<−NULL
for ( s in 1:5000)
{
xsim<−sample ( x )
yAsim<−y [ xsim==”A” ] ; yBsim<−y [ xsim==”B” ]
g1<− g . t s t a t ( yAsim , yBsim )
g2<− g . ks ( yAsim , yBsim )
Gsim<−r b i n d ( Gsim , c ( g1 , g2 ) )
}

These calculations give:

t-statistic : gt (yA , yB ) = 1.00 , Pr(gt (YA , YB ) ≥ 1.00) = 0.321

KS-statistic: gKS (yA , yB ) = 0.30 , Pr(gKS (YA , YB ) ≥ 0.30) = 0.043

The hypothesis test based on the t-statistic does not indicate strong evidence
against H0 , whereas the test based on the KS-statistic does. The reason is
that the t-statistic is only sensitive to differences in means. In particu-
lar, if ȳA = ȳB then the t-statistic is zero, its minimum value. In contrast, the
KS-statistic is sensitive to any differences in the sample distributions.
Now let’s consider a second dataset, shown in Figure 2.5, for which

• nA = nB = 40
CHAPTER 2. TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS21

6
0.6

5
Density

Density
3 4
0.4

2
0.2

1
0.0

0
0 1 2 3 4 0.1 0.2 0.3 0.4
t statistic KS statistic

Figure 2.4: Randomization distributions for the t and KS statistics for the
first example.

• ȳA = 10.11, ȳB = 10.73.

• sA = 1.75, sB = 1.85

The difference in sample means is about twice as large as in the previous ex-
ample, and the sample standard deviations are pretty similar. The B-samples
are slightly larger than the A-samples on average. Is there evidence that this
is caused by treatment? Again, we evaluate H0 using the randomization
distributions of our two test statistics.

t-statistic : gt (yA , yB ) = 1.54 , Pr(gt (YA , YB ) ≥ 1.54) = 0.122

KS-statistic: gKS (yA , yB ) = 0.25 , Pr(gKS (YA , YB ) ≥ 0.25) = 0.106

This time the two test statistics indicate similar evidence against H0 . This
is because the difference in the two sample distributions could primarily be
summarized as the difference between the sample means, which the t-statistic
can identify.
CHAPTER 2. TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS22

0.15 0.30

1.0
Density

0.8
0.00

0.6
8 10 12 14 16

F(y)
yA

0.4
Density
0.15

0.2
0.0
0.00

8 10 12 14 16 8 10 12 14 16
yB y

Figure 2.5: Histograms and empirical CDFs of the second two hypothetical
samples.
6
0.6

5
Density

Density
3 4
0.4

2
0.2

1
0.0

0 1 2 3 4 0.1 0.2 0.3 0.4 0.5

t statistic KS statistic

Figure 2.6: Randomization distributions for the t and KS statistics for the
second example.