Statistics 2B

Chapter 9: Testing Hypotheses and

Assessing Goodness of Fit

Vaughan van Appel

University of Johannesburg
vvanappel@[Link]

Department of Statistics

1 / 32
Overview
9.1 Introduction

9.2 The Neyman-Pearson Paradigm

9.2.1 Specification of the Significance Level and the Concept of

a p-value

9.2.2 The Null Hypothesis

9.2.3 Uniformly Most Powerful Tests

9.3 The Duality of Confidence Intervals and Hypothesis Tests

9.4 Generalized Likelihood Ratio Tests

9.8 Probability Plots

9.9 Tests for Normality

2 / 32
Aims of hypothesis testing

Often, we want to test a hypothesis we entertain on the basis of

some data

Examples:

1. Does talking on a cell phone impair driving?

2. Does a drug have a beneficial effect on patients suffering from
a certain disease?
3. Is a die fair?
I Roll the die many times. Can we say whether or not the die is
fair? e.g.: 134655353425364534251632546
4. Does the Higgs Boson exist? Is its mass larger than 120 GeV?

3 / 32
9.1 Introduction

Suppose that I have two coins, coin 0 has probability of heads

equal to 0.5 and coin 1 has probability of heads equal to 0.7. I
choose one of the coins, toss it 10 times and tell you the number
of heads, but do not tell you whether it was coin 0 or coin 1.

On the basis of the number of heads, your task is to decide which

coin it was. How should your decision rule be?

Let X denote the number of heads. The following Figure gives

p(x) for each of the coins.

4 / 32
Suppose that you observed two heads. Then PP10 (2)
(2)
is about 30,
which we will call the likelihood ratio, coin 0 was about 30 times
more likely to produce this result than was coin 1. This result
would favor coin 0.

On the other hand, if there were 8 heads, the likelihood ratio

would be 0.0439
0.2335 = 0.19, which would favor coin 1. We specify two
hypotheses, H0 and H1 , according to whether coin 0 or coin 1 was
tossed.

Read through the Bayesian methodology (Not for exam

purposes)

5 / 32
9.2 The Neyman-Pearson Paradigm

Rather than using a Bayesian approach, Neyman and Pearson

formulated their theory of hypothesis testing by casting it as a
decision problem and making the probabilities of the two types of
errors central, thus bypassing the necessity of specifying prior
probabilities.

In doing so, this approach introduced an asymmetry: one

hypothesis is singled out as the null hypothesis and the other as
the alternative hypothesis, the former usually denoted by H0 and
the latter by H1 or HA . The following terminology is standard:

I Rejecting H0 when it is true is called a type I error.

I The probability of a type I error is called the significance
level of the test and is usually denoted by α.

6 / 32
 I Accepting the null hypothesis when it is false is called a type
II error and its probability is usually denoted by β.
I The probability that the null hypothesis is rejected when it is
false is called the power of the test, and equals 1 − β.
I Ideally, we’d like to have α and β equal to 0 - Impossible in
practice. In practice, if one decreases α, β must increase.

Neyman-Pearson approach:

1. Fix α (probability of type I error) at a small value.

2. Then, find the best possible test, i.e, test with level α and
biggest power.

In other words, find the test that is best at differentiating

alternative from null, given that it does not reject the null too
much when the null is true.

7 / 32
 Table: Diagram type I-type II error

Decision \ Truth H0 HA
H0 type II error
HA type I error power

Analogy: trial: H0 : Innocent; HA : Guilty

We want α small (do not condemn innocent)
We want β small, too (condemn guilty)

8 / 32
I We have seen in this example how rejecting H0 when the
likelihood ratio is less than a constant c is equivalent to
rejecting when the number of heads is greater than some
value x0 . The likelihood ratio, or equivalently, the number of
heads, is called the test statistic.
I The set of values of the test statistic that leads to rejection of
the null hypothesis is called the rejection region, and the set
of values that leads to acceptance is called the acceptance
region.
I The probability distribution of the test statistic when the null
hypothesis is true is called the null distribution.

9 / 32
In the example in the introduction to this chapter, the null and
alternative hypotheses each completely specify the probability
distribution of the number of heads, as Bin(10, 0.5) or
Bin(10, 0.7), respectively. These are called simple hypotheses.

The Neyman-Pearson Lemma shows that basing the test on the

likelihood ratio as we did is optimal:
Lemma (Neyman-Pearson)
Suppose that H0 and H1 are simple hypotheses and that the test
that rejects H0 whenever the likelihood ratio is less than c and
significance level α. Then any other test for which the significance
level is less than or equal to α has power less than or equal to that
of the likelihood ratio test. 2

Conclusion: if we can design a likelihood ratio test with

significance level α, it is the most-powerful (i.e. best) test at this
significance level!!
Practical significance: we will try to design LR tests, as they are
in some sense the best possible tests.
10 / 32
How to design a test in practice?

1. Write likelihood ratio (LR)

2. Assume parameters known and fixed
I When is LR small as function of data?
I Equivalently, find T (X1 , . . . , Xn ) such that if T falls in the
rejection region R, LR is small.
3. Use the distribution of T under the null hypothesis (i.e
assuming it’s true) to find the explicit rejection region. (The
distribution of T under the null is called the null distribution
of T )

Recall: focusing on T makes computation easier than focusing on

the LR-statistic f0 /f1

11 / 32
Example (A)
Let X1 , . . . , Xn be a random sample from a normal distribution having
known variance σ 2 . Consider two simple hypotheses:

H0 : µ = µ0 HA : µ = µ1

where µ0 and µ1 are given constants. Let the significance level α be

prescribed. The Neyman-Pearson Lemma states that among all tests with
significance level α, the test that rejects for small values of the likelihood
ratio is most powerful. We thus calculate the likelihood ratio, which is:
 −1 Pn 2

f0 (X) exp 2σ 2 i=1 (Xi − µ0 )
=  −1 Pn
2

i=1 (Xi − µ1 )
f1 (X) exp 2σ 2

since the multipliers of the exponentials cancel. Small values of this

statistic
Pn correspond P to small values of
2 n 2
i=1 (X i − µ 1 ) − i=1 (Xi − µ0 ) . Expanding the squares, we see that
the latter expression reduces to:

2nX̄(µ0 − µ1 ) + nµ21 − nµ20

12 / 32
9.2.1 Specification of the Significance Level and the Concept of a p-value
I One of the strengths of the Neyman-Pearson approach is that
only the distribution under the null hypothesis is needed in
order to construct a test.
I The theory requires the specification of the significance level,
α, in advance of analyzing the data, but gives no guidance
about how to make this choice.
I In practice it is almost always the case that the choice of α is
essentially arbitrary, but is heavily influenced by custom. Small
values, such as 0.01 and 0.05, are commonly used.
I A criticism of the paradigm is that it is built on the
assumption that one must either reject or not reject a
hypothesis.
I Evidence is often summarized as a p-value, which is defined to
be the smallest significance level at which the null hypothesis
would be rejected.
I Moreover, the p-value is the probability under the null
hypothesis of a result as or more extreme than that actually
observed. Thus, the smaller the p-value, the stronger the
evidence against the null hypothesis.
13 / 32
9.2.2 The Null Hypothesis

I In Example B of Section 9.2, we chose as the null hypothesis

the hypothesis that the distribution was Poisson and as the
alternative hypothesis the hypothesis that the distribution was
not Poisson. In this case, the null hypothesis is simpler than
the alternative.
I The consequences of incorrectly rejecting one hypothesis may
be graver than those of incorrectly rejecting the other. In such
a case, the former should be chosen as the null hypothesis,
because the probability of falsely rejecting it could be
controlled by choosing α.
I In scientific investigations, the null hypothesis is often a simple
explanation that must be discredited in order to demonstrate
the presence of some physical phenomenon or effect.

14 / 32
Extra Example
I A machine is designed to fill boxes with 16 ounces of cereal.
I If X is the weight of a filled box, suppose X is N (µ, 0.16).
I To test the hypotheses
H0 : µ = 16
HA : µ 6= 16,
we decide to sample 10 boxes and reject H0 if
|X̄ − 16| ≥ 0.25.
I That is, we take X̄ as our test statistic, setting an acceptance
region of
A = (15.75, 16.25)
and a critical region of
B = (−∞, 15.75] ∪ [16.25, ∞)
.
15 / 32
I Then the significance level of the test is

α = P (X̄ ∈ B | H0 )
= P (|X̄ − 16| ≥ 0.25 | µ = 16)
= 1 − P (−0.25 < X̄ − 16 < 0.25 | µ = 16)
!
−0.25 X̄ − 16 0.25
=1−P 0.4 < 0.4 < 0.4 µ = 16
√ √ √
10 10 10
= 1 − (Φ(1.98) − Φ(−1.98))
= 1 − (0.9762 − 0.0238)
= 0.0476

16 / 32
I Note: we cannot compute

β = P (X̄ ∈ A | HA )

because HA is composite, and so does not uniquely specify a

value of µ.
I However, we can evaluate the operating characteristic

β(µ) = P (X̄ ∈ A | µ)

the probability of accepting H0 when µ is the true mean.

I And we can evaluate the power function,

π(µ) = 1 − β(µ) = P (X̄ ∈ B | µ)

the probability of rejecting H0 when µ is the true mean.

17 / 32
I Note: π(16) = α, the significance level of the test.
I For example,

β(16.5) = P (X̄ ∈ A | µ = 16.5)

= P (15.75 < X̄ < 16.25 | µ = 16.5)
!
15.75 − 16.5 X̄ − 16.5 16.25 − 16.5
=P 0.4 < 0.4 < 0.4 µ = 16.5
√ √ √
10 10 10
= Φ(−1.98) − Φ(−5.93)
= 0.0238

and

π(16.5) = 1 − β(16.5) = 0.9762

18 / 32
9.2.3 Uniformly Most Powerful Tests

The optimality result of the Neyman-Pearson Lemma requires that

both hypotheses be simple. In some cases, the theory can be
extended to include composite hypotheses. If the alternative H1 is
composite, a test that is most powerful for every simple alternative
in H1 is said to be uniformly most powerful.
Example (A)
Continuing with Example A of Section 9.2, consider testing
H0 : µ = µ0 versus H1 : µ > µ0 . In Example A, we saw that for a
particular alternative µ1 > µ0 , the most powerful test rejects for
X̄ > x0 , where x0 depends on µ0 , σ, n, but not on µ1 . Because
this test is most powerful and is the same for every alternative, it is
uniformly most powerful. 2

19 / 32
It can also be argued that the test is uniformly most powerful for
testing H0 : µ ≤ µ0 versus H1 : µ > µ0 . But it is not uniformly
most powerful for testing H0 : µ = µ0 versus H1 : µ 6= µ0 .

This follows from further examination of the example, which shows

that the test that is most powerful against the alternative that
µ > µ0 rejects for large values of X̄ − µ0 , whereas the test that is
most powerful against the alternative µ < µ0 rejects for small
values of X̄ − µ0 . The most powerful test is thus not the same for
every alternative.

In typical composite situations, there is no uniformly most powerful

test. The alternatives H1 : µ < µ0 and H1 : µ > µ0 are called
one-sided alternatives. The alternative H1 : µ 6= µ0 is a two-sided
alternative.

20 / 32
9.3 The Duality of Confidence Intervals and
Hypothesis Tests

There is a duality between confidence intervals (more generally,

confidence sets) and hypothesis tests. In this section, we will show
that a confidence set can be obtained by ”inverting” a hypothesis
test, and vice versa. Before presenting the general structure, we
consider an Example A.

We now demonstrate that this duality holds more generally. Let θ

be a parameter of a family of probability distributions, and denote
the set of all possible values of θ by Θ. Denote the random
variables constituting the data by X.

21 / 32
Theorem (A)
Suppose that for every value θ0 in Θ there is a test at level α of
the hypothesis H0 : θ = θ0 . Denote the acceptance region of the
test by A(θ0 ). Then the set

C(X) = {θ : X ∈ A(θ)}

is a 100(1 − α) % confidence region for θ. 2

Theorem (B)
Suppose that C(X) is a 100(1 − α)% confidence region for θ; that
is, for every θ0 ,

P [θ0 ∈ C(X)|θ = θ0 ] = 1 − α

Then an acceptance region for a test at level α of the hypothesis

H0 : θ = θ0 is
A(θ0 ) = {X|θ0 ∈ C(X)}
2
22 / 32
9.4 Generalized Likelihood Ratio Tests
I The likelihood ratio test is optimal for testing a simple
hypothesis versus a simple hypothesis.
I In this section, we will develop a generalization of this test for
use in situations in which the hypotheses are not simple.
I Such tests are not generally optimal, but they are typically
nonoptimal in situations for which no optimal test exists, and
they usually perform reasonably well.
I Generalized likelihood ratio tests have wide utility; they play
the same role in testing as mle’s do in estimation.
I It is frequently the case that the hypotheses under
consideration specify, or partially specify, the values of
parameters of the probability distribution that has generated
the data.
I Specifically, suppose that the observations X = (X1 , ..., Xn )
have a joint density or frequency function f (X|θ). Then H0
may specify that θ ∈ ω0 , where ω0 is a subset of the set of all
possible values of θ, and H1 may specify that θ ∈ ω1 , where
ω1 is disjoint from ω0 .
23 / 32
I Let Ω = ω0 ∪ ω1 . Based on the data, a plausible measure of
the relative tenability of the hypotheses is the ratio of their
likelihoods.
I If the hypotheses are composite, each likelihood is evaluated
at that value of θ that maximizes it, yielding the generalized
likelihood ratio
max l(θ|x)
θ∈ω0
Λ∗ =
max l(θ|x)
θ∈ω1

I Small values of Λ∗ tend to discredit H0 .

I It is preferable for certain technical reasons to use the test
statistic
max l(θ|x)
θ∈ω0
Λ=
max l(θ|x)
θ∈Ω

rather than Λ∗ .

24 / 32
Note that Λ = min(Λ∗ , 1) so small values of Λ∗ correspond to
small values of Λ. The rejection region for a likelihood ratio test
consists of small values of Λ, for example, all Λ ≤ λ0 . The
threshhold λ0 is chosen so that P (Λ ≤ λ0 |H0 ) = α, the desired
significance level of the test.

In order for the likelihood ratio test to have the significance level
α, λ0 must be chosen so that P (Λ ≤ λ0 ) = α if H0 is true. If the
sampling distribution of Λ under H0 is known, we can determine
λ0 . Generally, the sampling distribution is not of a simple form,
but in many situations the following theorem provides the basis for
an approximation to the null distribution.

25 / 32
Theorem (A)
Under smoothness conditions on the probability density or
frequency functions involved, the null distribution of −2 log Λ
tends to a chi-square distribution with degrees of freedom equal to
dim Ω − dim ω0 as the sample size tends to infinity. 2

In the statement of Theorem A, dim Ω and dim ω0 are the

numbers of free parameters under Ω and ω0 , respectively. In
Example A, the null hypothesis completely specifies µ and σ 2 ;
there are no free parameters under ω0 , so dim ω0 = 0. Under Ω, σ
is fixed but µ is free, so dim Ω = 1. For this example, the null
distribution of −2 log Λ is exactly χ21 .

26 / 32
9.8 Probability Plots

Probability plots are an extremely useful graphical tool for

qualitatively assessing the fit of data to a theoretical distribution.
Consider a sample of size n from a uniform distribution on [0, 1].
Denote the ordered sample values by X(1) < X(2) < ... < X(n) .
These values are called order statistics. It can be shown (see
Problem 17 at the end of Chapter 4) that

j
E(X(j) ) =
n+1
This suggests plotting the ordered observations X(1) , X(2) , ..., X(n)
1 n
against their expected values (n+1) , ..., (n+1) . If the underlying
distribution is uniform, the plot should look roughly linear. Figure
9.4 is such a plot for a sample of size 100 from a uniform
distribution.

27 / 32
9.9 Tests for Normality

A wide variety of tests are available for testing goodness of fit to

the normal distribution. We discuss some of them in this section;
more discussion may be found in the works referred to.

28 / 32
Leave Out!
You can leave out Sections 9.5, 9.6, & 9.7

29 / 32
Tutorial Exercises
Do the following exercises (Starting on p. 362):

1, 2, 5, 6, 9, 12

30 / 32
References
John A. Rice
Mathematical Statistics and Data Analysis
3rd edition (2007)
Cengage Learning.

31 / 32
Questions?

32 / 32